Real-time image scene virtual-real fusion processing method

Abstract

The present application relates to the technical field of image fusion, in particular to a real-time image scene virtual-real fusion processing method, comprising the following steps: acquiring scene geometric changes, light fluctuation and motion trajectory captured by a multi-modal sensor, identifying a light stable segment and marking a position, collecting a rotation angle and a virtual object size relationship, screening an embedding boundary, adjusting alignment, identifying a light consistency compensation state and an anti-aliasing trigger time intersection, and obtaining a virtual-real fusion processing record, in the present application, through the rationality of scene geometric structure changes, light stability and virtual object embedding position, accurate fusion of virtual content and real scene in a dynamic environment is realized, through delimiting a virtual-real interference area, fusion problems caused by unstable light or geometric changes are avoided, light consistency and visual coordination are ensured, the naturalness and consistency of virtual-real fusion are improved, the smoothness of rendering effect is improved, and the visual experience and interactivity of virtual-real fusion images are overall enhanced.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, and in particular to a method for real-time image scene virtual-real fusion processing. Background Technology

[0002] Image fusion technology involves analyzing, registering, and integrating images or image data from different sources to generate output images containing more semantic or visual information. It is an important branch of computer vision and image processing. The core aspects of this technology include image alignment, feature extraction, image registration, fusion strategy design, and multi-source information integration. It is widely used in remote sensing imaging, medical image processing, augmented reality, and computational photography. Especially in interactive visual scenarios with high real-time requirements, image fusion technology needs to achieve rapid processing and accurate overlay of multimodal data while ensuring image quality and scene consistency.

[0003] Traditional real-time image scene fusion processing methods involve acquiring image information of the real environment through sensors and overlaying pre-defined virtual elements onto the scene to achieve visual integration of the virtual and real worlds. These methods address the challenge of achieving geometric alignment and lighting coordination between virtual content and the real-world scene in a dynamically changing environment, thereby improving the naturalness and consistency of the fused image. Traditional approaches often employ geometric reconstruction-based visual positioning technology to model the real scene, combined with rule-driven image overlay strategies to embed virtual objects into the image.

[0004] Existing technologies struggle to effectively handle rapidly changing lighting fluctuations and geometric changes when dealing with dynamic scenes, affecting the fusion of virtual content with the real-world environment. Traditional methods rely on static modeling techniques, which cannot adjust the lighting and position of virtual content in a timely manner, causing virtual elements to appear unnatural or misaligned under different lighting conditions, thus impacting the overall visual effect. When embedding virtual elements, traditional methods often neglect the real-time adaptability to dynamic changes, making it difficult to adjust in real time when lighting is unstable or scene structure changes abruptly, resulting in reduced image quality and failure to achieve the desired fusion effect. Summary of the Invention

[0005] To address the technical problems existing in the prior art, embodiments of the present invention provide a real-time image scene virtual-real fusion processing method, including the following steps:

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a real-time image scene virtual-real fusion processing method, comprising the following steps:

[0007] S1: Acquire the sequence of scene geometric structure changes, illumination fluctuation time periods and motion trajectories captured by the multimodal sensor, count the number of geometric structure changes and abrupt change points of motion trajectory, perform position mapping with the scene response area, and generate virtual and real interference area delineation results;

[0008] S2: Based on the results of the virtual and real interference region delineation, identify the lighting stable segments in the real scene, mark the lighting stable positions, and generate lighting stable segment data;

[0009] S3: Based on the light-stable section data, collect the rotation angle sequence, embedding direction setting value and standard virtual object size range mapping relationship in the scene, filter virtual objects that can be directly embedded in the boundary range, and generate virtual-real fusion alignment range;

[0010] S4: Based on the virtual-real fusion alignment interval, determine whether the virtual object embedding position enters the target boundary range. If it is not within the range, perform the alignment adjustment operation and output the embedding record.

[0011] As a further aspect of the present invention, the result of defining the virtual and real interference region includes the distribution of the number of geometric structure changes, the location of the abrupt change point of the motion trajectory, and the mapping relationship of the interference region; the data of the stable illumination segment includes information on the stable illumination direction segment, the fluctuation decay time, and the location of the stable illumination segment; the virtual and real fusion alignment interval includes the rotation angle interval, the embedding direction interval, and the qualified embedding boundary; and the embedding record includes the virtual object embedding position, the alignment direction status, and the alignment holding time.

[0012] As a further aspect of the present invention, the rotation angle sequence within the scene refers to the critical embedding angle sequence of virtual objects calculated according to the scene rotation characteristics;

[0013] The boundary range refers to the acceptable range of standard virtual object size tolerance ±3%.

[0014] As a further aspect of the present invention, the step of obtaining the virtual-real interference region delineation result specifically includes:

[0015] S101: Acquire standard geometric structure change sequence data captured by multimodal sensors, illumination fluctuation time period information, and time series trajectory of motion trajectory sensors, perform time alignment, count continuous change points, change amplitude, and number of change points, and generate the number of geometric structure changes.

[0016] S102: Based on the number of changes in the geometric structure, calculate the first-order slope of the motion trajectory difference sequence within each fluctuation time period, match the time of change point with the trajectory change trend, extract trajectory abrupt change points, and generate trajectory abrupt change point distribution data.

[0017] S103: Based on the trajectory mutation point distribution data, map the mutation point index to the response area number, and summarize the mutation point count by area to generate the virtual and real interference area delineation result.

[0018] As a further aspect of the present invention, the step of acquiring data in the stable illumination section specifically comprises:

[0019] S201: Based on the delineation results of the virtual and real interference areas, read the illumination direction and amplitude sequence, extract the time according to the fluctuation period, and combine the time with the motion trajectory change sequence to determine the illumination direction difference value and amplitude fluctuation, mark the illumination stable segment, and generate the illumination stable interval sequence.

[0020] S202: Based on the light stability interval sequence, combined with the motion trajectory state signal and the light amplitude change sequence, locate the unclosed time period, and perform time-series statistics on the continuously decreasing light amplitude segment to generate the light stability absorption segment;

[0021] S203: Based on the light-stable absorption section, spatial index conversion is performed using the spatial coordinate data of the light sensor to locate the scene coordinate section, map and mark the start and end times of the stable absorption section, and generate light-stable section data.

[0022] As a further aspect of the present invention, the step of obtaining the virtual-real fusion alignment interval specifically comprises:

[0023] S301: After collecting the data of the stable illumination section, obtain the rotation angle sequence corresponding to each time node in the scene, calculate the critical embedding angle of the virtual object under the rotation angle at each time point, and analyze the embedding angle field of the virtual object at the differentiated radial position to obtain the embedding angle distribution data.

[0024] S302: Based on the embedded angle distribution data, combined with the embedded direction and the size of the virtual object, cut the angle field according to the deflection force line direction, calculate the boundary value of the embedded path region, filter out the path segments without embedded structure, and generate the embedded path layered interval.

[0025] S303: Based on the embedded path layered interval, compare the virtual object size with the standard tolerance range, extract the path boundary that meets the size conditions, count the continuous length and number, mark the embeddable path and generate a number index table, and output the virtual-real fusion alignment interval.

[0026] As a further aspect of the present invention, the embedded record acquisition step specifically comprises:

[0027] S401: Based on the embedded path hierarchical interval, obtain the corresponding virtual object number, read the displacement record, compare the coordinate position with the physical boundary of the target segment, identify the offset between the virtual object edge and the boundary, and generate displacement alignment offset data;

[0028] S402: Based on the displacement alignment offset data, determine whether the virtual object is within the target range, analyze whether the difference between the bottom and top coordinates is completely within the boundary. If it is not within the boundary, adjust the slide rail driver through the offset direction signal until the offset is zero, and obtain the virtual object boundary positioning determination result.

[0029] S403: Based on the virtual object boundary positioning determination result, start the time recorder for the current coordinate position when the virtual object is in position, collect the holding time of the virtual object from the completion of the displacement to the stationary state, and record the slide rail number and coordinates, and store the time and number data together to generate an embedded record.

[0030] As a further aspect of the present invention, the displacement alignment offset data refers to the offset between the virtual object and the physical boundary of the target segment;

[0031] The time recorder refers to the time it takes for a virtual object to go from the completion of its shift to its stationary state.

[0032] As a further aspect of the present invention, the method further includes step S5:

[0033] S5: Based on the embedded record, identify the intersection of the illumination consistency compensation state and the spatiotemporal anti-aliasing trigger time, record the compensation adjustment, anti-aliasing action and rendering path state stage information, and obtain the virtual-real fusion processing record;

[0034] The virtual-real fusion processing record includes a compensation adjustment time series, an anti-aliasing action time series, and a rendering path state sequence.

[0035] As a further aspect of the present invention, the virtual-real fusion processing record acquisition step specifically comprises:

[0036] S501: Based on the embedded record, extract the virtual object trigger timestamp and number information, identify the illumination consistency compensation status and timestamp, determine whether there is an adjustment status record in the window, and if so, mark the virtual object interference status and generate a compensation cross-interference mark record.

[0037] S502: Based on the compensation cross-interference mark record, extract the virtual object number of the interference section, detect the pause state after alignment is completed, record the end time of the invalid state, and if the compensation state is fully enabled, record the timestamp and bind the virtual object number to obtain the virtual object unlock timestamp.

[0038] S503: Based on the unlock timestamp of the virtual object as the time base point, read the action record of the corresponding virtual object number, extract the sequence of compensation state and rendering path on / off state, arrange them in time order, and output the virtual-real fusion processing record.

[0039] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0040] This invention achieves more precise fusion of virtual content and real-world scenes in dynamic environments by comprehensively considering changes in scene geometry, lighting stability, and the rationality of virtual object embedding positions. By identifying and delineating virtual-real interference areas, it avoids fusion problems caused by unstable lighting or scene geometry changes. Simultaneously, based on the identification of stable lighting sections, it optimizes the embedding positions of virtual objects to ensure lighting consistency and visual coordination. Combined with the control of rotation angle and embedding direction, virtual objects can be embedded more naturally into the real scene, thereby improving the naturalness and consistency of virtual-real fusion. Real-time monitoring and adjustment of anti-aliasing processing makes the rendering effect smoother, reduces visual discomfort, and improves the overall visual experience and interactivity of virtual-real fusion images. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of the steps of the present invention;

[0043] Figure 2 This is a detailed schematic diagram of S1 of the present invention;

[0044] Figure 3 This is a detailed schematic diagram of S2 of the present invention;

[0045] Figure 4 This is a detailed schematic diagram of S3 of the present invention;

[0046] Figure 5 This is a detailed schematic diagram of S4 of the present invention;

[0047] Figure 6 This is a detailed schematic diagram of S5 of the present invention. Detailed Implementation

[0048] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0049] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0050] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0051] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0052] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0053] Please see Figure 1 This invention provides a real-time image scene virtual-real fusion processing method, including the following steps:

[0054] S1: Acquire the sequence of scene geometric structure changes, illumination fluctuation time periods and motion trajectories captured by the multimodal sensor, count the number of geometric structure changes and abrupt change points of motion trajectory, perform position mapping with the scene response area, and generate virtual and real interference area delineation results;

[0055] S2: Based on the results of the virtual and real interference region delineation, identify the lighting stable segments in the real scene, mark the lighting stable positions, and generate lighting stable segment data;

[0056] S3: Based on the data of stable lighting sections, collect the rotation angle sequence, embedding direction setting value and standard virtual object size range mapping relationship in the scene, filter virtual objects that can be directly embedded in the boundary range, and generate virtual-real fusion alignment range;

[0057] S4: Based on the virtual-real fusion alignment interval, determine whether the virtual object embedding position enters the target boundary range. If it is not within the range, perform the alignment adjustment operation and output the embedding record.

[0058] S5: Based on embedded records, identify the intersection of illumination consistency compensation state and spatiotemporal anti-aliasing trigger time, record compensation adjustment, anti-aliasing actions and rendering path state stage information, and obtain virtual-real fusion processing records.

[0059] The results of virtual-real interference region delineation include the distribution of geometric structure change frequency, the location of abrupt change points in motion trajectory, and the mapping relationship of interference regions. The data of stable lighting sections include information on stable lighting direction sections, fluctuation decay time, and the location of stable lighting segments. The virtual-real fusion alignment interval includes the rotation angle interval, the embedding direction interval, and the qualified embedding boundary. The embedding record includes the virtual object embedding position, alignment direction status, and alignment holding time. The virtual-real fusion processing record includes the compensation adjustment time series, the anti-aliasing action time series, and the rendering path status sequence.

[0060] The scene rotation angle sequence refers to the critical embedding angle sequence of virtual objects calculated according to the scene rotation characteristics;

[0061] The boundary range refers to the acceptable range of the standard virtual object size tolerance of ±3%.

[0062] Please see Figure 2 The specific steps for obtaining the results of the virtual-real interference region delineation are as follows:

[0063] S101: Acquire standard geometric structure change sequence data captured by multimodal sensors, illumination fluctuation time period information, and time series trajectory of motion trajectory sensors, perform time alignment, count continuous change points, change amplitude, and number of change points, and generate the number of geometric structure changes.

[0064] After capturing standard geometric change sequence data, illumination fluctuation time period information, and time-series trajectory data from motion trajectory sensors using multimodal sensors, depth camera data (30 frames / second), ambient light sensor data (50 sampling points / second), and robot posture sensor data (100 sampling points / second) from the gripper of the collaborative robot's end effector are collected. All data are synchronized to a 10-millisecond time resolution using linear interpolation. Subsequently, the gripper opening and closing angle sequence is analyzed. When the angle change exceeds 1.5 degrees over three consecutive sampling points (30 milliseconds), it is marked as a change start point; if the angle change is less than 0.5 degrees within the following 100 milliseconds, it is considered a change end point. The start time, duration, and total angle change amplitude of each change are recorded. For example, if the gripper changes angle from 10 degrees to 60 degrees starting at t=1.2 seconds, lasting 0.5 seconds, the change amplitude and duration are recorded. The number of such significant changes occurring throughout the entire observation period is counted, ultimately generating a geometric change count, including the amplitude and duration of each change.

[0065] S102: Based on the number of geometric structure changes, calculate the first-order slope of the motion trajectory difference sequence within each fluctuation time period, match the time of change point with the trajectory change trend, extract trajectory abrupt change points, and generate trajectory abrupt change point distribution data.

[0066] By utilizing the number of geometric structure changes, a corresponding time window is determined. For example, the opening and closing of the gripper occurs between 1.2 seconds and 1.7 seconds. For the motion trajectory data of the robot's end effector within this time window, the displacement vector between adjacent time points is calculated, and the first-order slope of the displacement vector magnitude is further calculated, i.e., the rate of change of the velocity vector magnitude. For example, if the velocity at a certain point rapidly increases from 0.1 m / s to 0.8 m / s within 50 milliseconds, a velocity change rate threshold is set, such as 0.7 m / s². If this threshold is exceeded, it is marked as a velocity mutation. The time of the geometric structure change point (e.g., 1.2 seconds) is matched with the time of the velocity mutation in the motion trajectory. For example, if the velocity mutation occurs within 100 milliseconds before or after the geometric structure change point, it is identified as a trajectory mutation point. The precise three-dimensional coordinates and timestamps of the trajectory mutation points are recorded, ultimately forming the trajectory mutation point distribution data.

[0067] S103: Based on the distribution data of trajectory mutation points, map the mutation point index to the response area number, and summarize the mutation point count by area to generate the result of virtual and real interference area delineation.

[0068] Based on the distribution data of trajectory mutation points, such as a mutation point at coordinates (0.105, 0.208, 0.302) meters, the work scene space is pre-divided into multiple discrete response regions. For example, based on the workstation layout, the space is defined as region A: (0, 0, 0) to (0.5, 0.5, 0.5) meters, and region B: (0.5, 0, 0) to (1.0, 0.5, 0.5) meters. Each trajectory mutation point is traversed, and its corresponding response region is determined based on its three-dimensional coordinates. For example, if the coordinates of the mutation point (0.105, 0.208, 0.302) meters fall within the geometric boundary of region A, then this point is indexed to region A. Subsequently, the trajectory mutation points mapped to each response region are cumulatively counted. For example, after a period of data statistics, region A accumulated 25 mutation points and region B accumulated 8 mutation points. Finally, the result of delineating the virtual and real interference regions is output. This result includes the unique number of each response region and the total number of trajectory mutation points detected in that region, so as to quantify the activity density of each region.

[0069] Please see Figure 3 The specific steps for obtaining data in the stable illumination section are as follows:

[0070] S201: Based on the results of the virtual and real interference area delineation, read the illumination direction and amplitude sequence, extract the time according to the fluctuation period, and combine the motion trajectory change sequence to reconstruct the time, determine the illumination direction difference value and amplitude fluctuation, mark the illumination stable segment, and generate the illumination stable interval sequence.

[0071] Based on the delineation of virtual and real interference areas, and utilizing the delineated interference areas and the identified illumination fluctuation time periods (e.g., 1.2 to 1.7 seconds), the illumination direction vector sequence and illumination amplitude sequence are acquired from the illumination sensor, and the data during the fluctuation time periods are precisely extracted. Subsequently, combined with the robot's motion trajectory change sequence, the illumination data is temporally reorganized to ensure precise alignment between illumination observation and robot motion state. The angular difference of illumination direction between adjacent time points is calculated, for example, 2 degrees; simultaneously, the relative fluctuation of illumination amplitude is calculated, for example, if the current illuminance is 500 Lux and the previous moment was 505 Lux, the relative fluctuation is 0.01. The threshold for illumination direction difference is set to 5 degrees, and the threshold for relative fluctuation of illumination amplitude is set to 0.02. If the illumination direction difference is less than or equal to 5 degrees and the relative fluctuation of illumination amplitude is less than or equal to 0.02, and this condition is maintained continuously for more than 2 seconds, then this time period is marked as an illumination stable segment. Finally, the start and end timestamp sequences of multiple illumination stable segments are generated, i.e., the illumination stable interval sequence.

[0072] S202: Based on the stable light interval sequence, combined with the motion trajectory state signal and the light amplitude change sequence, locate the unclosed time period, and perform time-series statistics on the continuously decreasing light amplitude segment to generate the stable light absorption segment.

[0073] Based on the stable illumination interval sequence, such as [2.1s, 4.5s] and [6.0s, 8.2s], unmarked time gaps between the stable intervals are first identified, such as the unclosed time interval from 4.5 seconds to 6.0 seconds. For the unclosed time interval, the robot's motion trajectory state signal (e.g., whether it is in a uniform motion state) and the illumination amplitude sequence are further analyzed. For example, if the robot moves at a constant speed of 0.3 m / s between 4.5 and 6.0 seconds, and the light amplitude continuously decreases from 500 Lux to 200 Lux with a decrease of (500-200) / 500=0.6, and the decrease lasts for at least 1 second, this segment in which the light amplitude continuously and significantly decreases under a specific motion state is identified as a segment with a continuously decreasing light amplitude. Time-series statistics are performed on the decreasing segment, for example, recording its start and end times and the decrease amplitude, to generate a stable light absorption segment, for example, [4.8s, 5.5s]. This segment indicates that the light conditions exhibit a stable decrease over a specific time period.

[0074] S203: Based on the stable light absorption section, spatial index conversion is performed using the spatial coordinate data of the light sensor to locate the scene coordinate section, map and mark the start and end time of the stable absorption section, and generate stable light absorption section data.

[0075] Based on the stable illumination intake segment, such as [4.8s, 5.5s], the precise position of the illumination sensor is obtained from pre-calibrated spatial coordinate data (e.g., the sensor is located at 1.5, 2.0, or 1.8 meters in the scene). A spatial indexing conversion method is used, for example, to associate the illumination changes observed by the sensor with the physical space covered by the robot's trajectory within that intake segment. For example, during [4.8s, 5.5s], the robot's end effector moves from (0.3, 0.4, 0.5) meters to (0.5, 0.6, 0.7) meters. The space traversed by the robot is defined as a cubic scene coordinate segment with a side length of 0.3 meters. The start and end times of the stable intake segment [4.8s, 5.5s] are precisely associated with this specific scene coordinate segment and stored as a triple, for example: {Scene Coordinate Segment ID: XXX, Start Time: 4.8s, End Time: 5.5s}. Finally, the output data is the stable illumination segment data. This data accurately describes the time interval of stable illumination conditions within a specific physical scene area, providing a basis for the stability of ambient illumination for subsequent virtual-real fusion alignment.

[0076] Please see Figure 4 The specific steps for obtaining the alignment range in the virtual-real fusion process are as follows:

[0077] S301: After collecting data from the stable lighting section, obtain the rotation angle sequence corresponding to each time node in the scene, calculate the critical embedding angle of the virtual object under the rotation angle at each time point, and analyze the embedding angle field of the virtual object at the differentiated radial position to obtain the embedding angle distribution data.

[0078] After collecting data from areas with stable lighting, the system continuously collects rotation angle sequences of the user (or AR headset) within the scene, such as Euler angle data at 100 frames per second. For each time point and corresponding rotation angle, the critical embedding angle of a virtual object (e.g., a virtual screw hole) relative to a physical surface (e.g., a workpiece plane) is calculated. This calculation is based on the geometric comparison between the virtual object's 3D model and the real-time depth map. For example, when the visual deviation between the virtual object's edge and the physical surface exceeds 0.5 degrees, it is determined to be non-critical. Further analysis of the embedding angle field of the virtual object at different radial positions (e.g., 1 meter and 3 meters) is performed. Because perspective effects influence visual perception, a 2D angle field is constructed, filled with critical embedding angle values ​​at different radial distances. For example, at a radial distance of 1 meter, the critical angle is between -15 degrees and +15 degrees. Finally, embedding angle distribution data containing time points, rotation angles, radial distances, and critical embedding angles are obtained.

[0079] S302: Based on the embedding angle distribution data, combined with the embedding direction and the size of the virtual object, cut the angle field along the deflection force line direction, calculate the boundary value of the embedding path region, filter out path segments without embedding structure, and generate the embedding path layered interval;

[0080] Embedded path region boundary values, using the formula:

[0081] ;

[0082] in, Represents the boundary value of the embedded path region. Represents the number of path segments. This represents the embedding direction deflection weight of the i-th path segment. Represents the embedding angle of the i-th path segment. The size of the virtual object representing the i-th path segment;

[0083] Based on the embedding angle distribution data, it is combined with the preset virtual object embedding direction (e.g., along the normal direction of the physical workpiece surface) and the virtual object size (e.g., the diameter of the virtual screw hole is 10 mm) to set a deflection force line direction. This represents the direction in which the virtual object is expected to be "pushed in" or "aligned" in real space, for example, the movement direction of a robot gripper arm. Using this deflection force line direction as a cutting standard, the angle field formed by the embedding angle distribution data is cut to identify the path segments along which the virtual object may be embedded. When calculating the boundary value of the embedding path region, the following formula is used: ;

[0084] In this formula: This represents the boundary value of the embedded path region. This value quantifies the "embedding difficulty" or "geometric mismatch" of a given set of path segments. The larger the value, the less ideal the embedded path is or the greater the geometric deviation.

[0085] The number of path segments represents the logical division of a complete potential embedding path into segments. A series of continuous segments, for example, breaking down the robot's path into multiple small straight or curved segments for easier analysis;

[0086] Representing the The embedding direction deflection weight of the i-th path segment, whose value ranges from 0 to 1, measures the i-th path segment's embedding direction deflection. The degree of alignment between the actual direction of a path segment and the ideal embedding direction (i.e., the direction of the deflection force line), where 1 represents perfect alignment and 0 represents perfect perpendicularity. For example, if the ideal embedding direction is the positive Z-axis, and the angle between the actual direction of a path segment and the positive Z-axis is 30 degrees, then... It can be calculated as This value, used as a weight, reduces the contribution of path segments that deviate from the ideal direction to the overall result;

[0087] Representing the The embedding angle of each path segment, in radians, represents the angle between the virtual object and the physical surface on that path segment. A smaller value indicates better alignment. For example, when the angle between the virtual screw hole and the normal direction of the physical surface is 5 degrees, the value is [value missing]. radian;

[0088] Representing the The virtual object size of each path segment, in meters, refers to the characteristic size of the virtual object on that path segment. For example, the diameter of a virtual screw hole is 0.01 meters. This size, as the denominator, means that for the same angular deviation and directional deflection, the smaller virtual object has a greater impact on the embedded boundary value, reflecting the actual situation that small-sized objects have higher precision requirements.

[0089] The logical approach of this formula is: through The "bias contribution" for each path segment is calculated, where the orientation deflection weight adjusts for the influence of the embedding angle and is determined by the virtual object size. Normalization is performed, then the contributions of all path segments are summed, the absolute value is taken, and finally the square root is taken. The summation operation aggregates the cumulative geometric mismatch of the entire path, the absolute value ensures the non-negativity of the result, and the square root can be used to adjust the units to be closer to the intuitive perception of distance or geometric deviation. The innovation of this formula lies in combining three key parameters—direction alignment, embedding angle, and object size—to comprehensively evaluate the quality of the embedded path, achieving a fine quantification of the geometric suitability of the path.

[0090] Example calculation: Suppose a potential embedding path is divided into 3 path segments (i.e. And the size of the virtual object remains consistent across all segments, for example... rice;

[0091] Path segment 1: Good orientation alignment. Embedding angle radian;

[0092] Path segment 2: The direction is slightly off. Embedding angle radian;

[0093] Path segment 3: The direction deviates significantly. Embedding angle radian;

[0094] Substitute the parameters into the formula: ;

[0095] Set a critical embedding path region boundary value threshold, for example, a threshold of 5.0. This means that when the R value exceeds this threshold, the path segment is considered to lack a valid embedding structure. In this example, the calculated value is... This result indicates that embedding this path is difficult and involves significant geometric mismatches. Therefore, path segments with calculated boundary values ​​exceeding a threshold are excluded and classified as having no embedding structure. For example, if a path segment's... If the value is 7.237, which exceeds the preset threshold of 5.0, the path segment is filtered out. Through this filtering process, a series of paths that meet the embedding conditions are generated and are layered according to their geometric features and spatial location. For example, paths suitable for horizontal embedding are grouped into one layer, and paths suitable for vertical embedding are grouped into another layer, thus obtaining the embedding path layering interval.

[0096] S303: Based on the embedded path hierarchical interval, the virtual object size is compared with the standard tolerance range to extract the path boundary that meets the size conditions, count the continuous length and number, mark the embeddable path and generate a number index table, and output the virtual-real fusion alignment interval.

[0097] Based on the embedded path hierarchical interval, for example, containing path segment P1, for each path segment, its geometric dimensions (e.g., effective width, depth) are compared with the precise dimensions of the virtual object (e.g., virtual screw hole diameter 0.01 meters, height 0.005 meters) and preset standard tolerance ranges (e.g., diameter tolerance ±0.0005 meters, height tolerance ±0.0003 meters). If the effective width of path segment P1 is 0.0102 meters and the effective depth is 0.0051 meters, both within the tolerance range, then the path segment is identified as a path boundary that meets the size conditions. Subsequently, the continuous length of the path segments that meet the conditions (e.g., 0.05 meters) is counted and assigned a unique number (e.g., P1-001). All path segments that meet the conditions are marked as embeddable paths, and a numbered index table is generated, containing path number, spatial location, length, and virtual object type. Finally, the virtual-real fusion alignment interval is output. This interval is the set of all paths marked as embeddable, providing spatial guidance for the subsequent precise alignment of virtual objects with the actual scene.

[0098] Please see Figure 5 The specific steps for obtaining embedded records are as follows:

[0099] S401: Obtain the corresponding virtual object number based on the embedded path hierarchical interval, read the displacement record, compare the coordinate position with the physical boundary of the target segment, identify the offset of the virtual object edge and the boundary, and generate displacement alignment offset data;

[0100] Displacement alignment offset data refers to the offset between a virtual object and the physical boundary of the target segment;

[0101] Based on the embedded path hierarchical intervals, the virtual object VO-001 (e.g., a virtual screw) corresponding to the embeddable path P1-001 is identified. The real-time 3D coordinate sequence of VO-001 is obtained from the robot motion controller or virtual object tracking system. The physical boundaries of the target segment are predefined, for example, the X-axis range of the physical screw hole [0.100m, 0.110m], the Y-axis range [0.200m, 0.210m], and the Z-axis range [0.300m, 0.305m]. At each time point, the current coordinate position of VO-001 (e.g., its center point) is compared with the target physical boundary. For example, if the X-coordinate of the center of VO-001 is 0.112 meters, then the X-axis offset is 0.112 - 0.110 = 0.002 meters, accurately identifying the offset of the virtual object edge from the target boundary in various directions. Finally, displacement alignment offset data is generated, which accurately describes the degree of misalignment of the virtual object relative to its target physical boundary in each direction, for example, {VO-001: {X_offset: +0.002m, Y_offset: -0.001m, Z_offset: +0.0005m}}.

[0102] S402: Based on the displacement alignment offset data, determine whether the virtual object is within the target range, analyze whether the difference between the bottom and top coordinates is completely within the boundary. If it is not within the boundary, adjust the slide rail driver through the offset direction signal until the offset is zero, and obtain the virtual object boundary positioning determination result.

[0103] Based on the displacement alignment offset data, for example, {X_offset: +0.002m, Y_offset: -0.001m, Z_offset: +0.0005m}, the system first determines whether the virtual object is completely within the physical boundary of the target segment. This determination is made by checking whether the offsets in all directions are within a preset minimum tolerance threshold (e.g., ±0.0001 meters). For example, since the X-axis offset of 0.002 meters exceeds the threshold of 0.0001 meters, it is determined that the virtual object is currently not within the target segment. Next, the system analyzes whether the difference between the bottom and top coordinates of the virtual object (i.e., the height of the virtual object) is completely within the physical boundary height of the target segment (e.g., 0.005 meters). For example, the height of the virtual screw is 0.0048 meters, and the height of the target hole is 0.005 meters, with a difference of 0.0002 meters. This indicates that the height of the virtual object meets the target range requirements, but it needs to be adjusted downwards as a whole. Since the virtual object is not completely within the boundary, an adjustment signal is generated based on the positive and negative directions of the offset (for example, a positive offset on the X-axis indicates that the virtual object is to the right, and a positive offset on the Z-axis indicates that the virtual object is too high). For example, for an offset of +0.002 meters on the X-axis, a control command to "move 2 millimeters to the left" is issued to the slide rail driver; for an offset of +0.0005 meters on the Z-axis, a control command to "move 0.5 millimeters down" is issued. The slide rail driver is a high-precision linear or rotary motion mechanism that precisely controls the position of the AR projection plane or physical display through a stepper motor or servo motor, continuously monitors the offset, and cyclically sends adjustment commands. For example, the driver performs an adjustment, updating the displacement data to {X_offset: +0.0001m, Y_offset: -0.00005m, Z_offset: +0.00002m}, and checks again. This adjustment process continues until the offsets in all directions converge to near zero (i.e., less than the tolerance threshold of 0.0001 meters). When all offsets meet this condition, the virtual object boundary positioning result is obtained and marked as "in place," for example, {VO-001: "in place"}.

[0104] S403: Based on the virtual object boundary positioning determination result, start the time recorder for the current coordinate position when the virtual object is in position, collect the holding time of the virtual object from the completion of the displacement to the stationary state, and record the slide rail number and coordinates. Store the time and number data together to generate an embedded record.

[0105] A time recorder is a recorder that tracks the time from when a virtual object completes its shift to when it comes to rest.

[0106] Based on the virtual object boundary positioning result, such as {VO-001: "Positioned"}, when the virtual object VO-001 boundary is detected to be precisely positioned, a millisecond-level time recorder is immediately started to record the current positioning time, for example, t=12.345 seconds. This time recorder continues to run until the virtual object remains stationary in the positioned state, for example, its coordinate change is less than 0.00001 meters within 100 consecutive milliseconds. The time it takes for the virtual object to remain stationary from the first determination of "positioned" to complete stillness is collected. For example, if it remains stationary between 12.345 seconds and 12.645 seconds, the holding time is 0.3 seconds. Simultaneously, record the unique identifier of the currently used slide rail drive (e.g., SR-001) and the precise 3D coordinates of the virtual object after final stable alignment (e.g., X=0.1050m, Y=0.2050m, Z=0.3025m). Store this data in association, for example: {Embedded ID: E001, Virtual Object ID: VO-001, Slide Rail ID: SR-001, Final Coordinates: (0.1050, 0.2050, 0.3025)m, Holding Time: 0.3s, Position Timestamp: 12.345s}. Finally, the embedded record is obtained.

[0107] Please see Figure 6 The specific steps for obtaining records in the virtual-real fusion processing are as follows:

[0108] S501: Based on the embedded record, extract the virtual object trigger timestamp and number information, identify the illumination consistency compensation status and timestamp, determine whether there is an adjustment status record in the window, and if so, mark the virtual object interference status and generate a compensation cross-interference mark record.

[0109] Based on the embedded record, such as {Embedded ID: E001, Virtual Object Number: VO-001, Arrival Timestamp: 12.345s}, extract the VO-001 number and its arrival trigger timestamp of 12.345 seconds. Simultaneously, read the operation log of the illumination consistency compensation module, which contains the illumination compensation status and its timestamp. Define a time window, for example, a window extending 1 second before and after the arrival timestamp, i.e., [11.345s, 13.345s]. Retrieve the adjustment status record of the illumination compensation module within this time window. For example, if the illumination compensation record shows brightness adjustment in progress between 12.000 seconds and 12.500 seconds... If any adjustment status record is detected in this window, the virtual object VO-001 is marked as having been subjected to cross-interference of illumination compensation during the embedding process, for example: {Interference ID: C001, Virtual object number: VO-001, Interference status: "Illumination compensation intervention", Interference timestamp: [12.000s, 12.500s]}. Finally, the compensation cross-interference mark record is output.

[0110] S502: Based on the compensation cross-interference mark record, extract the virtual object number of the interference section, detect the pause state after alignment is completed, record the end time of the invalid state, and if the compensation state is fully enabled, record the timestamp and bind the virtual object number to obtain the virtual object unlock timestamp.

[0111] Based on the compensation cross-interference marker record, such as {interference ID: C001, virtual object number: VO-001, interference status: "light compensation intervention", interference timestamp: [12.000s, 12.500s]}, extract the interfered virtual object number VO-001 from it, and at the same time check the pause status after the virtual object is aligned, for example, VO-001 remains still from 12.345 seconds to 12.645 seconds. If the interference state and the pause state overlap (e.g., the overlap time is from 12.345 seconds to 12.500 seconds), the overlap period is considered an invalid state, and its end time is recorded, for example, 12.500 seconds. The illumination compensation state is continuously monitored until it is fully turned on and stable (e.g., the illumination intensity reaches the target value and fluctuates by less than 1% within 100 milliseconds). Once the illumination compensation state is stable, the timestamp at this time is recorded, for example, 12.700 seconds, and it is bound to VO-001, for example, {VO-001_unlock time: 12.700s}. Finally, the virtual object unlock timestamp is obtained.

[0112] S503: Based on the virtual object unlock timestamp as the time base point, read the action record of the corresponding virtual object number, extract the compensation state and rendering path on / off state sequence, arrange them in time order, and output the virtual-real fusion processing record.

[0113] Using the virtual object's unlock timestamp as a time base point, for example, the unlock timestamp of VO-001 is 12.700 seconds, this timestamp is used as the starting base point for analysis. All relevant action records of the virtual object VO-001 are read from the log. The action records include the state changes of lighting compensation in the subsequent time (e.g., compensation intensity, color adjustment) and the on / off status of the virtual object's rendering path (e.g., whether the virtual object is occluded and interrupts rendering, or whether it is re-displayed due to re-rendering). Key compensation state sequences (e.g., starting at 12.700 seconds, the lighting compensation intensity remains at 600 Lux) and rendering path on / off state sequences (e.g., starting at 12.700 seconds, the rendering path of VO-001 remains "on", indicating visibility) are extracted from complex logs. The extracted states and events are then sorted in ascending order according to precise timestamps to form a complete time-series record, outputting a virtual-real fusion processing record containing the following entries: {Time: 12.700s, Object: VO-001, Event: Unlocked, Compensation State: Stable 600 Lux, Rendering Path: On}, {Time: 12.850s, Object: VO-001, Event: Render Update, Compensation State: Stable 600 Lux, Rendering Path: On}. This record details the entire process of a virtual object from being "unlocked" by lighting interference to its subsequent rendering and interaction under stable lighting conditions, providing comprehensive historical data for performance evaluation and fault diagnosis of the virtual-real fusion system.

[0114] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A real-time image scene virtual-real fusion processing method, characterized in that, Includes the following steps: S1: Acquire the sequence of scene geometric structure changes, illumination fluctuation time periods and motion trajectories captured by the multimodal sensor, count the number of geometric structure changes and abrupt change points of motion trajectory, perform position mapping with the scene response area, and generate virtual and real interference area delineation results; The specific steps for obtaining the virtual-real interference region delineation result are as follows: S101: Acquire standard geometric structure change sequence data captured by multimodal sensors, illumination fluctuation time period information, and time series trajectory of motion trajectory sensors, perform time alignment, count continuous change points, change amplitude, and number of change points, and generate the number of geometric structure changes. S102: Based on the number of changes in the geometric structure, calculate the first-order slope of the motion trajectory difference sequence within each fluctuation time period, match the time of change point with the trajectory change trend, extract trajectory abrupt change points, and generate trajectory abrupt change point distribution data. S103: Based on the trajectory mutation point distribution data, map the mutation point index to the response area number, and summarize the mutation point count by area to generate the virtual and real interference area delineation result; S2: Based on the results of the virtual and real interference region delineation, identify the lighting stable segments in the real scene, mark the lighting stable positions, and generate lighting stable segment data; The specific steps for acquiring data in the stable illumination region are as follows: S201: Based on the delineation results of the virtual and real interference areas, read the illumination direction and amplitude sequence, extract the time according to the fluctuation period, and combine the time with the motion trajectory change sequence to determine the illumination direction difference value and amplitude fluctuation, mark the illumination stable segment, and generate the illumination stable interval sequence. S202: Based on the light stability interval sequence, combined with the motion trajectory state signal and the light amplitude change sequence, locate the unclosed time period, and perform time-series statistics on the continuously decreasing light amplitude segment to generate the light stability absorption segment; S203: Based on the light-stable absorption section, spatial index conversion is performed using the spatial coordinate data of the light sensor to locate the scene coordinate section, map and mark the start and end times of the stable absorption section, and generate light-stable section data. S3: Based on the light-stable section data, collect the rotation angle sequence, embedding direction setting value and standard virtual object size range mapping relationship in the scene, filter virtual objects that can be directly embedded in the boundary range, and generate virtual-real fusion alignment range; The specific steps for obtaining the virtual-real fusion alignment interval are as follows: S301: After collecting the data of the stable illumination section, obtain the rotation angle sequence corresponding to each time node in the scene, calculate the critical embedding angle of the virtual object under the rotation angle at each time point, and analyze the embedding angle field of the virtual object at the differentiated radial position to obtain the embedding angle distribution data. S302: Based on the embedded angle distribution data, combined with the embedded direction and the size of the virtual object, cut the angle field according to the deflection force line direction, calculate the boundary value of the embedded path region, filter out the path segments without embedded structure, and generate the embedded path layered interval. S303: Based on the embedded path layered interval, compare the virtual object size with the standard tolerance range, extract the path boundary that meets the size conditions, count the continuous length and number, mark the embeddable path and generate a number index table, and output the virtual-real fusion alignment interval. S4: Based on the virtual-real fusion alignment interval, determine whether the virtual object embedding position enters the target boundary range. If it is not within the range, perform the alignment adjustment operation and output the embedding record.

2. The real-time image scene virtual-real fusion processing method according to claim 1, characterized in that, The results of defining the virtual and real interference regions include the distribution of the number of geometric structure changes, the location of the abrupt change point of the motion trajectory, and the mapping relationship of the interference regions. The data of the stable illumination segment includes information on the stable illumination direction segment, the fluctuation decay time, and the location of the stable illumination segment. The virtual and real fusion alignment interval includes the rotation angle interval, the embedding direction interval, and the qualified embedding boundary. The embedding record includes the virtual object embedding position, the alignment direction status, and the alignment holding time.

3. The real-time image scene virtual-real fusion processing method according to claim 1, characterized in that, The rotation angle sequence within the scene refers to the critical embedding angle sequence of virtual objects calculated according to the scene rotation characteristics. The boundary range refers to the acceptable range of standard virtual object size tolerance ±3%.

4. The real-time image scene virtual-real fusion processing method according to claim 1, characterized in that, The specific steps for obtaining the embedded record are as follows: S401: Based on the embedded path hierarchical interval, obtain the corresponding virtual object number, read the displacement record, compare the coordinate position with the physical boundary of the target segment, identify the offset between the virtual object edge and the boundary, and generate displacement alignment offset data; S402: Based on the displacement alignment offset data, determine whether the virtual object is within the target range, analyze whether the difference between the bottom and top coordinates is completely within the boundary. If it is not within the boundary, adjust the slide rail driver through the offset direction signal until the offset is zero, and obtain the virtual object boundary positioning determination result. S403: Based on the virtual object boundary positioning determination result, start the time recorder for the current coordinate position when the virtual object is in position, collect the holding time of the virtual object from the completion of the displacement to the stationary state, and record the slide rail number and coordinates, and store the time and number data together to generate an embedded record.

5. The real-time image scene virtual-real fusion processing method according to claim 4, characterized in that, The displacement alignment offset data refers to the offset between the virtual object and the physical boundary of the target segment; The time recorder refers to the time it takes for a virtual object to go from the completion of its shift to its stationary state.

6. The real-time image scene virtual-real fusion processing method according to claim 1, characterized in that, The method also includes step S5: S5: Based on the embedded record, identify the intersection of the illumination consistency compensation state and the spatiotemporal anti-aliasing trigger time, record the compensation adjustment, anti-aliasing action and rendering path state stage information, and obtain the virtual-real fusion processing record; The virtual-real fusion processing record includes a compensation adjustment time series, an anti-aliasing action time series, and a rendering path state sequence.

7. The real-time image scene virtual-real fusion processing method according to claim 6, characterized in that, The specific steps for obtaining the virtual-real fusion processing record are as follows: S501: Based on the embedded record, extract the virtual object trigger timestamp and number information, identify the illumination consistency compensation status and timestamp, determine whether there is an adjustment status record in the window, and if so, mark the virtual object interference status and generate a compensation cross-interference mark record. S502: Based on the compensation cross-interference mark record, extract the virtual object number of the interference section, detect the pause state after alignment is completed, record the end time of the invalid state, and if the compensation state is fully enabled, record the timestamp and bind the virtual object number to obtain the virtual object unlock timestamp. S503: Based on the unlock timestamp of the virtual object as the time base point, read the action record of the corresponding virtual object number, extract the sequence of compensation state and rendering path on / off state, arrange them in time order, and output the virtual-real fusion processing record.

Images