An image processing method for AR / VR active alignment

Abstract

The present application belongs to the technical field of image processing, and particularly relates to an image processing method for AR / VR active alignment, which comprises the following steps: acquiring an optical acquisition image containing a composite test image, wherein the RGB three channels respectively carry a high-frequency Siemens star for detecting MTF and optical axis tilt, a sparse spot grid for detecting distortion and field of view angle, and a gray color block for detecting brightness uniformity and color deviation; based on pre-calibrated spectral geometric parameters, the image is subjected to channel decomposition and spatial correction to acquire independent feature spectrum; multi-dimensional optical parameters are solved in parallel and a global quality score is constructed; finally, a six-axis robot is driven to perform closed-loop iterative adjustment based on the score until the optimal pose is reached. Through spectral multiplexing and spatial encoding technology, the present application realizes single-frame full-parameter detection, avoiding the problems of low efficiency and mechanical vibration affecting precision in traditional serial detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing. More particularly, the present application relates to an image processing method for AR / VR active alignment. BACKGROUND

[0002] In the assembly production of AR / VR devices, in order to ensure the final imaging quality, it is usually necessary to use a six-axis robot to clamp the optical engine module, and to adjust its spatial pose relative to the optical element in real time according to the imaging quality feedback by the image sensor until the optimal state is reached.

[0003] In order to achieve the above-mentioned precise adjustment, the prior art usually adopts a feedback control algorithm based on image analysis. The main means is to adopt a serial detection mode, that is, the optical engine to be tested projects different functional special test patterns in turn in sequence, for example, first projects a black and white stripe pattern to calculate the sharpness by using the modulation transfer function algorithm, then switches to project a checkerboard pattern to calculate the geometric distortion, and then projects a pure color picture to detect the brightness uniformity and color deviation. The system constructs an evaluation function based on these step-by-step acquired optical indicators, and drives the robot to perform iterative adjustment.

[0004] However, this serial step-by-step image processing method has significant defects in actual application: on the one hand, the switching and stabilization between different test patterns require a long buffer time, resulting in a long time consumption for a single alignment, which limits the production rhythm; on the other hand, in the process of multiple step-by-step acquisition, the mechanical platform inevitably has slight vibration, which causes the key geometric features such as the center of sharpness and the center of distortion acquired at different times to be unable to be accurately aligned in the same coordinate system, thereby reducing the accuracy of multi-dimensional index fusion calculation and the final binocular image precision. SUMMARY

[0005] To solve the technical problem of poor active alignment effect, the present application provides an image processing method for AR / VR active alignment, comprising:

[0006] An optical acquisition image containing a composite test pattern is acquired, the composite test pattern containing RGB projection source patterns respectively carrying different test patterns in three orthogonal color channels R, G and B, wherein the R channel carries a high-frequency Siemens star pattern for detecting MTF and optical axis tilt, the G channel carries a sparse spot grid for detecting distortion and field of view angle, and the B channel carries a regional step grayscale color block for detecting brightness uniformity and color deviation; channel decomposition and spatial correction are performed based on the optical acquisition image and pre-labeled spectral geometric parameters to obtain independent feature maps of the RGB three channels, wherein the correction includes spectral unmixing using a pre-labeled color crosstalk matrix and geometric alignment according to a pre-set color difference model; optical and mechanical tilt angles, modulation transfer functions, geometric distortions, field of view angles, brightness uniformities and color deviations are respectively calculated based on the independent feature maps of the RGB three channels in parallel; and a global quality score is constructed based on the calculated multi-dimensional optical parameters, and a six-axis robot is driven to perform closed-loop iterative adjustment until an optimal imaging pose is reached.

[0007] The present application realizes single-frame shooting and full-parameter coverage by respectively carrying MTF detection, distortion detection and color detection patterns in the RGB three orthogonal color channels. This spectral multiplexing and parallel calculation technology not only eliminates the buffering time required for switching test patterns in the traditional method, improves the production rhythm, but also avoids the problem of inconsistent MTF center and distortion center coordinates caused by mechanical micro-vibration during multiple acquisitions, ensuring the accurate fusion and closed-loop control of multi-dimensional optical indicators at the same physical time and in the same coordinate system.

[0008] Preferably, the method for constructing the composite test pattern further comprises:

[0009] A spatial code is introduced at a specific position of the sparse spot grid, and a standard spot is replaced by a pre-set special topological pattern as an absolute field of view anchor.

[0010] Preferably, the calculation of the optical and mechanical tilt angle comprises:

[0011] The blur kernel profile of the high-frequency Siemens star pattern is extracted, ellipse fitting is performed on the blur kernel profile to obtain the major axis, minor axis and rotation angle of the fitted ellipse, and the amplitude of the optical and mechanical tilt angle is directly proportional to the natural logarithm of the ratio of the major and minor axes of the fitted ellipse, and the direction angle is orthogonal to the rotation angle of the major axis of the fitted ellipse.

[0012] The present application uses the astigmatism principle of the Siemens star pattern in the defocus state, directly linearly maps the blur degree of the image to the physical tilt angle and direction of the optical and mechanical system by fitting the ellipse characteristics of the blur kernel profile, simplifies the detection process and improves the alignment efficiency.

[0013] Preferably, the calculation of the modulation transfer function comprises:

[0014] A preset target spatial frequency is set, a sampling circle corresponding to the target spatial frequency is constructed, a luminance sequence is extracted along the sampling circle, the luminance sequence is divided into multiple independent modulation units, and an average modulation contrast of all modulation units is calculated as the modulation transfer function.

[0015] The application can reflect the real performance of the optical system under the limit resolution by constructing the sampling circle corresponding to the sensor half-Nyquist frequency on the Siemens star and calculating the contrast by dividing the luminance sequence into independent modulation units, and provides a high-sensitivity definition feedback index for active alignment.

[0016] Preferably, the calculation of the geometric distortion comprises:

[0017] The centroid coordinates of all the sparse spot grids are detected, a physical coordinate system is established based on the absolute field of view anchor points, and the actual imaging height of the 1.0 field of view anchor point is obtained; the geometric distortion is equal to the normalized deviation of the actual imaging height and the theoretical imaging height.

[0018] The application compares the actual imaging height of the spot with the theoretical design height in the physical coordinate system established by the absolute field of view anchor points, can calculate the normalized geometric distortion rate, and enables the system to distinguish the inherent distortion of the optical system from the assembly error, thereby providing accurate geometric performance data for subsequent weighted scoring.

[0019] Preferably, the acquisition of the pre-calibrated color crosstalk matrix comprises:

[0020] The to-be-tested light machine is controlled to project a pure red picture, a pure green picture and a pure blue picture in turn, the binocular bionic camera is used to collect images corresponding to each pure color picture, and the average gray scale response value of the central region of the image on the original RGB channel of the camera is counted; and the three-channel response value corresponding to each pure color picture is normalized and taken as a column of the color crosstalk matrix.

[0021] Preferably, the geometric alignment according to the preset chromatic aberration model comprises:

[0022] The independent feature spectrum of the G channel is taken as a geometric reference, radial scaling is performed on the independent feature spectrum of the R channel and the B channel, and the radial scaling satisfies the expression: wherein, is the distance of the corrected pixel point to the center, is the distance before correction, , is a second-order radial distortion correction coefficient, and is a fourth-order radial distortion correction coefficient.

[0023] Preferably, the establishment of the global quality score comprises:

[0024] The modulation transfer function is taken as a forward gain term, and the optical machine tilt angle, geometric distortion and brightness non-uniformity are taken as negative punishment terms, and linear weighted summation is performed.

[0025] Preferably, the global quality score satisfies the expression:

[0026] ;

[0027] In the formula, Q represents the global quality score under the current optical machine pose; MTF represents the normalized modulation transfer function; Tilt represents the normalized optical machine tilt angle; TV represents the normalized TV distortion rate; Uniformity represents the normalized brightness uniformity; 、 、 、 respectively represent 、 、 、 weight coefficients of the terms; norm represents the norm of a vector; abs represents the absolute value symbol.

[0028] Preferably, the driving six-axis robot performs closed-loop iterative adjustment, including:

[0029] calculating partial derivatives of the global quality score with respect to six motion degrees of freedom of the six-axis robot, and adaptively constructing a gain matrix to form a pose adjustment vector; and controlling the six-axis robot to perform fine adjustment in the direction and step length of the pose adjustment vector; the adaptive gain matrix is composed of a position step length coefficient and an attitude step length coefficient.

[0030] The present application has the beneficial effects that: the present application simultaneously acquires multi-dimensional optical indicators in a single exposure by respectively carrying MTF, distortion and color test patterns in RGB three channels, and combining channel demixing and geometric correction technology, overcomes the problems of low efficiency of traditional serial detection and non-uniformity of multi-indicators caused by mechanical vibration, and improves the efficiency and precision of AR / VR optical machine active alignment by constructing a global quality score to drive the robot in a closed loop. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is a flowchart schematically showing an image processing method for AR / VR active alignment in the present application;

[0032] Figure 2 is an optical image acquisition;

[0033] Figure 3is a schematic diagram showing the iterative change curve of the global quality score. DETAILED DESCRIPTION

[0034] The embodiment of the application discloses an image processing method for AR / VR active alignment, referring to Figure 1 , comprising steps S1-S4:

[0035] S1: based on the multi-dimensional detection requirements of the active alignment process and the spectral characteristics of the display unit, a full-parameter composite test image is constructed, and the light machine is controlled to project to obtain an optical acquisition image.

[0036] It should be noted that in the AA (Active Alignment) process of the AR / VR light machine module, the six-axis robot needs to adjust the light machine pose in real time according to the imaging quality. The traditional detection scheme adopts a serial mode, and a plurality of single-function patterns such as MTF stripe patterns for detecting clarity, checkerboard patterns for detecting distortion, RGB color block patterns for detecting color, etc. are projected in turn. In this way, a certain buffer time is required for switching between different patterns, usually 15 to 30 seconds, which limits the production rhythm; and mechanical micro-vibration inevitably exists in the multiple acquisition processes, which will cause the key geometric features such as MTF center and distortion center to be unable to be accurately aligned in the same coordinate system, affecting the accuracy of binocular image. Therefore, the independent driving capability of the RGB channel of the display unit is used, and through spectral multiplexing and spatial coding technology, the features of all detection schemes are fused into a single frame image, realizing one-time projection and full-parameter coverage.

[0037] Specifically, based on the multi-dimensional detection requirements of the active alignment process and the spectral characteristics of the display unit, a full-parameter composite test image is constructed, and the light machine is controlled to project to obtain an optical acquisition image, comprising:

[0038] An RGB projection source image is generated, and the three orthogonal color channels thereof respectively carry different frequency domain characteristic test patterns, the R channel is a high-frequency Siemens star carrying MTF (Modulation Transfer Function) and Tilt information (Optical Axis Tilt); the G channel is a sparse spot grid carrying distortion and FOV information (Field of View); and the B channel is a regional step grayscale color block distributed in the gap between the R channel pattern and the G channel pattern, which is used to detect the brightness uniformity and color deviation of the full field of view, and avoid background interference on high-frequency feature extraction.

[0039] Preferably, spatial coding is introduced at specific positions of the sparse spot grid, such as the center point, 0.5 field of view and 1.0 field of view, and a standard circle is replaced by a special topological pattern such as a cross, which is used as an absolute field of view anchor point, replacing the physical multi-field of view calibration process.

[0040] It should be noted that the high-frequency Siemens star is composed of black and white stripes radiating outward from the center, contains omnidirectional high-frequency edge information, and is sensitive to defocus height, and is used to realize static Tilt detection.

[0041] The light machine to be tested is controlled to be turned on to project the RGB projection source image; after the light is transmitted through the optical system, the optical acquisition image is obtained by the binocular bionic camera. The optical system includes a lens and an optical waveguide.

[0042] It should be noted that, as Figure 2 For the optical acquisition image, the image is obtained by imaging the full-parameter composite test image projected by the light machine to be tested after being transmitted through the optical system. The image contains mixed information of the RGB three channels, and is directly presented as the superposition state of the Siemens star, the sparse spot grid and the area color block.

[0043] At this point, the optical acquisition image containing full-dimensional optical test information is obtained.

[0044] S2: based on the optical acquisition image and the pre-marked spectral geometric parameters, performing channel decomposition and spatial correction to obtain independent feature spectra of the RGB three channels.

[0045] It should be noted that since the AR / VR optical system such as the Pancake folding light path generally has magnification chromatic aberration, the magnification of light of different wavelengths on the imaging plane is different; and the display screen pixels have spectral leakage, that is, when the green pixels emit light, a small amount of red light component will be accompanied. Therefore, directly separating the RGB channels will cause the red MTF texture and the green distortion point to be misaligned in space, and the signal is not pure, which affects the algorithm accuracy. Therefore, before extracting the features, the spectral geometric mapping model is established, the spectral crosstalk is removed through matrix transformation, and then the geometric coordinates are aligned through the radial scaling model. Based on this, the spectral geometric decomposition model is constructed.

[0046] Specifically, based on the optical acquisition image and the pre-marked spectral geometric parameters, channel decomposition and spatial correction are performed to obtain independent feature spectra, including:

[0047] Perform spectral unmixing to obtain a pure spectral signal.

[0048] It should be noted that in order to restore the true channel signal from the mixed sensor response, the inverse transformation of the color correction matrix needs to be applied, and the RGB value collected by the camera sensor is the convolution result of the display screen spectrum and the Bayer filter response, and there is aliasing. Assuming that the signal aliasing is linear, the mixed signal can be unblended into independent signals through the inverse matrix of the pre-marked color crosstalk matrix.

[0049] Let the optical acquisition image be , , , , is a RGB three-channel pixel value vector of , is a transpose symbol.

[0050] The light machine to be tested projects a pure red picture , a pure green picture , and a pure blue picture in turn. For each projection, the binocular bionic camera is used to collect images, and the average gray scale response value of the image center area on the camera original RGB channel is counted. A color crosstalk matrix is constructed, the first column of which is the normalized response value of the camera RGB channel when the pure red picture is projected , the second column is the normalized response value when the pure green picture is projected , and the third column is the normalized response value when the pure blue picture is projected , . It should be noted that the color crosstalk matrix is used to eliminate signal leakage between channels.

[0051] The spectral unmixing satisfies the expression:

[0052] ;

[0053] In the formula, denotes the optical unmixing image; denotes the color crosstalk matrix; denotes a dot product symbol; denotes the optical acquisition image.

[0054] In the formula, by linear algebraic transformation, mathematically eliminates physical crosstalk such as display screen red light exciting the camera green pixel, ensuring that the subsequent calculation of MTF in the red channel will not be disturbed by the background of the green channel distortion grid pattern.

[0055] Let be denoted as , , , is a RGB three-channel pixel value vector of , and for , , LCA geometric correction is performed to obtain pure and aligned RGB three-channel independent feature maps, which are denoted as , , respectively. For example, the green channel For geometric reference, the red and blue channels are radially scaled so that their geometric features are spatially aligned with the green channel, where is the distance from the center of the corrected pixel, is the distance before correction, , is the second-order radial distortion correction coefficient, the fourth-order radial distortion correction coefficient, and is exemplary is set to , indicating that the correction needs to be contracted, is set to , indicating that the edge high-order fine-tuning is needed.

[0056] At this point, three spatially aligned and spectrally pure independent feature maps , , are obtained.

[0057] S3: Based on the independent feature maps of the RGB three channels, astigmatism analysis, absolute coordinate addressing and photometric statistics are performed respectively to obtain multi-dimensional optical parameters such as optical machine tilt angle, distortion rate and brightness uniformity.

[0058] It should be noted that in the active alignment process of the AR / VR optical machine module, the traditional parameter solving method often relies on step-by-step physical measurement. For example, detecting the optical machine tilt relies on mechanical scanning of a physical parallel light tube or multi-frame defocus image acquisition. This way not only takes time, but also cannot meet the production rhythm; at the same time, due to the large field of view characteristics of the AR / VR optical machine, the distortion measurement of the edge field of view often leads to calculation deviation due to the inability to find an accurate center reference. The present application uses a parallel computing strategy based on the physical properties of the independent feature maps of the RGB three channels. Using the high-frequency Siemens star of the red channel, based on the principle of optical astigmatism, when the optical system has a tilt, the meridian and sagittal rays of the off-axis field of view are separated in focus, causing the circular blur spot at the center of the star to degenerate into an ellipse, thereby realizing single-frame static tilt detection; using the absolute field of view anchor point of the green channel, an absolute physical coordinate system with zero geometric error is established; using the regional gray scale information of the blue channel, the photometric properties of the system are evaluated.

[0059] Specifically, based on the independent feature map of the red channel, the optical machine tilt angle and the modulation transfer function are solved, including:

[0060] Edge detection is performed on to extract the blur kernel profile of the high-frequency Siemens star at the center of the image; least squares method is performed on the blur kernel profile to perform ellipse fitting to obtain the long axis and short axis parameters of the fitted ellipse. Exemplarily, the edge detection uses a canny operator.

[0061] It should be noted that the accuracy of the optical machine tilt angle calculation is highly dependent on the tilt sensitivity coefficient, which is determined by the inherent physical quantity of the F number and aberration characteristics of the lens.

[0062] The tilt sensitivity coefficient is obtained in advance through the optical simulation calibration process: based on the optical design file of the to-be-tested optical machine, such as Zemax or CodeV data, an ideal optical simulation model is constructed; a discrete known tilt angle variable is introduced in the model, for example, set from 0° to 2.0° with a step of 0.1°; for each tilt angle, the point spread function generated at the center of the imaging surface is simulated, and the length-to-short axis ratio of the PSF spot is measured; the natural logarithmic function value of the length-to-short axis ratio is taken as the abscissa, and the known tilt angle is taken as the ordinate; linear regression analysis is performed, the slope of the regression straight line is extracted, and the slope is defined as the tilt sensitivity coefficient.

[0063] It should be noted that when the optical machine optical axis is perpendicular to the imaging plane, the defocus blur spot is a regular circle; when the optical axis is tilted, astigmatism causes the blur spot to stretch into an ellipse. Considering that the length-to-short axis ratio of the ellipse is positively correlated with the amplitude of the tilt angle, and the rotation angle of the long axis of the ellipse is orthogonal to the direction angle of the tilt, the present application establishes an expression for calculating the tilt angle vector of the optical machine.

[0064] The tilt angle of the optical machine satisfies the expression:

[0065] ;

[0066] In the formula, represents the tilt angle of the optical machine; represents the tilt amplitude of the optical axis of the optical machine relative to the ideal normal line; 、 represents the long axis and short axis of the fitted ellipse; represents the direction azimuth of the tilt; represents the rotation angle of the long axis of the fitted ellipse relative to the horizontal axis of the image; represents the tilt sensitivity coefficient; represents the natural logarithmic function.

[0067] In the formula, linearly maps the astigmatism degree to a physical angle, ensuring that when the physical fact that the tilt angle is 0; Single-frame static tilt detection is realized through the astigmatism characteristics, without the need for mechanical scanning.

[0068] It should be noted that the Siemens star chart features alternating black and white stripes radiating outwards from the center. The physical width of the stripes on the circumference increases with the radius, implying that the spatial frequency decreases as the radius increases. Therefore, each radius value on the image corresponds to a specific spatial frequency. To evaluate the performance of the optical system at the limit of resolution, it is necessary to accurately locate the specific radius position corresponding to the target spatial frequency.

[0069] Based on the physical properties of the camera sensor, the sampling radius is calculated as follows: multiply the target spatial frequency by twice pi (referred to as the first product), and divide the total number of black and white fringe pairs in the Siemens star map by the first product to obtain the sampling radius. The target spatial frequency is the half-Nyquist frequency of the sensor.

[0070] A circular brightness sequence is constructed with the center of the Siemens star map as the center and the sampling radius as the radius. The circular brightness sequence is divided into K independent modulation units using the zero-crossing detection method. Each modulation unit contains a complete "bright-dark" change cycle.

[0071] The modulation transfer function satisfies the expression:

[0072] ;

[0073] In the formula, Represents the modulation transfer function; This represents the total number of complete modulation units identified from the circumferential brightness sequence; , The values ​​represent the peak and trough values ​​of the luminance waveform within the i-th modulation unit.

[0074] Thus, the optomechanical tilt angle and modulation transfer function were obtained.

[0075] Preferably, based on the green channel independent feature map, the geometric distortion and field of view are calculated, including:

[0076] right The sparse speckled mesh in the image is binarized and connected component analysis is performed to identify the centroid coordinates of all specks.

[0077] Perform absolute coordinate addressing and use a template matching algorithm to search for absolute field-of-view anchor points in the sparse speckle mesh; set the centroid coordinates of the identified anchor points at their center positions as follows. The physical zero point of the pixel coordinate system; obtain the anchor point position at the 1.0 field of view, denoted as the field of view angle.

[0078] It should be noted that distortion reflects the degree of deviation between the actual imaging height and the ideal optical imaging height.

[0079] Therefore, the TV distortion rate satisfies the expression:

[0080] ;

[0081] In the formula, represents the distortion rate; represents the Euclidean distance from the physical zero point to the field anchor point mass center located at 1.0 field of view in the green channel independent feature map; represents the theoretical imaging height.

[0082] In the formula, represents the normalized relative deviation ratio of the actual imaging height relative to the ideal imaging height; represents the conversion of the relative deviation ratio into a percentage.

[0083] It should be noted that, The acquisition of the design effective focal length in the specification of the light machine to be measured, the ratio of the pixel size of the image sensor, and the tangent function value of the field of view angle are multiplied to obtain the theoretical imaging height.

[0084] At this point, the distortion rate and the field of view angle are obtained.

[0085] Preferably, based on the blue channel independent feature map, the luminance uniformity and the color deviation are calculated, including:

[0086] Through the preset binary mask, the step grayscale area distributed in the gap in The average grayscale value of each step color block is counted, and the luminance uniformity of the full field of view is calculated. At the same time, the difference with the standard color card value is calculated to calculate the RGB color deviation.

[0087] It should be noted that, since the B channel pattern is located between the R / G channel gap, combined with spectral unmixing, the color data calculated here is not disturbed by high-frequency stripes, ensuring the accuracy of color calibration.

[0088] At this point, the luminance uniformity and the color deviation are obtained.

[0089] S4: Based on the multi-dimensional optical parameters, a global weighted evaluation function is constructed to drive the six-axis robot to perform closed-loop iterative adjustment until the optimal imaging pose is reached.

[0090] It should be noted that in the AA process of the AR / VR light machine module, there is often physical coupling and constraint between various optical indicators. For example, excessive pursuit of the clarity of the central field of view may lead to a sharp increase in the astigmatism of the edge field of view, or the luminance uniformity is sacrificed in order to correct the geometric distortion. Therefore, the traditional single indicator feedback regulation is easy to lead the system into a local optimal solution. The present application introduces a multi-objective fusion strategy, which normalizes the multi-dimensional parameters into a comprehensive score to guide the robot to find the global optimal balance point.

[0091] Specifically, based on the multi-dimensional optical parameters, a global weighted evaluation function is constructed, and a six-axis robot is driven to perform closed-loop iterative adjustment, including:

[0092] It should be noted that for an ideal optical system, the imaging quality is determined by the "positive gain attribute" and the "negative loss attribute". Among them, the modulation transfer function represents the sharpness, which belongs to the positive attribute that should be maximized; and the optical machine tilt, distortion and brightness non-uniformity represent aberration and defects, which belong to the negative attribute that should be minimized. In order to convert this multi-objective optimization problem into a single-objective optimization task executable by the six-axis robot, a linear superposition model is needed to be established, taking the positive attribute as the plus item and the negative attribute as the minus item. Considering that different optical products have different sensitivities to indicators, for example, VR devices prefer to guarantee central sharpness, while waveguide AR devices pay more attention to geometric distortion, therefore, weight coefficients are introduced to define the relative importance of each physical quantity in the evaluation system by adjusting the size of the weight.

[0093] Let the current optical machine pose be denoted as , representing the spatial three-dimensional coordinates and three-dimensional rotation angles maintained by the six-axis robot at the current time.

[0094] The global quality score satisfies the expression:

[0095] ;

[0096] In the formula, represents the global quality score under the current optical machine pose; represents the normalized modulation transfer function; represents the normalized optical machine tilt angle; represents the normalized TV distortion rate; represents the normalized brightness uniformity; , , , respectively represent , , , weight coefficients of the items; represents the modulus of the vector; represents the absolute value symbol. It should be noted that is a positive gain item, is a negative penalty item, is a negative penalty item, is a negative penalty item.

[0097] In the formula, , , , All are greater than 0, according to the sensitivity of each index in the product specification, for example, for the VR display device which emphasizes the central picture, considering its higher requirements for clarity and parallel to the optical axis, and the characteristics that distortion can be corrected by software algorithm, is set to , is set to , is set to 0.1, is set to 0.1.

[0098] It should be noted that the six degrees of freedom of the six-axis robot and the global quality score form a high-dimensional nonlinear surface. The goal of the AA process is to find the global highest point on this surface. In order to efficiently locate this highest point and avoid blind random exploration, the present application uses the gradient ascent method to calculate the partial derivative with respect to each motion degree of freedom. The physical meaning of this partial derivative is sensitivity, which indicates the direction and distance that can cause the fastest growth of the score. Based on this sensitivity, the pose adjustment instruction containing direction and step length is constructed.

[0099] The pose adjustment vector satisfies the expression:

[0100] ;

[0101] ;

[0102] In the formula, represents the single-step feed vector of the six-axis robot; represents the adaptive gain matrix; represents the transpose symbol; represents the diagonal matrix; represents the position step coefficient, represents the attitude step coefficient.

[0103] It should be noted that, contains the adjustment amount of the spatial position and the rotation angle ; the adaptive gain matrix is used to control the adjustment step, taking a large value at the beginning of the iteration to quickly converge, and taking a small value at the later stage to avoid oscillation; is used to adjust the convergence speed of the spatial displacement , is used to adjust the convergence speed of the rotation angle . By respectively setting and , the adjustment amplitude of translation and rotation can be targetedly balanced, the convergence problem caused by the non-uniformity of dimensions is eliminated, and the robot is ensured to smoothly approach the optimal pose in the six-dimensional space.

[0104] wherein, represents the global quality score, each item represents the rate of change of the optical quality score when the robot produces a small displacement in the corresponding degree of freedom, indicating the direction of motion that can make the fastest growth.

[0105] Perform closed-loop iteration and termination judgment: judge whether the global quality score under the current optical-mechanical pose meets the termination condition, if wherein, is a preset target threshold, or the score change amount of three consecutive iterations wherein, is a convergence threshold, then the AA process is successful; if the termination condition is not met, then perform pose adjustment, control the six-axis robot to move according to perform micro-motion adjustment; after the adjustment is completed, return to perform re-projection and image acquisition, then sequentially obtain the RGB independent feature map and the multi-dimensional optical parameter, recalculate and perform the next round of judgment.

[0106] It should be noted that, as Figure 3 is the iterative change curve of the global quality score, the horizontal axis represents the number of adjustment iterations, and the vertical axis represents the normalized comprehensive score. The curve shows a clear upward trend, indicating that the pose adjustment vector calculated by the six-axis robot according to the gradient ascent method continuously corrects the optical-mechanical position, maximizes the positive gain attribute and minimizes the negative loss attribute, and finally converges to the global optimal balance point.

[0107] At this point, the active alignment and detection of the optical-mechanical module based on the full-parameter composite test map are completed.

[0108] Although the present specification has shown and described several embodiments of the present application, it will be apparent to those skilled in the art that such embodiments are provided only by way of example. Those skilled in the art will think of many changes, changes and alternatives without departing from the idea and spirit of the present application.

Claims

1. An image processing method for active alignment in AR / VR, characterized in that, include: Acquire an optical image containing a composite test pattern, which includes RGB projection source images carrying different test patterns in three orthogonal color channels: R, G, and B. The R channel carries a high-frequency Siemens star pattern for detecting MTF and optical axis tilt, the G channel carries a sparse speckle grid for detecting distortion and field of view, and the B channel carries regional stepped grayscale patches for detecting brightness uniformity and color deviation. Based on the optically acquired image and pre-calibrated spectral geometric parameters, channel decomposition and spatial correction are performed to obtain independent feature maps of the RGB three channels. The correction includes spectral demixing using a pre-calibrated color crosstalk matrix and geometric alignment according to a preset color difference model. The system calculates the optomechanical tilt angle, modulation transfer function, geometric distortion, field of view, brightness uniformity, and color deviation in parallel based on the independent feature maps of the RGB three channels. It also constructs a global quality score based on the calculated multidimensional optical parameters and drives a six-axis robot to perform closed-loop iterative adjustments until the optimal imaging pose is achieved.

2. The image processing method for active alignment in AR / VR according to claim 1, characterized in that, The method for constructing the composite test chart further includes: Spatial coding is introduced at specific locations in a sparse speckle grid, replacing standard specks with preset special topological graphics as absolute field-of-view anchor points.

3. The image processing method for active alignment in AR / VR according to claim 1, characterized in that, The calculation of the optical-mechanical tilt angle includes: Extract the blurred kernel contour of the high-frequency Siemens star map; perform ellipse fitting on the blurred kernel contour to obtain the major axis, minor axis and rotation angle of the fitted ellipse; the amplitude of the optomechanical tilt angle is proportional to the natural logarithm of the ratio of the major axis to the minor axis of the fitted ellipse, and its direction angle is orthogonal to the rotation angle of the major axis of the fitted ellipse.

4. The image processing method for active alignment in AR / VR according to claim 1, characterized in that, The solution of the modulation transfer function includes: A target spatial frequency is preset, and a sampling circle corresponding to the target spatial frequency is constructed; a brightness sequence is extracted along the sampling circle; the brightness sequence is divided into multiple independent modulation units, and the average modulation contrast of all modulation units is calculated as the modulation transfer function.

5. The image processing method for active alignment in AR / VR according to claim 2, characterized in that, The solution to the geometric distortion includes: The centroid coordinates of all spots in the sparse speckle grid are detected; a physical coordinate system is established based on the absolute field of view anchor point, and the actual imaging height of the 1.0 field of view anchor point is obtained; the geometric distortion is equal to the normalized deviation between the actual imaging height and the theoretical imaging height.

6. The image processing method for active alignment in AR / VR according to claim 1, characterized in that, The acquisition of the pre-calibrated color crosstalk matrix includes: The optical engine under test is controlled to project pure red, pure green and pure blue images in sequence; a binocular bionic camera is used to acquire images corresponding to each pure color image, and the average grayscale response value of the central region of the image on the camera's original RGB channels is calculated; the three-channel response values ​​corresponding to each pure color image are normalized and used as a column of the color crosstalk matrix.

7. The image processing method for active alignment in AR / VR according to claim 1, characterized in that, The geometric alignment based on the preset color difference model includes: Using the independent feature map of channel G as the geometric reference, radial scaling is performed on the independent feature maps of channels R and B, wherein the radial scaling satisfies the expression: ,in, The distance from the center of the corrected pixel. To correct the distance before, , These are the second-order radial distortion correction coefficients and the fourth-order radial distortion correction coefficients.

8. The image processing method for active alignment in AR / VR according to claim 1, characterized in that, The establishment of the global quality score includes: The modulation transfer function is used as a positive gain term, and the optomechanical tilt angle, geometric distortion, and brightness non-uniformity are used as negative penalty terms, and then a linear weighted sum is performed.

9. The image processing method for active alignment in AR / VR according to claim 1, characterized in that, The global quality score satisfies the expression: ； In the formula, This represents the global quality score under the current optomechanical pose. Represents the normalized modulation transfer function; Indicates the normalized optical engine tilt angle; This represents the normalized TV distortion rate; Indicates normalized brightness uniformity; , , , They represent , , , The weight coefficient of the item; The sign representing the magnitude of a vector; Represents the absolute value symbol.

10. The image processing method for active alignment in AR / VR according to claim 1, characterized in that, The drive for the six-axis robot to perform closed-loop iterative adjustments includes: Calculate the partial derivatives of the global quality score with respect to the six degrees of freedom of the six-axis robot, and construct the pose adjustment vector using an adaptive gain matrix; control the six-axis robot to make micro-adjustments according to the direction and step size of the pose adjustment vector; the adaptive gain matrix consists of position step size coefficients and attitude step size coefficients.

Images