Method and system for automatically correcting light path of helmet-mounted display

Abstract

The invention discloses a method and system for automatically correcting a light path of a helmet-mounted display, and belongs to the technical field of image correction in the field of helmet-mounted displayers, and the method comprises the steps: determining a standard helmet-mounted display and a standard projection image, and obtaining a standard reflection image of an outer mask in the standard helmet-mounted display; acquiring a first distortion parameter of the standard helmet-mounted display based on the standard projection image and the standard reflection image; determining input data through the standard projection image and the standard reflection image, determining output data through the first distortion parameter, and performing model training to obtain a distortion model; determining a to-be-detected helmet-mounted display, and acquiring a reflection image of an outer mask in the to-be-detected helmet-mounted display as a to-be-detected reflection image; inputting the standard projection image and the to-be-measured reflection image into a distortion model to obtain a second distortion parameter; and correcting a to-be-projected image of the HUD in the to-be-tested helmet-mounted display based on the second distortion parameter.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image correction technology in the field of helmet display, and particularly relates to a method and system for automatically correcting the light path of a helmet display. BACKGROUND

[0002] A helmet display is a new type of image display system. In a common structure, the helmet is generally composed of an inner cover and an outer cover. The principle is that standard projection image information is projected onto a reflecting mirror or a film-coated layer on the inner side of the outer cover from a HUD fixed on the left and right sides of the inner cover or on the head of the pilot, and is reflected to the eyes of the pilot, and is superimposed with the real external information seen through the outer cover to form an augmented reality effect, thereby providing the pilot with overall information observed by both eyes or one eye, so as to prompt the pilot of road information, and has good practical engineering value.

[0003] At present, although such devices are emerging, some problems have been found with the increase of use time. One main problem is that, since the helmet is a daily necessity, the user may put it down at hand, and it may be hit by a hard object, so that the outer cover is deformed, some areas are uneven, or the internal HUD is displaced, which may cause changes in the light path, and the displayed image may be translated, rotated, or even distorted. Such adjustment involves adjustment of the HUD angle, image parameters, and the like. Some can be repaired by adjustment, and some cannot be repaired and need to be replaced. The existing situation is that the adjustment and judgment work is very troublesome and needs to be completed by professionals, and a large amount of manpower is consumed. In addition, the factory setting of the light path of the helmet display also involves such work, which also needs to consume a large amount of manpower. SUMMARY

[0004] The application develops a method and system for automatically correcting the light path of a helmet display to solve the above technical problems.

[0005] Technical scheme: In a first aspect, the embodiments of the application provide a method for automatically correcting the light path of a helmet display, comprising:

[0006] determining a standard helmet display and a standard projection image, and obtaining a standard reflection image of an outer cover in the standard helmet display;

[0007] obtaining first distortion parameters of the standard helmet display based on the standard projection image and the standard reflection image;

[0008] determining input data through the standard projection image and the standard reflection image, determining output data through the first distortion parameters, performing model training, and obtaining a distortion model;

[0009] Identify the helmet display to be tested, and acquire the reflected image of the outer cover of the helmet display to be tested as the reflected image to be tested;

[0010] The standard projection image and the reflection image to be tested are input into the distortion model to obtain the second distortion parameter;

[0011] The projected image of the HUD in the helmet-mounted display under test is corrected based on the second distortion parameter.

[0012] In some embodiments

[0013] In some embodiments, the step of acquiring a standard reflective image of the outer shroud (1) in the standard helmet-mounted display includes:

[0014] Determine the angular step distance, and rotate the standard projection image based on the angular step distance to obtain standard projection images corresponding to multiple rotation angles, as well as the corresponding standard reflection images;

[0015] A first linear step size is determined, and the standard projection image is moved based on the first linear step size to obtain standard projection images corresponding to multiple moving positions, as well as the corresponding standard reflection images;

[0016] A second linear step size is determined, the direction of the first linear step size intersects the direction of the first linear step size, and the standard projection image is moved based on the second linear step size to obtain standard projection images corresponding to multiple moving positions, as well as the corresponding standard reflection images.

[0017] In some embodiments, the step of obtaining the first distortion parameter of the standard helmet-mounted display based on the standard projected image and the standard reflected image includes:

[0018] Determine the sample box and the movement step size of the sample box in the standard projection image;

[0019] Move the sample box within the standard projection image to obtain multiple sample positions;

[0020] Projection samples are extracted from the sample locations of the standard projection image, and reflection samples are extracted from the sample locations of the standard reflection image.

[0021] Obtain the first distortion parameter of the reflected sample relative to the projected sample.

[0022] In some embodiments, the input data includes the pose data of the projection sample, the position data of the corner points of the projection sample, and the reflection sample; the pose data includes the angular step distance, the first linear step distance, or the second linear step distance marker.

[0023] In some embodiments, the distortion model includes a cascaded improved LeNet network and an SVR support vector regression model;

[0024] The improved LeNet network includes:

[0025] The 17th layer feature matrix is ​​connected to the 16th layer matrix of the C3 layer of the LeNet network; the corner value of the 17th layer feature matrix is ​​the normalized coordinate value of the corner of the projected sample in the first linear step direction, multiplied by 255; the center point position of the 17th layer feature matrix is ​​the pose data.

[0026] The 18th layer feature matrix is ​​connected to the 17th layer matrix of the C3 layer of the LeNet network; the corner value of the 18th layer feature matrix is ​​the normalized coordinate value of the corner of the projected sample in the second linear step direction, multiplied by 255; the center point position of the 18th layer feature matrix is ​​the pose data.

[0027] In some embodiments, the method further includes:

[0028] The standard projection image is pre-corrected using the second distortion parameter to obtain a pre-corrected image;

[0029] The pre-calibrated image is projected onto the outer cover (1) through the HUD (4) of the helmet display under test, and the image reflected by the outer cover (1) is obtained as the image to be tested;

[0030] Obtain the SIFT eigenvalues ​​of the scale-invariant feature transform of the feature points in the image to be tested;

[0031] If the SIFT feature value is within the acceptable range, the feature point correction is deemed acceptable.

[0032] If the number of calibrated feature points is less than the pass threshold, the helmet display under test is determined to be calibrated successfully.

[0033] In some embodiments, the step of obtaining the qualified range includes:

[0034] Multiple feature points are configured in the standard projected image, and the position data of the feature points are determined in the standard reflected image;

[0035] Multiple location data of the same feature point are obtained in the standard reflection images of multiple standard helmet displays, and the average value and variance of the multiple location data are obtained based on scale-invariant feature transformation;

[0036] The acceptable range is obtained based on the variance and the mean, and the formula for representing the acceptable range includes:

[0037] [AvgFeat(ziti,nfi)-3*DevFeat(ziti,nfi)]~[AvgFeat(ziti,nfi)+3*DevFeat(ziti,nfi)];

[0038] Where AvgFeat(ziti,nfi) is the average value of multiple identical feature points nfi under pose ziti; DevFeat(ziti,nfi) is the variance of multiple identical feature points nfi under pose ziti.

[0039] In some embodiments, the standard projection image includes the Zhang Zhengyou chessboard correction pattern;

[0040] The feature points include the intersections of the horizontal and vertical lines on the Zhang Zhengyou chessboard correction pattern;

[0041] The direction of the first linear step is the direction of the horizontal line on the Zhang Zhengyou chessboard correction pattern, and the direction of the second linear step is the direction of the up and down on the Zhang Zhengyou chessboard correction pattern.

[0042] Secondly, embodiments of this application also provide a system for automatically correcting the optical path of a helmet-mounted display, comprising:

[0043] A standard image acquisition module is used to determine a standard helmet-mounted display and a standard projected image, and to acquire a standard reflected image of the outer cover of the standard helmet-mounted display;

[0044] The first distortion parameter acquisition module is used to acquire the first distortion parameter of the standard helmet display based on the standard projected image and the standard reflected image.

[0045] A distortion model acquisition module is used to determine input data through the standard projection image and the standard reflection image, determine output data through the first distortion parameter, perform model training, and acquire a distortion model.

[0046] The image acquisition module is used to determine the helmet display under test and acquire the reflected image of the outer cover of the helmet display under test as the reflected image under test.

[0047] The second distortion parameter acquisition module is used to input the standard projection image and the reflection image to be tested into the distortion model to obtain the second distortion parameter.

[0048] A calibration projection module is used to correct the projected image of the HUD in the helmet-mounted display under test based on the second distortion parameter.

[0049] Beneficial Effects: Compared with the prior art, the present application provides a method for automatically correcting the optical path of a helmet-mounted display (HUD), including determining a standard HUD and a standard projected image, and acquiring a standard reflected image of the outer cover of the standard HUD; acquiring a first distortion parameter of the standard HUD based on the standard projected image and the standard reflected image; determining input data through the standard projected image and the standard reflected image, determining output data through the first distortion parameter, performing model training, and acquiring a distortion model; determining the HUD to be tested, and acquiring the reflected image of the outer cover of the HUD to be tested as the reflected image to be tested; inputting the standard projected image and the reflected image to be tested into the distortion model to obtain a second distortion parameter; and correcting the projected image of the HUD in the HUD to be tested based on the second distortion parameter. This application uses the first distortion parameter of a standard helmet-mounted display as the output, and the standard projected image and standard reflected image as the output to train the model and obtain the distortion model of the standard helmet-mounted display as the standard model; then, the reflected image to be tested and the standard projected image of the helmet-mounted display under test are input into the distortion model to obtain the second distortion parameter of the helmet-mounted display under test; finally, the projected image is corrected by the second distortion parameter and then output through the helmet under test. The correction is performed before the projected image is projected. When the pre-corrected image is reflected in the distorted optical path of the helmet-mounted display under test, the distortion of the optical path is canceled out, so as to project a qualified target image. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 A flowchart illustrating the steps of a method for automatically correcting the optical path of a helmet-mounted display provided in this application embodiment;

[0052] Figure 2 A flowchart illustrating the steps of obtaining a standard reflection image of the outer cover of a standard helmet display in a method for automatically correcting the optical path of a helmet display provided in this application embodiment;

[0053] Figure 3 A flowchart of the steps for obtaining the first distortion parameter in the method for automatically correcting the optical path of a helmet display provided in this application embodiment;

[0054] Figure 4A flowchart illustrating the steps of performing a conformity inspection on the helmet display under test in the method for automatically correcting the optical path of a helmet display provided in this application embodiment;

[0055] Figure 5 A flowchart illustrating the steps of obtaining a qualified range in the method for automatically correcting the optical path of a helmet-mounted display provided in this application embodiment;

[0056] Figure 6 A module connection diagram of a system for automatically correcting the optical path of a helmet-mounted display provided in an embodiment of this application;

[0057] Figure 7 This is the main view of the helmet display;

[0058] Figure 8 Side view of the helmet display;

[0059] Figure 9 A schematic diagram of the corrected pattern for Zhang Zhengyou's chessboard;

[0060] Figure 10 This is a structural diagram of the distortion model in the method for automatically correcting the optical path of a helmet-mounted display provided in an embodiment of this application.

[0061] Reference numerals: 1. Outer cover; 2. Inner cover; 3. Processor; 4. HUD; 10. Standard image acquisition module; 20. First distortion parameter acquisition module; 30. Distortion model acquisition module; 40. Image to be tested acquisition module; 50. Second distortion parameter acquisition module; 60. Correction projection module. Detailed Implementation

[0062] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0063] A helmet-mounted display (HUD) is a new type of image display system. In its common structure, the helmet generally consists of two parts: an inner visor 2 and an outer visor 1. The principle is that inside the driver's helmet, standard projected image information is projected from the left or right side of the inner visor 2 or from the HUD 4 on the head onto a reflector or the inner coating layer of the outer visor 1, and reflected to the driver's eyes. This image is superimposed on the real external information seen through the outer visor 1, forming an augmented reality effect. This allows the driver to observe overall information with both eyes or one eye, thereby providing the driver with road information and has great practical engineering value.

[0064] Currently, although such devices are gaining popularity, some problems have been discovered with increased usage. One major issue is that, because helmets are everyday items, users may carelessly discard them, causing them to come into contact with hard objects, resulting in deformation of the outer casing, unevenness in certain areas, or displacement of the internal HUD4. This alters the optical path, causing the displayed image to appear shifted, rotated, or even distorted. These adjustments involve modifying the HUD4's angle and image parameters; some issues can be corrected through adjustment, while others reach their limits and cannot be repaired, requiring replacement. Currently, this adjustment and assessment process is extremely cumbersome, requiring specialized personnel and consuming significant manpower. Furthermore, the factory settings for the helmet display's optical path also involve this process, which also requires substantial manpower.

[0065] In view of this, embodiments of this application provide a method for automatically correcting the optical path of a helmet-mounted display (HUD), including determining a standard HUD and a standard projected image, and acquiring a standard reflected image of the outer cover 1 in the standard HUD; acquiring a first distortion parameter of the standard HUD based on the standard projected image and the standard reflected image; determining input data through the standard projected image and the standard reflected image, determining output data through the first distortion parameter, performing model training, and acquiring a distortion model; determining the HUD to be tested, and acquiring the reflected image of the outer cover 1 in the HUD to be tested as the reflected image to be tested; inputting the standard projected image and the reflected image to be tested into the distortion model to obtain a second distortion parameter; and correcting the projected image of the HUD4 in the HUD to be tested based on the second distortion parameter. This application uses the first distortion parameter of a standard helmet-mounted display as the output, and the standard projected image and standard reflected image as the output to train the model and obtain the distortion model of the standard helmet-mounted display as the standard model; then, the reflected image to be tested and the standard projected image of the helmet-mounted display under test are input into the distortion model to obtain the second distortion parameter of the helmet-mounted display under test; finally, the projected image is corrected by the second distortion parameter and then output through the helmet under test. The correction is performed before the projected image is projected. When the pre-corrected image is reflected in the distorted optical path of the helmet-mounted display under test, the distortion of the optical path is canceled out, so as to project a qualified target image.

[0066] In some embodiments, please refer to Figure 7 and Figure 8 , Figure 7 This is the main view of the helmet display. Figure 8This is a side view of the helmet-mounted display. The helmet-mounted display provided in this embodiment includes an outer cover 1 and an inner cover 2. The inner cover 2 is relatively soft and designed to contact the human brain. The outer cover 1 is relatively rigid and designed to protect the human brain from the outside. A processor 3 and a HUD4 projector are installed in the middle of the outer cover 1 and the inner cover 2. The processor 3 includes a Bluetooth communication module, a 5G communication module, headphones, and a headset. Batteries and buttons are installed on the outside of the outer cover 1. A transparent reflective coating is applied to the inside of the outer cover 1. The HUD4 projector projects onto this coating, allowing the image to be reflected to the human eye, thus superimposing it onto the external scene to provide augmented reality information. In this embodiment, a helmet-mounted display that can communicate via Bluetooth uses a custom-designed helmet-mounted display from Nanjing Baituo Vision Technology Co., Ltd., where the outer cover 1 is a cylindrical structure with a 120-degree fan-shaped surface. The remaining structural relationships of the helmet-mounted display can be found in the invention patent "Helmet-Mounted Display Follow-up System and Method Based on Reinforcement Learning" (Patent No. ZL202311088059.6).

[0067] The camera is fixed to the eye area of ​​a mannequin head, the same size as the inside of the helmet. It is integrated with the mannequin head into the helmet-mounted display, which communicates via Bluetooth, ensuring a normal viewing experience for the user. It can be installed in either the left or right eye position. It is used to observe the augmented reality effects of the HUD4. It transmits relevant video data to a Bluetooth-enabled computer for analysis and control, performing image quality analysis. In this specific embodiment, a Hikvision 4K color camera is used, which can be focused via communication.

[0068] The role of a Bluetooth-enabled computer for analysis and control is to collect camera images, provide computational resources for algorithm analysis, and control the parameters displayed on the HUD4 projector via Bluetooth, performing correction feedback operations. In a specific embodiment of the invention, a Huawei Matebook computer is used.

[0069] Understandably, the main task of the automatic correction method in this application is to calculate the distortion of the image projected onto the inner reflective coating of the outer cover 1 and then reflected to the human eye based on the existing optical path conditions, and then compensate for it. If the compensation exceeds the range of the HUD4 projection, it indicates a problem with the optical path that cannot be repaired, and an alarm will be triggered to inform the user. The overall algorithm mainly runs on a computer. The principle of the automatic correction method in this application differs from traditional camera lens correction methods (e.g., the classic Zhang Zhengyou chessboard correction method). Traditional lens correction methods address the effect that the amplitude of distortion in an optical lens increases with the distance from the center of the lens due to manufacturing process issues. However, the automatic correction method in this application addresses deformation caused by external collisions and aging, such as unevenness in the area. Therefore, this invention cannot completely use the original Zhang Zhengyou chessboard correction method and must be improved upon. The main scheme of the automatic correction method in this application is to use the HUD4 projection Zhang Zhengyou chessboard correction method as a basis to perform nonlinear image correction based on CNN.

[0070] Understandably, geometric distortion is often the most significant problem encountered in camera correction, and it comprises two parts: radial and tangential. Radial distortion causes image points to shift along radial lines starting from the principal point.

[0071] Radial error can be approximated by a polynomial:

[0072] ;

[0073] or

[0074] ;

[0075] Where r is the distance between the image point and the principal point. In this specific embodiment of the invention, parameters 2, 4, and 6 are used. There are no further orders after 8.

[0076] Tangential distortion occurs in the direction perpendicular to the radius and is usually less severe than radial distortion.

[0077] After distortion, a point (u,v) that should have been projected onto the phase plane becomes... The distorted coordinates are given by the following formula.

[0078] ;

[0079] ;

[0080] ;

[0081] For different (u,v) values, the displacement vector can be plotted, showing the displacement required to correct distortion at different points in the image. , .Right now:

[0082] ;

[0083] ;

[0084] In some embodiments, this application projects a Zhang Zhengyou chessboard image from HUD4 onto a mask on the outer cover 1, and then a camera fixed to the doll's eyes captures the chessboard image. For example... Figure 9 As shown, the image of the Zhang Zhengyou chessboard correction method set in this application is a square, with 10 smaller squares in each of the horizontal and vertical directions, totaling 100 smaller squares. If the maximum physical range of the displayed pattern seen by the camera on the doll's eye is Wph (width) * Hph (height), then the physical size of each smaller square is Wph / 10 for width and Hph / 10 for height. Furthermore, due to the inconsistent length and width, the Zhang Zhengyou chessboard image in this application is set to be aligned to the left and right edges multiple times. In the 4:3 aspect ratio of this specific embodiment, it is aligned to the left once and to the right once. The distortion parameter of the intersecting part is the average of the calculation results of the two left and right aligned patterns. Figure 9 In the image, the dark green section in the middle forms the central intersection, while the light green section and the dark green section in the middle together form a checkerboard correction pattern. Similarly, the dark green section in the middle and the pink section on the right together form a checkerboard correction pattern. Specifically, during projection and image capture, under the same initial light path conditions of the mask, the original HUD4 is not adjusted. Only the Zhang Zhengyou chessboard image displayed on HUD4 was adjusted, and different rotation and translation poses were set. 360 images were taken with rotations of 1 degree each from 0 to 360 degrees. 100 images were taken with no rotation and equidistant translation (each translation distance is one pixel) to the left and right edges of the image. 100 images were also taken with no rotation and equidistant translation (each translation distance is one pixel) to the top and bottom edges of the image, for a total of 560 images. That is, under poses Zit(ziti) (ziti=1~560), the projected image Xzha was used as the image dataset (Xzha, Ysta) for quality analysis, where Xzha is the projected image and Ysta is the corresponding captured image.

[0085] In some embodiments, please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a method for automatically correcting the optical path of a helmet-mounted display (HMD) according to an embodiment of this application. The method for automatically correcting the optical path of a HMD in this application is specifically implemented through steps 100 to 600:

[0086] Step 100: Determine the standard helmet display and standard projected image, and acquire the standard reflected image of the outer cover 1 in the standard helmet display.

[0087] In some embodiments, the standard projected image includes the Zhang Zhengyou chessboard correction pattern.

[0088] In some embodiments, please refer to Figure 2 , Figure 2 This is a flowchart illustrating the steps of obtaining a standard reflection image of the outer cover 1 in a method for automatically correcting the optical path of a helmet-mounted display provided in this application embodiment. The method for obtaining the standard reflection image of the outer cover 1 in this application is specifically implemented through steps 110 to 130:

[0089] Step 110: Determine the angular step distance, and rotate the standard projection image based on the angular step distance to obtain the standard projection images corresponding to multiple rotation angles, as well as the corresponding standard reflection images.

[0090] In some embodiments, the angular step size is 1 degree, and rotating the standard projection image 360 ​​degrees can obtain 360 standard projection images at different angles, as well as the corresponding 360 standard projection images at different angles.

[0091] Step 120: Determine the first linear step size, and move the standard projection image based on the first linear step size to obtain the standard projection images corresponding to multiple moving positions, as well as the corresponding standard reflection images.

[0092] In some embodiments, the direction of the first linear step is the direction of the horizontal line on the Zhang Zhengyou chessboard correction pattern. Each time the image is translated by one pixel, a total of 100 standard reflection images are acquired along the horizontal line direction of the Zhang Zhengyou chessboard correction pattern.

[0093] Step 130: Determine the second linear step size. The direction of the first linear step size intersects with the direction of the first linear step size. Move the standard projection image based on the second linear step size to obtain the standard projection images corresponding to multiple moving positions, as well as the corresponding standard reflection images.

[0094] In some embodiments, the direction of the second linear step is the direction of the vertical lines on the Zhang Zhengyou chessboard correction pattern. Each time the image is translated by one pixel, a total of 100 standard reflection images are acquired along the vertical lines of the Zhang Zhengyou chessboard correction pattern.

[0095] Specifically, this application used 100 qualified standard helmet displays and collected qualified image datasets (Dataset(Xzha,Ysta)(nmi)) of analysis quality for each of them, where nmi=1~100, and each dataset contains 560 images of different poses (Zit(ziti)(ziti=1~560).

[0096] Step 200: Obtain the first distortion parameters of the standard helmet display based on the standard projected image and the standard reflected image.

[0097] Understandably, obtaining a qualified image dataset, Dataset(Xzha,Ysta)(nmi), only yields a collection of individual images. Further segmentation of the overall sample is needed to obtain individual learnable samples. Please refer to [link / reference]. Figure 3 , Figure 3 This is a flowchart illustrating the steps of obtaining the first distortion parameter in the method for automatically correcting the optical path of a helmet-mounted display provided in this application embodiment. The method for obtaining the first distortion parameter in this application is specifically implemented through steps 210 to 240:

[0098] Step 210: Determine the bounding box and the step size of the bounding box in the standard projection image.

[0099] Specifically, the standard reflectance image Ysta contains 10*10 squares, such as... Figure 9 As shown, this application uses each 2*2 square in the image as a sample frame.

[0100] Step 220: Move the sample box in the standard projection image to obtain multiple sample positions.

[0101] Specifically, such as Figure 9 As shown, the 2x2 square contained in the red area in the upper left corner is used as the first sample, and then the 2x2 square contained in the orange area on the right is used as the second sample. And so on. In this way, an image Ysta can be segmented into 9*9=81 samples, resulting in 560 poses * 81 = 45360 samples for each standard helmet-mounted display.

[0102] Step 230: Extract projection samples from the sample locations of the standard projection image and extract reflection samples from the sample locations of the standard reflection image.

[0103] Specifically, projection samples Xgs(si) are extracted from the sample locations of the standard projection image Xzha, and reflection samples Ygs(si) are extracted from the sample locations of the standard reflection image Ysta. Then, these image samples of different sizes are normalized to a size of 32*32 for later model learning.

[0104] Step 240: Obtain the first distortion parameter of the reflection sample relative to the projection sample.

[0105] Specifically, since this application uses the Zhang Zhengyou chessboard correction pattern as the standard projection pattern, the Zhang Zhengyou chessboard correction method is used to obtain the first distortion parameters. The first distortion parameters include three radial distortion parameters {k1,k2,k3} and two tangential distortion parameters {p1,p2}.

[0106] Step 300: Determine the input data using the standard projected image and the standard reflected image, determine the output data using the first distortion parameter, train the model, and obtain the distortion model.

[0107] In some embodiments, the input data includes pose data of the projected sample, position data of the corner points of the projected sample, and a reflection sample; the pose data includes a marker of the angular step distance, the first linear step distance, or the second linear step distance.

[0108] Specifically, among 100 standard helmet-mounted displays, data from each projection sample, corner data of the projection sample, data from the reflection sample, and the corresponding first distortion parameter are acquired from each standard helmet-mounted display to form a standard helmet-mounted display sample set. The characterization formula for this set includes:

[0109] Datasetgood2(nmi)={Xgs(si,nmi),Corner1(si,nmi),Corner2(si,nmi),Corner3(si,nmi),Corner4(si, nmi),Ygs(si, nmi),k1(nmi),k2(nmi),k3(nmi),p1(nmi),p2(nmi)};

[0110] Wherein, Datasetgood2(nmi) is the sample set of the nmi-th standard helmet-mounted display; Xgs(si,nmi) is the projection sample in the pose si of the nmi-th standard helmet-mounted display; Ygs(si, nmi) is the reflection sample in the pose si of the nmi-th standard helmet-mounted display; Corner1 is the upper left corner coordinate of the sample position, Corner2 is the upper right corner coordinate of the sample position, Corner3 is the lower left corner coordinate of the sample position, and Corner4 is the lower right corner coordinate of the sample position; k1(nmi), k2(nmi), and k3(nmi) are three radial distortion parameters in the nmi-th standard helmet-mounted display; and p1(nmi) and p2(nmi) are two tangential distortion parameters in the nmi-th standard helmet-mounted display. For the same helmet nmi, although it has different poses si, its distortion parameters are the same.

[0111] Understandably, since this application addresses deformation caused by external collisions and aging, such as unevenness in small areas, the distortion correction algorithm cannot employ the overall radial and tangential distortion theories of traditional optical lenses. Instead, a region-specific, non-linear approach is required. Therefore, this application must choose an artificial intelligence-based solution. Furthermore, considering the limited computing power of the helmet-mounted controller, a model with minimal computational requirements must be chosen. Additionally, the model's accuracy must be adaptable to variations in computing power to accommodate user-defined needs regarding accuracy and computational efficiency.

[0112] In some embodiments, please refer to Figure 10 , Figure 10 This is a structural diagram of the distortion model in the method for automatically correcting the optical path of a helmet display provided in this application embodiment. This application uses a part of the LeNet network with the fewest parameters as a feature extractor, which has about 5,000 parameters. The features are then extracted and fed to the SVR (Support Vector Regression Model) for regression.

[0113] Specifically, the distortion model comprises a cascaded improved LeNet network and an SVR support vector regression model. This model first processes the 32*32 pattern samples according to the classic LeNet approach. Then, at its C3 layer, corner positions are added. Since the C3 layer is a 16-layer 10*10 matrix, this invention directly adds two 10*10 matrices to the original 16 layers.

[0114] Specifically, the two 10*10 matrices include the 17th and 18th layer feature matrices; the 17th layer feature matrix is ​​connected to the 16th layer matrix of the C3 layer of the LeNet network; the corner values ​​of the four corners of the 17th layer feature matrix are the normalized coordinates of the four corners of the projected sample in the first linear step direction X, multiplied by 255. The (1,1) position of this matrix represents the X value of the top-left corner Corner1, the (1,10) position represents the X value of the top-right corner Corner2, the (10,1) position represents the X value of the top-left corner Corner3, and the (10,10) position represents the X value of the bottom-right corner Corner4; all others are 0. The center point of the 17th layer feature matrix (… The position (6,6) is the marker for pose data si; ​​the 18th layer feature matrix is ​​connected to the 17th layer matrix of the C3 layer of the LeNet network; the corner values ​​of the four corner points of the 18th layer feature matrix are the normalized coordinates of the four corner points of the projected sample in the second linear step direction Y, multiplied by 255. The (1,1) position of this matrix is ​​the Y value of the top left corner Corner1, the (1,10) position of this matrix is ​​the Y value of the top right corner Corner2, the (10,1) position of this matrix is ​​the Y value of the top left corner Corner3, the (10,10) position of this matrix is ​​the Y value of the bottom right corner Corner4, and all others are 0; the center point (6,6) of the 18th layer feature matrix is ​​the marker for pose data si.

[0115] In some embodiments, in layers C5 and F6 of the LeNet network, by averaging neighboring values, such as averaging the first and second, and the third and fourth, layers C5 is reduced to 60 layers and layers F6 is reduced to 42 layers. The improved LeNet network can extract 42 features from the image, and then submit these 42 features to an SVR (Support Vector Regression) model for regression to obtain a regression value (i.e., k1, k2, k3, p1, p2).

[0116] Understandably, by configuring the input and output of Datasetgood2d, a LeNetSVR network model capable of calculating distortion parameters is trained, resulting in dedicated models for each of the five distortion parameters: Modelk1, Modelk2, Modelk3, Modelp1, and Modelp2. These distortion parameters correspond to the distortion of a 2x2 chessboard-sized sample region in Zhang Zhengyou's chessboard, achieving region-specific correction for a more refined restoration effect. However, this also results in the highest computational cost. Modifying the sample region size, such as to a 4x4 square, increases the area corresponding to each distortion parameter, leading to a coarser correction while reducing computational cost. This algorithm can be adaptively modified based on hardware computing power for better adaptability.

[0117] Step 400: Determine the helmet display to be tested, and acquire the reflection image of the outer cover 1 in the helmet display to be tested as the reflection image to be tested.

[0118] Specifically, similar to the image acquisition scheme in a standard helmet-mounted display, the reflective images of the helmet-mounted display under test are acquired at different poses to obtain the image dataset:

[0119] Datasetunknown2={Xgs(si,unk),Corner1(si,unk),Corner2(si,unk),Corner3(si,unk),Corner4(si,unk),Ygs(si,unk)};

[0120] Wherein, Datasetunknown2 is the dataset of images to be tested; Xgs(si,unk) is the projection sample of the standard projection image of the helmet display unk to be tested in pose si; Corner1 is the upper left corner coordinate of the sample position, Corner2 is the upper right corner coordinate of the sample position, Corner3 is the lower left corner coordinate of the sample position, and Corner4 is the lower right corner coordinate of the sample position; Ygs(si,unk) is the reflection sample of the reflection image of the helmet display unk to be tested in pose si.

[0121] Step 500: Input the standard projection image and the reflection image to be tested into the distortion model to obtain the second distortion parameter.

[0122] Specifically, si, Corner1(si,unk), Corner2(si,unk), Corner3(si,unk), and Corner4(si,unk) are input into the distortion models Modelk1, Modelk2, Modelk3, Modelp1, and Modelp2, respectively, to obtain five second distortion parameters k1(unk), k2(unk), k3(unk), p1(unk), and p2(unk).

[0123] In some embodiments, after obtaining the second distortion parameter, the helmet-mounted display under test can be inspected for conformity using the second distortion parameter. Please refer to [link to relevant documentation]. Figure 4 , Figure 4 This is a flowchart illustrating the steps of the helmet display under test in the method for automatically correcting the optical path of a helmet display provided in this application. The method for conducting the conformity inspection of the helmet display under test in this application is specifically implemented through steps a to e:

[0124] Step a: Obtain the pre-corrected image by pre-correcting the standard projection image using the second distortion parameter.

[0125] Step b: Project a pre-calibrated image onto the outer cover 1 through the HUD4 of the helmet display under test, and obtain the image reflected by the outer cover 1 as the image to be tested.

[0126] Step c: Obtain the SIFT eigenvalues ​​of the scale-invariant feature transform of the feature points in the image to be tested.

[0127] Step d: In response to the SIFT feature value being within the acceptable range, the feature point correction is deemed acceptable.

[0128] In some embodiments, please refer to Figure 5 , Figure 5 This is a flowchart illustrating the steps for obtaining a qualified range in the method for automatically correcting the optical path of a helmet-mounted display provided in this application embodiment. The method for obtaining the qualified range in this application is specifically implemented through steps d1 to d3:

[0129] Step d1: Configure multiple feature points in the standard projected image and determine the location data of the feature points in the standard reflected image.

[0130] In some embodiments, the feature points include the intersections of horizontal and vertical lines on the Zhang Zhengyou chessboard correction pattern.

[0131] Specifically, each standard projected image contains 121 feature points. For the feature F(nfi) at the top left corner of the first pose ziti=1, nfi=1, it should have corresponding features from 100 different qualified helmets.

[0132] Step d2: Obtain multiple location data of the same feature points in the standard reflection images of multiple standard helmet displays, and obtain the mean and variance of the multiple location data based on scale-invariant feature transformation.

[0133] Specifically, the SIFT feature mean AvgFeat(1,1) and variance DevFeat(1,1) of these 100 F(nfi) are calculated as the result of the first feature (nfi=1) in this pose (ziti=1). There are a total of 560 poses * 121 features * 2 parameters (mean + variance) results.

[0134] Step d3: Obtain the acceptable range based on variance and mean.

[0135] In some embodiments, a threshold is constructed using the 3S variance standard, and the acceptable range is determined by the threshold. The characterization formula includes:

[0136] [AvgFeat(ziti,nfi)-3*DevFeat(ziti,nfi)]~[AvgFeat(ziti,nfi)+3*DevFeat(ziti,nfi)];

[0137] Where AvgFeat(ziti,nfi) is the average value of multiple identical feature points nfi under pose ziti; DevFeat(ziti,nfi) is the variance of multiple identical feature points nfi under pose ziti.

[0138] Understandably, this application uses the average value AvgFeat(ziti,nfi) as the ideal standard position of different feature points Feat(nfi) (nfi=1~121) under each different pose Zit(ziti) (ziti=1~560) when analyzing the image. The range [AvgFeat(ziti,nfi)-3*DevFeat(ziti,nfi)] to [AvgFeat(ziti,nfi)+3*DevFeat(ziti,nfi)] is defined as its acceptable range. In this way, each feature point in the 560 poses has an ideal standard position and an acceptable range.

[0139] Step e: In response to the number of calibrated feature points being less than the calibrated threshold, the helmet display under test is determined to be calibrated successfully.

[0140] Specifically, the computer inputs the calculated k1(unk), k2(unk), k3(unk), p1(unk), and p2(unk) to processor 3 in the helmet display via Bluetooth. Processor 3, based on the distortion correction principle (referencing Chapter 11 of the book *Robotics, Machine Vision and Control: MATLAB Algorithm Fundamentals* (by Corke, P.; translated by Liu Rong), Homogeneous Correction Method), uses k1(unk), k2(unk), k3(unk), p1(unk), and p2(unk) to correct the standard projected image, obtaining a corrected image of Zhang Zhengyou's chessboard. This image is then reprojected and displayed, captured by the camera, and submitted to the computer. At this point, the computer extracts its SIFT feature values ​​again and determines whether the feature points are within the normal range of the feature points in step D. It counts the number of feature points outside the normal range. If the number is greater than or equal to the threshold Tpo, it is considered that the algorithm cannot correct it and hardware processing is required, triggering an alarm for the user. If the number is below the threshold Tpo, the correction is considered successful.

[0141] In some embodiments, the judgment threshold can be controlled according to specific costs, for example, changing 3S to 5S or 4S, or modifying Tpo to a larger value, thereby reducing the quality cost of production. In this application, Tpo is 10% of the total number of feature points.

[0142] Step 600: Correct the projected image of HUD4 in the helmet display under test based on the second distortion parameter.

[0143] Understandably, the method for automatically correcting the optical path of a helmet-mounted display provided in this application includes determining a standard helmet-mounted display and a standard projected image, and acquiring a standard reflected image of the outer cover 1 in the standard helmet-mounted display; acquiring a first distortion parameter of the standard helmet-mounted display based on the standard projected image and the standard reflected image; determining input data through the standard projected image and the standard reflected image, determining output data through the first distortion parameter, performing model training, and acquiring a distortion model; determining the helmet-mounted display to be tested, and acquiring the reflected image of the outer cover 1 in the helmet-mounted display to be tested as the reflected image to be tested; inputting the standard projected image and the reflected image to be tested into the distortion model to obtain a second distortion parameter; and correcting the projected image of the HUD4 in the helmet-mounted display to be tested based on the second distortion parameter. This application uses the first distortion parameter of a standard helmet-mounted display as the output, and the standard projected image and standard reflected image as the output to train the model and obtain the distortion model of the standard helmet-mounted display as the standard model; then, the reflected image to be tested and the standard projected image of the helmet-mounted display under test are input into the distortion model to obtain the second distortion parameter of the helmet-mounted display under test; finally, the projected image is corrected by the second distortion parameter and then output through the helmet under test. The correction is performed before the projected image is projected. When the pre-corrected image is reflected in the distorted optical path of the helmet-mounted display under test, the distortion of the optical path is canceled out, so as to project a qualified target image.

[0144] In some embodiments, please refer to Figure 6 , Figure 6 This is a module connection diagram of a system for automatically correcting the optical path of a helmet-mounted display provided in this application embodiment. The system for automatically correcting the optical path of a helmet-mounted display provided in this application embodiment includes:

[0145] The standard image acquisition module 10 is used to determine the standard helmet display and the standard projected image, and to acquire the standard reflected image of the outer cover 1 in the standard helmet display;

[0146] The first distortion parameter acquisition module 20 is used to acquire the first distortion parameter of the standard helmet display based on the standard projection image and the standard reflection image.

[0147] The distortion model acquisition module 30 is used to determine the input data through the standard projection image and the standard reflection image, determine the output data through the first distortion parameter, perform model training, and acquire the distortion model.

[0148] The image acquisition module 40 is used to determine the helmet display under test and acquire the reflected image of the outer cover 1 in the helmet display under test as the reflected image under test.

[0149] The second distortion parameter acquisition module 50 is used to input the standard projection image and the reflection image to be tested into the distortion model to obtain the second distortion parameters.

[0150] The calibration projection module 60 is used to correct the projected image of HUD4 in the helmet display under test based on the second distortion parameter.

[0151] This application has provided a detailed description of a method and system for automatically correcting the optical path of a helmet-mounted display, as provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for automatically calibrating the optical path of a helmet-mounted display, characterized in that, include: Determine the standard helmet display and standard projected image, and obtain the standard reflected image of the outer cover (1) in the standard helmet display; The first distortion parameter of the standard helmet display is obtained based on the standard projected image and the standard reflected image; Input data is determined by the standard projected image and the standard reflected image, and output data is determined by the first distortion parameter. Model training is then performed to obtain the distortion model. Identify the helmet display to be tested, and obtain the reflection image of the outer cover (1) of the helmet display to be tested as the reflection image to be tested; The standard projection image and the reflection image to be tested are input into the distortion model to obtain the second distortion parameter; The projected image of the HUD(4) in the helmet display under test is corrected based on the second distortion parameter.

2. The method for automatically correcting the optical path of a helmet-mounted display according to claim 1, characterized in that, The step of acquiring the standard reflection image of the outer shroud (1) in the standard helmet display includes: Determine the angular step distance, and rotate the standard projection image based on the angular step distance to obtain standard projection images corresponding to multiple rotation angles, as well as the corresponding standard reflection images; A first linear step size is determined, and the standard projection image is moved based on the first linear step size to obtain standard projection images corresponding to multiple moving positions, as well as the corresponding standard reflection images; A second linear step size is determined, the direction of the first linear step size intersects the direction of the first linear step size, and the standard projection image is moved based on the second linear step size to obtain standard projection images corresponding to multiple moving positions, as well as the corresponding standard reflection images.

3. The method for automatically correcting the optical path of a helmet-mounted display according to claim 2, characterized in that, The step of obtaining the first distortion parameter of the standard helmet display based on the standard projected image and the standard reflected image includes: Determine the sample box and the movement step size of the sample box in the standard projection image; Move the sample box within the standard projection image to obtain multiple sample positions; Projection samples are extracted from the sample locations of the standard projection image, and reflection samples are extracted from the sample locations of the standard reflection image. Obtain the first distortion parameter of the reflected sample relative to the projected sample.

4. The method for automatically correcting the optical path of a helmet-mounted display according to claim 3, characterized in that, The input data includes the pose data of the projection sample, the position data of the corner points of the projection sample, and the reflection sample; the pose data includes the angular step distance, the first linear step distance, or the second linear step distance marker.

5. The method for automatically correcting the optical path of a helmet-mounted display according to claim 4, characterized in that, The distortion model includes a cascaded improved LeNet network and an SVR support vector regression model. The improved LeNet network includes: The 17th layer feature matrix is ​​connected to the 16th layer matrix of the C3 layer of the LeNet network; the corner value of the 17th layer feature matrix is ​​the normalized coordinate value of the corner of the projected sample in the first linear step direction, multiplied by 255; the center point position of the 17th layer feature matrix is ​​the pose data. The 18th layer feature matrix is ​​connected to the 17th layer matrix of the C3 layer of the LeNet network; the corner value of the 18th layer feature matrix is ​​the normalized coordinate value of the corner of the projected sample in the second linear step direction, multiplied by 255; the center point position of the 18th layer feature matrix is ​​the pose data.

6. The method for automatically correcting the optical path of a helmet-mounted display according to claim 1, characterized in that, The method further includes: The standard projection image is pre-corrected using the second distortion parameter to obtain a pre-corrected image; The pre-calibrated image is projected onto the outer cover (1) through the HUD (4) of the helmet display under test, and the image reflected by the outer cover (1) is obtained as the image to be tested; Obtain the SIFT eigenvalues ​​of the scale-invariant feature transform of the feature points in the image to be tested; If the SIFT feature value is within the acceptable range, the feature point correction is deemed acceptable. If the number of calibrated feature points is less than the pass threshold, the helmet display under test is determined to be calibrated successfully.

7. The method for automatically correcting the optical path of a helmet-mounted display according to claim 6, characterized in that, The steps for obtaining the qualified range include: Multiple feature points are configured in the standard projected image, and the position data of the feature points are determined in the standard reflected image; Multiple location data of the same feature point are obtained in the standard reflection images of multiple standard helmet displays, and the average value and variance of the multiple location data are obtained based on scale-invariant feature transformation; The acceptable range is obtained based on the variance and the mean, and the formula for representing the acceptable range includes: [AvgFeat(ziti,nfi)-3*DevFeat(ziti,nfi)]~[AvgFeat(ziti,nfi)+3*DevFeat(ziti,nfi)]; Where AvgFeat(ziti,nfi) is the average value of multiple identical feature points nfi under pose ziti; DevFeat(ziti,nfi) is the variance of multiple identical feature points nfi under pose ziti.

8. The method for automatically correcting the optical path of a helmet-mounted display according to claim 2 or 6, characterized in that... The standard projection image includes Zhang Zhengyou's chessboard correction pattern; The feature points include the intersections of the horizontal and vertical lines on the Zhang Zhengyou chessboard correction pattern; The direction of the first linear step is the direction of the horizontal line on the Zhang Zhengyou chessboard correction pattern, and the direction of the second linear step is the direction of the up and down on the Zhang Zhengyou chessboard correction pattern.

9. A system for automatically correcting the optical path of a helmet-mounted display, characterized in that, include: A standard image acquisition module (10) is used to determine a standard helmet display and a standard projection image, and to acquire a standard reflection image of the outer cover (1) in the standard helmet display; The first distortion parameter acquisition module (20) is used to acquire the first distortion parameter of the standard helmet display based on the standard projection image and the standard reflection image. The distortion model acquisition module (30) is used to determine input data through the standard projection image and the standard reflection image, determine output data through the first distortion parameter, perform model training, and acquire the distortion model. The image acquisition module (40) is used to determine the helmet display under test and acquire the reflection image of the outer cover (1) in the helmet display under test as the reflection image under test. The second distortion parameter acquisition module (50) is used to input the standard projection image and the reflection image to be tested into the distortion model to obtain the second distortion parameter; The calibration projection module (60) is used to correct the projected image of the HUD (4) in the helmet display under test based on the second distortion parameter.

Images