A panoramic image stitching method in an optical simulation training system

Abstract

The present invention belongs to the field of image processing technology, and relates to a panoramic image stitching method in an optical simulation training system. The method comprises: a photoelectric theodolite shoots according to a preset theoretical trajectory; preprocesses the acquired images; establishes an association mapping model between images to realize image registration; establishes a panoramic image stitching mapping table; fuses the images according to the mapping relationship to form a panoramic image; and searches the panoramic image stitching mapping table by a secondary addressing method according to the required display angle to realize partial display of the image. The panoramic stitching method of the present invention solves the problem that when the existing panoramic image stitching method is directly implemented, the cost is too high and the image accuracy is affected when multiple sets of photoelectric theodolites are used for data collection, and also solves the problem that there are no obvious features in the captured image, and thus the feature point matching stitching method cannot be used.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a panoramic image stitching method in an optical simulation training system. Background Art

[0002] Photoelectric theodolite has photoelectric tracking and measurement functions and is widely used in image and attitude measurement tasks. Traditional training methods are greatly limited by time and cost, so in actual work, more and more training methods use simulation training methods.

[0003] In order to make the simulation training process more realistic, the real-life image stitching method is used in the optical simulation training system, thus solving the problem of unrealistic training scenes caused by the use of mathematical modeling methods such as 3D modeling in other simulation training systems.

[0004] The images taken by the photoelectric theodolite have high resolution and a small field of view. In order to meet the system's requirements for training scenarios, it is necessary to expand the field of view of the image so that images with different fields of view can be retrieved during training. The ideal situation is to obtain a 360-degree panoramic image.

[0005] In the process of shooting with the photoelectric theodolite, in order to ensure the integrity of the image, there is partial overlap between images at adjacent angles, so panoramic image stitching technology is needed.

[0006] It only relies on a single photoelectric theodolite to take pictures and achieve panoramic image stitching in both horizontal and vertical dimensions.

[0007] Due to practical needs, most of the backgrounds of photoelectric theodolites have no obvious features, and splicing cannot be achieved by matching feature points.

[0008] A method for stitching panoramic images in a zigzag pattern in an array camera (application number 202010769524.2, authorization announcement number CN 111738925 B) uses existing registration transformation parameters to stitch images from four cameras arranged in a zigzag pattern, solving the problem that the existing stitching technology cannot process zigzag images at the same time; a panoramic video synchronous stitching system, method and panoramic video display device (application number 201710764095.8, application announcement number CN 107396068 A) also sets multiple cameras, and sets the cameras on the same plane in a preset order, so that the angles of two adjacent cameras meet the range of shooting video images and there is an overlapping area, and each camera combination shoots a panoramic video image. From the perspective of economy and reliability, if an array camera or a multi-camera data acquisition method is used as a photoelectric theodolite as a shooting device, many problems will arise. When multiple sets of photoelectric theodolites are shot simultaneously, the layout requirements of the data acquisition equipment are high and the cost is high. There are systematic errors in photoelectric theodolite photography, so shooting with different devices of the same model will also affect the accuracy of the stitched image.

[0009] A panoramic image stitching method and device (application number 201811071685.3, application publication number CN109389555 A) extracts feature points from two images to be stitched, matches the two sets of key points, obtains a matched feature point pair set, and then divides the grid to perform registration through local adaptive homography estimation and other methods to perform image stitching. This method is only applicable to situations where there are obvious features in the image, but is not applicable to situations where the background is as clean as possible and there is no excessive interference, and cannot achieve the expected effect. Summary of the invention

[0010] (I) Technical Problems to be Solved by the Present Invention

[0011] In view of the above technical problems, the present invention provides a panoramic image stitching method in an optical simulation training system, uses a single set of photoelectric theodolite to collect data, and proposes a corresponding panoramic stitching method, which solves the problems of high cost and affected image accuracy that may occur when multiple sets of photoelectric theodolites are used for data collection when the existing panoramic image stitching method is directly implemented. In addition, in the panoramic stitching process, a mapping model is established in combination with the position information attached to the image taken by the photoelectric theodolite, which solves the problem that there are no obvious features in the captured image and the feature point matching stitching method cannot be used.

[0012] (II) Complete technical solution provided by the present invention

[0013] The present invention provides a panoramic image stitching method in an optical simulation training system, comprising the following steps:

[0014] S1. The photoelectric theodolite takes pictures according to the preset theoretical trajectory to complete data collection;

[0015] S2, preprocessing the acquired images;

[0016] S3, establish the correlation mapping model between images to achieve image registration;

[0017] S4, establishing a panoramic image stitching mapping table;

[0018] S5. According to the mapping relationship, the images are fused to form a panoramic image;

[0019] S6. According to the required display angle, the panoramic image stitching mapping table is searched through a secondary addressing method to realize partial image display.

[0020] Furthermore, the theoretical trajectory preset in step S1 is the angle guidance data of the photoelectric theodolite.

[0021] Furthermore, in step S1, when shooting, the photoelectric theodolite is kept in a fixed position, and the preset theoretical trajectory guides the photoelectric theodolite to rotate 360 ​​degrees at a constant speed at each fixed pitch angle in the order of pitch angle from low to high, and according to the field of view size of the captured image, ensure that there is an overlapping area in the range of the captured video images at adjacent azimuths and pitch angles.

[0022] Further, the preprocessing in step S2 includes the following steps:

[0023] S21, judging the collected image;

[0024] If it is an infrared image, proceed to step S22;

[0025] If it is a visible light image, proceed to step S24;

[0026] S22, performing image enhancement on the infrared image;

[0027] S23, brightness enhancement, proceed to step S24;

[0028] S24, performing denoising processing on the collected image by using a median filtering method.

[0029] Furthermore, the step S3 of establishing the association mapping model between images includes the following steps:

[0030] S31, estimating the size of the panoramic image according to the field of view (horizontally α, longitudinally β) and the pixel size μ of the photoelectric theodolite, horizontally: Width = 360 / (α·μ), longitudinally: Height = 90 / (β·μ);

[0031] S32, allocate a space of size Width*Height in the memory, recorded as the spliced ​​image M;

[0032] S33, reading the video image taken by the photoelectric theodolite frame by frame, and obtaining the encoder angle, pixel size and other information attached to the file header information of the image;

[0033] S34, according to the synthesis angle of each pixel point in the image, reversely calculate the position information of the point in the spliced ​​image M, and write it into M;

[0034] S35, repeating steps S33-S34 until the positions of all pixels in the video image in the spliced ​​image M are obtained, thereby establishing a relationship mapping model between images.

[0035] Further, in step S34, according to the synthesis angle of each pixel point in the image, the position information of the point in the spliced ​​image M is calculated inversely, including the following steps:

[0036] S341. The formula for establishing the synthetic target angle is as follows, where A represents the azimuth angle, E represents the elevation angle, and f represents the focal length:

[0037]

[0038] A=A 0 +ΔA

[0039]

[0040] S342. Simplify the formula to obtain an approximate formula for the synthetic target angle:

[0041]

[0042]

[0043] S343. According to the above formula, if the angular offset of a point relative to the center point is known, the positional relationship between the two points can be inversely calculated to achieve the conversion of the angle value (A, E) and coordinate value (x, y) of any point on the image.

[0044] Further, the step S4 of establishing the panoramic image stitching mapping table includes the following steps:

[0045] S41. During the splicing process, record the corresponding pitch angle value;

[0046] S42, the pitch angle starts from 0° and is separated by 0.5° as a pitch angle band, and in each angle band, the azimuth angle starts from 0° and is separated by 3° to record the angle value and pixel position of the nearest real image corresponding to the degree;

[0047] S43, repeat S41 and S42 until all image mapping relationships are established.

[0048] Furthermore, the image fusion in step S5 includes the following steps:

[0049] S51, judging whether the points on the image are located at overlapping positions according to the established mapping model;

[0050] If it is located in the overlapping position, then go to step S52;

[0051] If it is not at the overlapping position, proceed to step S54;

[0052] S52, calculate the point in all images S 1 , S 2 , …, S n The weight I′ in 1 , I′ 2 ,…I′ n ; n is the number of overlapping images where the point is located;

[0053] S53, the gray value R of the point on different images 1 , R 2 …R n and the corresponding weight I′ 1 , I′ 2 ,…I′ n Multiply them to get the gray value of the point after fusion Go to step S55;

[0054] S54, retain the gray value R of the point 1 As the gray value R of the point after fusion 0 =R 1 , proceed to step S55;

[0055] S55, obtain the gray value R corresponding to the point in the final spliced ​​image 0 ;

[0056] S56, processing all points in the mapping model according to steps S51-S54 respectively, to obtain a fused image.

[0057] Furthermore, the weight of the point in all the images to which it belongs in step S52 is calculated as follows:

[0058] S521, calculate the point (x, y) in all images to which the point belongs, and the image center point O corresponding to it on the corresponding image 1 (x 1 ,y 1 ), O 2 (x 2 ,y 2), …, O n (x n ,y n ) n is the number of overlapping images where the point is located;

[0059] S522, calculate the weight corresponding to the point

[0060] S523, repeat steps S521-S522 to obtain all the value I corresponding to the point 1 , I 2 ,…,I n ;

[0061] S524: Normalize all weights to obtain the final weight I′ of the point on each overlapping image. 1 , I′ 2 ,…I′ n .

[0062] Further, the step S6 of partial image display includes the following steps:

[0063] S61, looking up the angle value of the center point of the image to be displayed in the panoramic image stitching mapping table, assuming that the shooting angle corresponding to the center point of a certain local image to be displayed is the azimuth angle A and the pitch angle E;

[0064] S62, querying the mapping table according to the angle value of the pitch angle E, obtaining the corresponding vertical axis addressing reference, and marking the vertical coordinate of the pixel as Y;

[0065] S63, in the angle value table corresponding to the pitch angle E, calculate the 3° interval to which the azimuth angle A under the pitch angle belongs, and set the boundary azimuth angle value corresponding to the interval to A. n and A n+1 , compared with AA n and A n+1- The absolute value of A, if AA n If the absolute value of is smaller, An is used as the addressing reference. Otherwise, A is used as the addressing reference. n+1 For addressing reference, mark the pixel horizontal coordinate of the addressing reference as X;

[0066] S64, after determining the addressing reference of the azimuth angle, it is necessary to perform secondary addressing on the pixel coordinates, and use the calculation method of the miss amount to obtain the pixel coordinate offset ΔX and ΔY of the point relative to the addressing reference, thereby obtaining the pixel coordinate position (X+ΔX, Y+ΔY) of the point on the panoramic image;

[0067] S65, taking the pixel point obtained by the query as the center of the displayed image, and selecting half of the image display size on the horizontal and vertical axes of the center point for display.

[0068] Beneficial effects brought by the technical solution of the present invention:

[0069] The image stitching method designed by the present invention uses only a single set of photoelectric theodolite for shooting. Compared with the commonly used multi-camera shooting stitching method, it saves the equipment mobilization cost, overcomes the influence caused by the system error between different devices, and improves the imaging accuracy of the panoramic image;

[0070] A panoramic image stitching method that conforms to the actual shooting of the photoelectric theodolite is designed, which overcomes the problem of being unable to use feature point registration due to the lack of feature markers on site during the image registration process. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 A flowchart of a panoramic image stitching method implemented in an optical simulation training system provided by the present invention;

[0072] Figure 2 Image preprocessing flow chart in the optical simulation training system provided by the present invention;

[0073] Figure 3 Image registration flow chart in the optical simulation training system provided by the present invention;

[0074] Figure 4 Image fusion flow chart in the optical simulation training system provided by the present invention. DETAILED DESCRIPTION

[0075] The specific implementation of the panoramic image stitching method in the optical simulation training system of the present invention is described in detail below with reference to the accompanying drawings.

[0076] refer to Figure 1 , an embodiment of the present invention provides a panoramic image stitching method in an optical simulation training system, comprising the following steps:

[0077] S1. The photoelectric theodolite takes pictures according to the preset theoretical trajectory to complete data collection;

[0078] S2, preprocessing the acquired images;

[0079] S3, establish the correlation mapping model between images to achieve image registration;

[0080] S4, establishing a panoramic image stitching mapping table;

[0081] S5. According to the mapping relationship, the images are fused to form a panoramic image;

[0082] S6. According to the required display angle, the panoramic image stitching mapping table is searched through a secondary addressing method to realize partial image display.

[0083] It should be noted that the theoretical trajectory preset in step S1 is the angle guidance data of the photoelectric theodolite, which is written according to the required field of view. Because this method has no requirements for real-time performance, it is possible to select windless weather for shooting, so as to artificially reduce the interference of weather conditions on the image effect as much as possible. When shooting, keep the photoelectric theodolite in place, and the preset theoretical trajectory will guide the photoelectric theodolite to rotate 360 ​​degrees at a constant speed at each fixed pitch angle in the order of pitch angle from low to high. According to the field of view size of the captured image, ensure that there is an overlapping area in the range of the captured video image at adjacent azimuths and pitch angles.

[0084] Figure 2 FIG. 1 is a flowchart of a specific implementation of an image stitching method in one embodiment of the present invention. Figure 2 As shown, the preprocessing in step S2 of the image stitching method may include the following steps:

[0085] S21, judging the collected image;

[0086] If it is an infrared image, proceed to step S22;

[0087] If it is a visible light image, proceed to step S24;

[0088] S22, performing image enhancement on the infrared image;

[0089] S23, brightness enhancement, proceed to step S24;

[0090] S24, performing denoising processing on the collected image by using a median filtering method.

[0091] Figure 3 FIG. 1 is a flowchart of a specific implementation of an image stitching method in one embodiment of the present invention. Figure 3 As shown, the image registration in step S3 of the image stitching method may include the following steps:

[0092] S31, estimating the size of the panoramic image according to the field of view (horizontally α, longitudinally β) and the pixel size μ of the photoelectric theodolite (horizontally: Width=360 / (α·μ), longitudinally: Height=90 / (β·μ));

[0093] S32, allocate a space of size Width*Height in the memory, recorded as the spliced ​​image M;

[0094] S33, reading the video image taken by the photoelectric theodolite frame by frame, and obtaining the encoder angle, pixel size and other information attached to the file header information of the image;

[0095] S34, according to the synthesis angle of each pixel point in the image, reversely calculate the position information of the point in the spliced ​​image M, and write it into M;

[0096] S35, repeating steps S33-S34 until the positions of all pixels in the video image in the spliced ​​image M are obtained, thereby establishing a relationship mapping model between images.

[0097] Further, in this embodiment, in step S34, according to the synthesis angle of each pixel point in the image, the position information of the point in the spliced ​​image M is calculated inversely, including the following steps:

[0098] S341. Establish a formula for synthetic target angle, where A represents the azimuth angle, E represents the elevation angle, and f represents the focal length:

[0099]

[0100] A=A 0 +ΔA

[0101]

[0102] S342. Simplify the formula to obtain an approximate formula for the synthetic target angle:

[0103]

[0104]

[0105] S343. According to the above formula, if the angular offset of a point relative to the center point is known, the positional relationship between the two points can be inversely calculated to achieve the conversion of the angle value (A, E) and coordinate value (x, y) of any point on the target surface.

[0106] Figure 4 FIG. 1 is a flowchart of a specific implementation of an image stitching method in one embodiment of the present invention. Figure 4 As shown, the image fusion in step S5 of the image stitching method includes the following steps:

[0107] S51, judging whether the points on the image are located at overlapping positions according to the established mapping model;

[0108] If it is located in the overlapping position, then go to step S52;

[0109] If it is not at the overlapping position, proceed to step S54;

[0110] S52, calculate the point in all images S 1 , S 2 , …, S n The weight I′ in 1 , I′ 2,…I′ n ; n is the number of overlapping images where the point is located;

[0111] S53, the gray value R of the point on different images 1 , R 2 …R n and the corresponding weight I′ 1 , I′ 2 ,…I′ n Multiply them to get the gray value of the point after fusion Go to step S55;

[0112] S54, retain the gray value R of the point 1 As the gray value R of the point after fusion 0 =R 1 , proceed to step S55;

[0113] S55, obtain the gray value R corresponding to the point in the final spliced ​​image 0 ;

[0114] S56, processing all points in the mapping model according to steps S51-S54 respectively, to obtain a fused image.

[0115] Furthermore, the weight of the point in all the images to which it belongs in step S52 is calculated as follows:

[0116] S521, calculate the point (x, y) in all images to which the point belongs, and the image center point O corresponding to it on the corresponding image 1 (x 1 ,y 1 ), O 2 (x 2 ,y 2 ), …, O n (x n ,y n ) n is the number of overlapping images where the point is located;

[0117] S522, calculate the weight corresponding to the point

[0118] S523, repeat steps S521-S522 to obtain all the value I corresponding to the point 1 ,I 2 ,…,I n ;

[0119] S524: Normalize all weights to obtain the final weight I′ of the point on each overlapping image. 1 , I′ 2 ,…I′n .

[0120] Further, in this embodiment, the partial display of the image in step S6 may include the following steps:

[0121] S61, looking up the angle value of the center point of the image to be displayed in the panoramic image stitching mapping table, assuming that the shooting angle corresponding to the center point of a certain local image to be displayed is the azimuth angle A and the pitch angle E;

[0122] S62, querying the mapping table according to the angle value of the pitch angle E, obtaining the corresponding vertical axis addressing reference, and marking the vertical coordinate of the pixel as Y;

[0123] S63, in the angle value table corresponding to the pitch angle E, calculate and obtain the 3° interval to which the azimuth angle A under the pitch angle belongs, and the minimum and maximum azimuth angle values ​​corresponding to the interval are A n and A n+1 , compared with AA n and A n+1- The absolute value of A, if AA n The absolute value of A is smaller. n As the addressing reference, otherwise, A n+1 For addressing reference, mark the pixel horizontal coordinate of the addressing reference as X;

[0124] S64, after determining the addressing reference of the azimuth angle, it is necessary to perform secondary addressing on the pixel coordinates, and use the calculation method of the miss amount to obtain the pixel coordinate offset ΔX and ΔY of the point relative to the addressing reference, thereby obtaining the pixel coordinate position (X+ΔX, Y+ΔY) of the point on the panoramic image;

[0125] S65, taking the pixel point obtained by the query as the center of the displayed image, and selecting half of the image display size on the horizontal and vertical axes of the center point for display.

[0126] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A panoramic image stitching method in an optical simulation training system, characterized in that: The following steps are involved: S1. The photoelectric theodolite takes pictures according to the preset theoretical trajectory to complete data collection; S2, preprocessing the acquired images; The pre-processing in step S2 comprises the following steps: S21, judging the collected image; If it is an infrared image, proceed to step S22; If it is a visible light image, proceed to step S24; S22, performing image enhancement on the infrared image; S23, brightness enhancement, proceed to step S24; S24, performing denoising processing on the collected image using a median filter method; S3, establish the correlation mapping model between images to achieve image registration; The step S3 of establishing the association mapping model between images includes the following steps: S31, estimating the size of the panoramic image according to the field of view (horizontally α, longitudinally β) and the pixel size μ of the photoelectric theodolite, horizontally: Width = 360 / (α·μ), longitudinally: Height = 90 / (β·μ); S32, allocate a space of size Width*Height in the memory, recorded as the spliced ​​image M; S33, reading the video image taken by the photoelectric theodolite frame by frame, and obtaining the encoder angle, pixel size and other information attached to the file header information of the image; S34, according to the synthesis angle of each pixel point in the image, reversely calculate the position information of the point in the spliced ​​image M, and write it into M; S35, repeating steps S33-S34 until the positions of all pixels in the video image in the spliced ​​image M are obtained, thereby establishing a relationship mapping model between images; S4, establishing a panoramic image stitching mapping table; The step S4 of establishing the panoramic image stitching mapping table includes the following steps: S41. During the splicing process, record the corresponding pitch angle value; S42, the pitch angle starts from 0° and is separated by 0.5° as a pitch angle band, and in each angle band, the azimuth angle starts from 0° and is separated by 3° to record the angle value and pixel position of the nearest real image corresponding to the degree; S43, repeat S41 and S42 until all image mapping relationships are established; S5. According to the mapping relationship, the images are fused to form a panoramic image; The image fusion in step S5 includes the following steps: S51, judging whether the points on the image are located at overlapping positions according to the established mapping model; If it is located in the overlapping position, then go to step S52; If it is not at the overlapping position, proceed to step S54; S52, calculate the point in all images S1, S2, ..., S n The weights I′1, I′2, …I′ n ; n is the number of overlapping images where the point is located; S53, the grayscale values ​​R1, R2, ..., R of the point on different images are n and the corresponding weights I′1, I′2, …I′ n Multiply them to get the gray value of the point after fusion Go to step S55; S54, retain the gray value R1 of the point as the gray value R0=R1 of the point after fusion, and go to step S55; S55, obtaining the gray value R0 corresponding to the point in the final stitched image; S56, processing all points in the mapping model according to steps S51-S54 respectively, to obtain a fused image; S6. According to the required display angle, the panoramic image stitching mapping table is searched through a secondary addressing method to realize partial image display.

2. The panoramic image stitching method in the optical simulation training system according to claim 1, characterized in that: The theoretical trajectory preset in step S1 is the angle guidance data of the photoelectric theodolite.

3. The panoramic image stitching method in the optical simulation training system according to claim 2, characterized in that: In step S1, when shooting, the photoelectric theodolite is kept in a fixed position, and the preset theoretical trajectory guides the photoelectric theodolite to rotate 360 ​​degrees at a constant speed at each fixed pitch angle in the order of pitch angle from low to high, and according to the field of view size of the captured image, it is ensured that there is an overlapping area in the range of the captured video images at adjacent azimuths and pitch angles.

4. The panoramic image stitching method in the optical simulation training system according to claim 1, characterized in that: In step S34, the position information of each pixel in the image is calculated based on the synthesis angle of the pixel in the image, which includes the following steps: S341. The formula for establishing the synthetic target angle is as follows, where A represents the azimuth angle, E represents the elevation angle, and f represents the focal length: A=A0+ΔA S342. Simplify the formula to obtain an approximate formula for the synthetic target angle: S343. According to the above formula, if the angular offset of a point relative to the center point is known, the positional relationship between the two points can be inversely calculated to achieve the conversion of the angle value (A, E) and coordinate value (x, y) of any point on the image.

5. The panoramic image stitching method in the optical simulation training system according to claim 4, characterized in that: The weight of the point in all the images to which it belongs in step S52 is calculated as follows: S521, calculate the point (x, y) in all images to which the point belongs, and the image center points O1(x1, y1), O2(x2, y2), ..., O3(x3, y4), ..., O4(x4, y5), ..., O5(x5, y6), ..., O6(x6, y7), ..., O7(x7, y8), ..., O8(x8, y9), ..., O9(x9 n (x n ,y n ) n is the number of overlapping images where the point is located; i = 1, 2, ... n; S522, calculate the weight corresponding to the point S523, repeat steps S521-S522 to obtain all the value I1, I2, ..., I corresponding to the point n ; S524, normalize all weights to obtain the final weights I'1, I'2, ... I' of the point on each overlapping image. n .

6. The panoramic image stitching method in the optical simulation training system according to claim 1, characterized in that: The step S6 of partial image display comprises the following steps: S61, looking up the angle value of the center point of the image to be displayed in the panoramic image stitching mapping table, assuming that the shooting angle corresponding to the center point of a certain local image to be displayed is the azimuth angle A and the pitch angle E; S62, querying the mapping table according to the angle value of the pitch angle E, obtaining the corresponding vertical axis addressing reference, and marking the vertical coordinate of the pixel as Y; S63, in the angle value table corresponding to the pitch angle E, calculate the 3° interval to which the azimuth angle A under the pitch angle belongs, and set the boundary azimuth angle value corresponding to the interval to A. n and A n+1 , compared with AA n and A n+1 -The absolute value of A, if AA n The absolute value of A is smaller. n As the addressing reference, otherwise, A n+1 For addressing reference, mark the pixel horizontal coordinate of the addressing reference as X; S64, after determining the addressing reference of the azimuth angle, it is necessary to perform secondary addressing on the pixel coordinates, and use the calculation method of the miss amount to obtain the pixel coordinate offset ΔX and ΔY of the point relative to the addressing reference, thereby obtaining the pixel coordinate position (X+ΔX, Y+ΔY) of the point on the panoramic image; S65, taking the pixel point obtained by the query as the center of the displayed image, and selecting half of the image display size on the horizontal and vertical axes of the center point for display.

Images