A method of processing panoramic skew

Abstract

The application discloses a kind of methods for handling panorama skew, the present application relates to panoramic image processing technical field, comprising: S1: obtaining panorama to be handled, S2: panorama pretreatment, S3: interference object removal, S4: straight line extraction, S5: calculate x, y, z three main shaft direction, and determine correction matrix, S6: check calculation result, S7: correct original panorama to obtain target panorama.In the present application, the correction matrix calculation introduces a refined module, which can accurately solve the three main directions, and the correction result has high precision.In the present application, the refined module obtains an initial value through local optimization when iteratively solving, to speed up the convergence rate of iteration, prevent falling into local optimal solution, and does not rely on hardware for calibration.In the present application, interference factors are removed based on semantic information, and the result is checked based on semantic information, to further improve the robustness of the algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of panoramic image processing, and particularly relates to a method for processing panoramic skew. BACKGROUND

[0002] The method for processing panoramic image skew is an important technology in panoramic image processing. By correcting the panoramic image, the horizontal and vertical directions are kept consistent, thereby improving the authenticity and viewing effect of the panoramic image. There are usually the following ways:

[0003] I. Panoramic image shooting technology

[0004] The skew problem of the panoramic image is usually caused by camera posture, shooting angle or deviation of splicing position during shooting. Therefore, in order to avoid the skew problem of the panoramic image, a professional panoramic camera or panoramic gimbal is needed when shooting the panoramic image, and the horizontal plane and vertical plane of the camera need to be kept horizontal and vertical during shooting to minimize the posture error.

[0005] II. Panoramic image correction algorithm

[0006] The main method for processing the skew problem of the panoramic image is to correct the panoramic image through a panoramic image correction algorithm. The correction algorithm usually includes image transformation and image distortion correction steps.

[0007] III. Panoramic image processing software

[0008] There are many special panoramic image processing software on the market, such as PTGui, AutopanoPro, Hugin, etc. These software have functions of correcting, splicing and processing the panoramic image.

[0009] IV. Horizontal correction instrument

[0010] There are many panoramic cameras on the market, which have a horizontal correction instrument that can give the inclination angle relative to the horizontal plane to improve the shooting inclination problem.

[0011] The panoramic image shooting technology in the above centralized way has high requirements for the photographer and the shooting equipment.

[0012] The panoramic image processing software needs to be manually processed after shooting, which is low in efficiency.

[0013] The horizontal correction instrument has the problem of error accumulation, and the device needs to be corrected for error after being used for a period of time.

[0014] The panoramic image skew processing method provided in the present application refers to the optimization of the panoramic image correction algorithm. Although the existing panoramic image correction algorithm has achieved certain results, there are still the following shortcomings:

[0015] 1. Lack of accuracy

[0016] The accuracy of the algorithm can be affected by image noise and imaging model noise. A correction deviation of 2 degrees or more can result in unnatural image preview effects.

[0017] 2. Lack of robustness

[0018] In actual scenarios, panoramic image shooting can be affected by various factors such as light changes, object occlusion, image blur, etc. These factors can affect the robustness of the panoramic image rectification algorithm. Therefore, an algorithm with higher robustness is needed to handle these factors.

[0019] 3. High time complexity

[0020] Panoramic image rectification algorithms usually require feature extraction (points or lines), mathematical model calculation, projection transformation, and other steps, which have high computational complexity and take a long time. Especially for high-resolution panoramic image processing, the complexity is higher and the processing time is longer.

[0021] To solve the above problems, we provide a method for processing panoramic skew. SUMMARY

[0022] To solve the above problems of the prior art, the present application provides a method for processing panoramic skew, which solves the skew problem caused by camera pose, shooting angle or deviation of the splicing position during the splicing process of the panoramic image, thereby improving the authenticity and viewing effect of the panoramic image.

[0023] To achieve the above purpose, the present application adopts a method for processing panoramic skew, comprising the following steps:

[0024] S1: Obtain the panoramic image to be processed, and the method for obtaining the panoramic image to be processed includes but is not limited to obtaining the real-time shot panoramic image from the image shooting device, obtaining the stored panoramic image from the image library, and obtaining the real-time shot or pre-stored panoramic image from other electronic devices connected to the current electronic device for communication;

[0025] S2: Panoramic pre-processing, color enhancement processing is performed on the panoramic image to be processed;

[0026] S3: Interference object removal, scene semantics are segmented by training a neural network;

[0027] S4: Straight line extraction, the panoramic image is projected into a plurality of pinhole images with overlapping regions, straight line segments are extracted on each pinhole image to form a straight line segment set, and finally the straight line segments with overlapping regions in the straight line segment set are fused to obtain the final straight line segment set;

[0028] S5: Calculate the x, y, z three main axis directions, and determine the correction matrix, determine the x, y, z three main axis directions, can be converted to get the correction matrix;

[0029] S6: Check the calculation result, and use the determined semantics to assist in checking the result;

[0030] S7: Correct the original panorama to get the target panorama, use the correction matrix to act on the pixels of the panorama in turn, get the coordinates of the original panorama, and then use the pixel interpolation method to obtain the RGB color value, complete the data filling, and get the corrected target panorama after traversing all pixels.

[0031] As a further optimization of the above scheme, the S2 is enhanced for contrast, sharpness and definition.

[0032] As a further optimization of the above scheme, the sub-process in S4 is as follows:

[0033] S401: Panorama cutting pinhole graph, panorama view horizontal 360 degrees, vertical 180 degrees, horizontal direction is latitude, vertical direction is longitude, north and south poles, first determine the pinhole graph view parameter, then cut the pinhole graph along the 0 degree latitude circle evenly, ensure that the adjacent pinhole graphs have view overlap area, finally take one pinhole graph at the north and south poles, determine the pinhole graph view parameter, the longitude and latitude parameter of the imaging center, and use the pinhole imaging projection model to realize the projection transformation result from the local area of the panorama to the pinhole graph;

[0034] S402: Extract straight line segment, use any one of the straight line segment extraction algorithm in LSD algorithm and LBD algorithm to extract straight line segment from pinhole image, get the straight line segment set to be fused;

[0035] S403: Construct the normal of straight line segment and the field of view angle, project the two end points Pstart2D(xStart, yStart), Pend2D(xEnd, yEnd) of the straight line segment extracted from the pinhole graph to the panorama sphere, get two three-dimensional end points Pstart3D(xStart, yStart, zStart), Pend3D(xEnd, yEnd, zEnd), combine the center O of the panorama sphere, Pstart3D and Pend3D three points to form a plane. Calculate the normal of the plane, which is the normal of the straight line segment;

[0036] Calculate the circular arc length corresponding to the projection of the straight line segment on the panorama sphere, which is the field of view angle of the straight line segment. The longer the circular arc length after projection, the larger the field of view angle of the straight line segment;

[0037] S404: Merge line segments, take out two line segments from the line segment set, check if the normal angle is less than a given threshold, if less than the first threshold, and there is an overlap between the two line segments, i.e. line segment fusion, get a new line segment, adopt the above method to get the final line segment set {line i}.

[0038] As a further optimization of the above scheme, the sub-process in S5 is as follows:

[0039] S501: Estimate the main axis direction, adopt a method similar to Hough voting to estimate the main axis direction, grid the spherical surface according to the second threshold, and calculate the score of each gridded vertex with {line i} respectively. The calculation method is: when the angle between the line segment normal and the vertex vector is less than the third threshold, add the field of view angle of the line segment to the score of the vertex. Find 3 vertices from all the vertices of the second threshold gridded spherical surface, which satisfy the vector angle not less than the fourth threshold, and the sum of the scores of the 3 vertices is the maximum. The 3 vertex vectors are recorded as the main axis to be estimated

[0040] S502: Fine calculation of the main axis direction, calculate the angle between the normal of {line i} line segment and The result greater than the third threshold is recorded as the first sub-line segment set {linex i}、{liney i}、linez i}, where {linex i}、{liney i}、{linez i} are the first sub-line segment sets of x-axis, y-axis and z-axis respectively. The normal sets corresponding to these first sub-line segments are recorded as The corresponding field of view angle sets are recorded as {FovLinex i}、{FovLiney i}、{FovLinez i} respectively. Extract the maximum value in the field of view angle set respectively, and do normalization to get the optimization weight set, recorded as WeightLinex i 、WeightLiney i 、WeightLinez i ;

[0041] Let be the three directions after fine optimization, construct the objective function:

[0042]

[0043] where the symbol "·" represents the vector dot product, and the solution is that is, the objective function F reaches the minimum, and satisfies the condition that the vector module length is 1 and orthogonal to each other, as follows:

[0044]

[0045] S503: Determine the correction matrix, which is a 3x3 rotation matrix, representing the transformation method from the corrected coordinate system to the uncorrected coordinate system, removing the rotation amount around the z-axis, ensuring that the horizontal view of the panoramic skew correction has no change, calculating the Euler angle through the three principal directions, and obtaining the rotation component θ around the z-axis rotZ , construct a rotation matrix M around the z-axis by-θ rotZ , the correction matrix M RotZ , the correction matrix M Adjust is calculated as follows:

[0046]

[0047] As a further optimization of the above scheme, the principal axis direction needs to be iterated, and each iteration improves the effective search range by narrowing the threshold based on the last iteration, making the result more accurate.

[0048] As a further optimization of the above scheme, the Lagrange multiplier method and the Newton iteration method for solving non-linear equations are used, and when the Newton iteration method for solving non-linear equations is used, the initial values of the x-axis, y-axis and z-axis to be estimated need to be set

[0049] As a further optimization of the above scheme, the local optimization method is used to obtain The following objective function is constructed:

[0050]

[0051] The least square method is used to obtain Then the following objective function is constructed:

[0052]

[0053] The least square method is used to obtain

[0054] Finally

[0055] As a further optimization of the above scheme, the specific steps of using the determined semantics to assist in verifying the result in S6 are as follows:

[0056] S601: Calculate the angle between the normal of each straight line segment of {line i} and the horizontal plane, and record the result that is not less than the fifth threshold value as a second sub straight line segment set {linez }; i

[0057] S602: Obtain the semantic information of the panoramic image to be corrected;

[0058] S603: Determine whether a third sub straight line segment set {lineZInSemantic i} in {linez i} contains the above-mentioned semantic information;

[0059] S604: Calculate the total field angle of the third sub straight line segment set {lineZInSemantic i}, and if the value is lower than the sixth threshold value, it indicates that the correction result fails.

[0060] The method for processing panoramic skew of the present application has the following beneficial effects:

[0061] 1. The method for processing panoramic skew of the present application introduces an actuarial module in the correction matrix calculation, which can accurately solve three main directions and has high correction result accuracy.

[0062] 2. In the present application, the actuarial module obtains initial values through local optimization when iteratively solving, so as to accelerate the iteration convergence speed, prevent falling into a local optimal solution, and calibrate without relying on hardware.

[0063] 3. In the present application, interference factors are removed based on semantic information, and the result is verified based on semantic information, which further improves the robustness of the algorithm.

[0064] The specific embodiments of the present application are disclosed in detail in the following description and drawings, which indicate the principles of the present application that can be adopted, and it should be understood that the embodiments of the present application are not limited in scope, and the embodiments of the present application include many changes, modifications and equivalents within the scope of the appended claims and clauses. BRIEF DESCRIPTION OF DRAWINGS

[0065] Fig. 1 The flowchart of the method for processing panoramic skew of the present application;

[0066] Fig. 2 The flowchart of the method for processing panoramic skew of the present application;

[0067] Fig. 3 The flowchart of the method for processing panoramic skew of the present application; DETAILED DESCRIPTION

[0068] ​In order to make the objects, technical solutions and advantages of the present application clearer, the following further describes the present application in detail with reference to the accompanying drawings and examples. However, it should be understood that the specific examples described herein are only used to explain the present application and are not intended to limit the scope of the present application.

[0069] It should be noted that when an element is referred to as "provided on" or "provided with" another element, it can be directly on the other element or there can be a middle element, and when an element is referred to as "connected to" or "connected with" another element, it can be directly connected to the other element or there can be a middle element, and "fixedly connected" means fixed connection, and there are many ways of fixed connection, which are not intended to limit the scope of the present application, and the terms "vertical", "horizontal", "left", "right" and the like used herein are only for the purpose of illustration and do not represent the only implementation.

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs, and the terms used herein in the specification are only for the purpose of describing specific embodiments and are not intended to limit the present application, and the term "and / or" used herein includes any and all combinations of one or more related listed items;

[0071] Please refer to the accompanying drawings of the specification Figs. 1-3 The present application provides a technical solution: a method for processing panoramic skew, comprising the following processes:

[0072] 1. Obtain a panoramic image to be processed. The projection type of the panoramic image to be processed can be any one of cylindrical projection, equidistant projection, equirectangular projection, pinhole projection type, and the present application does not limit the projection type of the panoramic image to be processed. The present application can obtain a real-time captured panoramic image from an image capturing device, can obtain a stored panoramic image from a gallery, and can obtain a real-time captured or pre-stored panoramic image from another electronic device connected and communicated with the current electronic device. The present application does not limit the specific way of obtaining the panoramic image to be processed.

[0073] 2. Panoramic pre-processing. Color enhancement processing is performed on the panoramic image to be processed, such as contrast, sharpness, and definition enhancement processing. The present application does not limit the specific enhancement method used, and the goal is to improve the stability of the subsequent algorithm for extracting line segments.

[0074] 3. Interference object removal. There are irregular oblique lines in the actual scene (such as curtains, stair railings, etc.). When the length of these oblique lines is relatively long, it will affect the robustness of the skew correction algorithm. The present application trains a neural network to segment the scene semantics, removes elements such as curtains and stair railings to improve the robustness of the subsequent algorithm. The present application does not limit the specific semantic segmentation neural network used.

[0075] 4. Linear extraction. Since the imaging field of view of the panoramic image is 360 degrees horizontally and 180 degrees vertically, most of the straight lines in the actual scene are curved after panoramic projection imaging, so it is difficult to directly extract straight line segments from the panoramic image. The panoramic image is projected into a plurality of pinhole images with overlapping regions in the present application, and straight line segments are extracted on each pinhole image to form a straight line segment set. Finally, the straight line segments with overlapping regions in the straight line segment set are fused to obtain the final straight line segment set. The sub-process is as follows:

[0076] 4.1 Panoramic image cutting pinhole image. The field of view of the panoramic image is 360 degrees horizontally and 180 degrees vertically. It can be assumed that it is a world map, the horizontal direction is the latitude, the vertical direction is the longitude, and the north and south poles are the earth.

[0077] First, determine the pinhole image field of view parameters, then evenly cut the pinhole image along the 0-degree latitude circle, and ensure that there is a field of view overlap between adjacent pinhole images. In a similar way, take several pinhole images on the mid-high latitude circle to ensure that there is a field of view overlap between adjacent pinhole images in the latitude and longitude directions. For example: the pinhole image field of view is set to 90 degrees; evenly cut 8 pinhole images on the 0-degree latitude circle, then the field of view overlap of each pinhole image is 45 degrees; evenly cut 6-8 times on the 45-degree latitude circle. It can be ensured that there is a field of view overlap between adjacent pinhole images in the latitude and longitude directions; finally, take one pinhole image on the north and south poles.

[0078] After determining the pinhole image field of view parameters and the latitude and longitude parameters of the imaging center, the pinhole imaging projection model can be used to realize the projection transformation result from the panoramic image to the pinhole image.

[0079] The present application does not limit the pinhole image field of view and the number of pinhole images cut along the latitude and longitude directions, as long as there is enough field of view overlap between adjacent pinhole images to facilitate later straight line segment fusion.

[0080] 4.2 Extracting straight line segments. The present application can use any of the straight line segment extraction algorithms such as the LSD (Line Segment Detector) algorithm, the LBD (Line Band Discriptor) algorithm, etc. to extract straight line segments from the pinhole image to obtain the straight line segment set to be fused.

[0081] 4.3 Construct the normal of the line segment and the field of view angle. First introduce the panorama sphere. Panorama sphere can be understood as a globe, the camera is set at the center of the globe, and then rotate around the center to take several pictures, which will cover a horizontal 360-degree, vertical 180-degree field of view. Splicing these pictures can build a panorama. Relative to the globe, the panorama can be considered as a world map. Any object point in the environment can find the corresponding image point on the panorama, and also find the corresponding projection point on the panorama sphere. Since the globe can be large or small, the radius is not fixed, in order to unify the subsequent calculation, the radius of the panorama sphere is specified as 1, that is, the unit sphere.

[0082] Project the two end points Pstart2D(xStart, yStart), Pend2D(xEnd, yEnd) of the line segment extracted from the pinhole diagram onto the panorama sphere to obtain two three-dimensional end points Pstart3D(xStart, yStart, zStart), Pend3D(xEnd, yEnd, zEnd). Combine the three points of the panorama sphere center O, Pstart3D, and Pend3D to form a plane. Calculate the normal of the plane, denoted as the normal of the line segment.

[0083] Calculate the circular arc length corresponding to the projection of the line segment on the panorama sphere, denoted as the field of view angle of the line segment. The longer the circular arc length after projection, the larger the field of view angle of the line segment.

[0084] 4.4 Fusion of line segments. Take out two line segments from the line segment set, check if the angle between their normals is less than a given threshold. If it is less than the first threshold, and there is overlap between the two line segments, that is, line segment fusion is performed to obtain a new line segment. Use the above method to fuse multiple times to obtain the final line segment set {line i}, the first threshold is the angle between the two direction vectors, and the first threshold is used to determine the parallel relationship between the two direction vectors. The smaller the first threshold and the closer to 0 degrees, the closer the two direction vectors are to parallel, and the closer the normal is to parallel, the closer the corresponding line is to overlap.

[0085] 5. Calculate the three principal axis directions x, y, z and determine the correction matrix. Take the indoor scene as an example: most indoor scenes meet the Manhattan structure, that is, the house structure and the furniture arrangement contain a large number of parallel, orthogonal, coplanar and other line, surface relationships. The three principal axis directions are established in the local coordinate system of the house: the principal axis z is the direction perpendicular to the ground. The walls, furniture and the like generally contain a large number of straight lines perpendicular to the ground, which are parallel to the principal axis z; from the orthogonal walls or the orthogonal furniture side surfaces, a direction is selected as the principal axis x, and the other orthogonal direction is the y axis. Because the furniture is generally placed parallel or perpendicular to the wall, a large number of straight lines parallel to the x and y axes may exist in the walls, furniture, doors, windows, ground and the like. Determining the three principal axis directions x, y, z, that is, finding the three directions that can collect most of the parallel straight lines in the scene, the correction matrix can be obtained.

[0086] The sub-process is as follows: the principal axis direction is estimated and refined, and each iteration is based on the last iteration to improve the effective search range by reducing the threshold, so that the result is more accurate. The number of iterations should not be too large, and the target is to effectively resist noise and retain accuracy.

[0087] Note: If there are no straight lines perpendicular to the ground in the scene, such as walls or furniture placed obliquely on the ground, the principal axis z cannot be perpendicular to the ground after construction, and the skew correction effect of the present application will be incorrect; if the principal axis z is calculated correctly, but the walls in the scene are not regularly constructed or the furniture is placed randomly, the calculated principal axes x and y can only represent the direction of the small range of parallel straight lines, and have no effect on the skew correction effect of the present application. Why increase the calculation of x and y axes? First, the data noise involved in the calculation of the principal axis z is reduced by increasing the data of two directions; second, the correct and effective calculation results of the x and y principal axes are helpful to the subsequent house structure estimation, point registration and the like.

[0088] 5.1 Estimate the principal axis direction. A method similar to Hough voting is used to estimate the principal axis direction. The second threshold is used to grid the spherical surface, and each gridded vertex is calculated with {line i} to get the score. The calculation method is: when the angle between the straight line segment normal and the vertex vector is less than the third threshold, the field of view angle of the straight line segment is added to the score of the vertex. From all the vertices on the spherical surface, find three vertices that meet the following conditions: the vector angle between each other is not less than the fourth threshold, and the sum of the scores of the three vertices is the maximum. Then the three vertex vectors are denoted as the x axis y axis

[0089] z axis The second threshold value represents a variable of an angle, which is used to discretize the sphere into triangular mesh data, and the variable can control the mesh fineness, i.e., the approximation degree of triangular patches to the sphere; the fourth threshold value represents the size of an included angle between two direction vectors, and the fourth threshold value is used to determine the perpendicular relationship between the direction vectors, and the closer to 90 degrees, the closer to perpendicular the two direction vectors are.

[0090] 5.2 Actuarial principal axis direction. Calculate i The included angle between the normal of the straight line segment and The result greater than the third threshold value is recorded as a first sub straight line segment set {line i}、{liney i}、linez i}, wherein the first sub straight line segment sets of the x-axis, the y-axis and the z-axis are respectively denoted as i}、{liney i}、{linez i}, and the corresponding field of view angle sets are respectively denoted as {FovLinex i}、{FovLiney i}、{FovLinez i}. The maximum values in the field of view angle sets are respectively extracted, normalized, and the optimized weight sets are obtained, which are respectively denoted as WeightLinex i 、WeightLiney i 、WeightLinez i The third threshold value represents the size of an included angle between two direction vectors, and the third threshold value is used to determine the parallel relationship between the direction vectors, and the closer to 0 degrees, the closer to parallel the two direction vectors are.

[0091] Let be the three directions after fine optimization. The objective function is constructed as follows:

[0092]

[0093] Wherein, the symbol “·” represents vector dot product. The solution is obtained, i.e., the objective function F reaches the minimum, and satisfies the conditions that the vector module length is 1 and the vectors are orthogonal to each other, as follows:

[0094]

[0095] This problem belongs to a constrained optimization problem, which can be solved by using the Lagrange multiplier method and the Newton iteration method of nonlinear equations. When the Newton iteration method of nonlinear equations is used for solving, an initial value needs to be set The present application does not use the initial value as an iterative solution To accelerate the convergence speed of iteration and prevent falling into local optimal solution, the present application adopts a local optimization method to obtain Firstly, the following objective function is constructed:

[0096]

[0097] The least square method is used to obtain Then, the following objective function is constructed:

[0098]

[0099] The least square method is used to obtain

[0100] Finally

[0101] 5.3 Determine the correction matrix. The correction matrix is a 3x3 rotation matrix, which represents the transformation method from the corrected coordinate system to the uncorrected coordinate system. Here, the rotation amount around the z-axis is removed to ensure that the view field is not changed after the panoramic skew correction. The Euler angle is calculated through the three principal directions, and the rotation component θ rotZ around the z-axis is obtained. The matrix M rotZ around the z-axis is rotated by-θ RotZ . The correction matrix M Adjust is calculated as follows:

[0102]

[0103] 6. Check the calculation result. Step 3 can remove explicit interference objects using semantics, but cannot remove objects with unclear semantics or difficult to train and label, such as: wooden boards leaning against the wall at an angle or crossing irregular cables in the air, etc. These objects can provide clear straight lines, but do not meet the mutual parallel, orthogonal and other constraints, so they will have a negative impact on the correction result, and even cause errors. The present application uses determined semantics to assist in checking the result, and the steps are as follows:

[0104] 6.1 Calculate the angle between the normal of each straight line segment of {line i} and , and record the second sub-straight line segment set {linez i} if the result is not less than the fifth threshold value. The fifth threshold value represents the angle between two direction vectors, and the fifth threshold value is used to determine the perpendicular relationship between the direction vectors. If the angle is not less than the fifth threshold value and closer to 90 degrees, it means that the two direction vectors are closer to perpendicular, and the normal is closer to perpendicular, so the corresponding straight line segment is more likely to be perpendicular to the ground.

[0105] 6.2 Obtain semantic information of the panorama to be corrected, and extract walls, doors, windows and other semantics. These objects contain multiple straight line segments perpendicular to the ground.

[0106] 6.3 Determine whether the third sub-line segment set {lineZInSemantic i} in the above semantics is contained in the third sub-line segment set {lineZInSemantic i}.

[0107] 6.4 Calculate the total field of view angle of the third sub-line segment set {lineZInSemantic i}. If the value is lower than the sixth threshold value, it indicates that the correction result fails; the sixth threshold value represents the total sum of the field of view angles corresponding to all straight line segments in the third sub-line segment set, and the greater the value, the more effective vertical lines are extracted, and the higher the confidence of the correction result.

[0108] Note: The extraction of {line i} in step d cannot be performed only on the semantics of walls, doors, windows and the like, so as to prevent the anti-noise ability from being reduced and the accuracy of the correction result from being insufficient due to insufficient number of extracted straight line segments.

[0109] 7. Correct the original panorama to obtain a target panorama. The correction matrix is used to act on the pixels of the panorama in sequence to obtain the coordinates of the original panorama, and then the pixel interpolation method is used to obtain the RGB color values, and data filling is completed. After all the pixels are traversed, the corrected target panorama is obtained.

[0110] In the present application, the calculation of the correction matrix introduces an actuarial module, which can accurately solve three main directions, and the correction result has high accuracy. In the present application, the actuarial module obtains an initial value through local optimization when iteratively solving, so as to accelerate the convergence speed of iteration, prevent falling into a local optimal solution, and does not rely on hardware for calibration. In the present application, the interference factors are removed based on semantic information, and the result is verified based on semantic information, which further improves the robustness of the algorithm.

[0111] The above merely describes the preferred embodiments of the present application and should not be used to limit the present application, and any modification, equivalent replacement or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method of processing panoramic skew, characterized by, Comprise the following steps: S1: acquire the panorama to be processed, the way to acquire the panorama to be processed includes but is not limited to acquiring the real-time shot panorama from the image shooting device, acquiring the stored panorama from the gallery, acquiring the real-time shot or pre-stored panorama from other electronic devices connected with the current electronic device; S2: panorama preprocessing, color enhancement processing is carried out for the panorama to be processed; S3: interference object removal, the scene semantics is segmented by training the neural network; S4: straight line extraction, project the panoramic image into multiple pinhole images with overlapping regions, extract straight line segments on each pinhole image to form a straight line segment set, and finally fuse the straight line segments with overlapping regions in the straight line segment set to obtain a straight line set straight line segment set S5: calculate the x, y, z three main shaft directions, and determine the correction matrix, determine the x, y, z three main shaft directions, and the correction matrix can be obtained by transformation; The sub-process in S5 is as follows: S501: estimate the principal axis direction, estimate the principal axis direction in a similar way to Hough voting, grid the sphere according to the second threshold value, and respectively add the field of view angle of each linear segment to the score of the vertex Calculate the score: when the angle between the normal of the linear segment and the vertex vector is less than the third threshold value, add the field of view angle of the linear segment to the score of the vertex. From all the vertices of the second threshold value grid sphere, find three vertices that satisfy the vector angle not less than the fourth threshold value, and the sum of the scores of the three vertices is the maximum. The three vertex vectors are the principal axis to be estimated 、 、 ; S502: Calculate the main axis direction The angle between the normal of each line segment in the line segment set and , , The result greater than the third threshold is recorded as the first sub-line segment set , , Wherein, , , The first sub-line segment set of x-axis, y-axis and z-axis respectively, the normal set corresponding to these first sub-line segments is recorded as , , The corresponding field of view angle set is recorded as , , The maximum value in the field of view angle set is extracted respectively, and the normalization processing is done to obtain the optimization weight set, which is recorded as , , ; Set , , are three directions of fine optimization, and the objective function is constructed: where the symbol represents the vector dot product, solving , , i.e. the objective function F is minimized and the condition that the vector modulus is 1, orthogonal to each other, as follows: ； S503: determine a correction matrix, the correction matrix is a 3x3 rotation matrix, which represents a transformation method from the corrected coordinate system to the uncorrected coordinate system, removes the rotation amount around the z axis, ensures that the view field after panorama skew correction is horizontal and has no change, calculates the Euler angle through three main directions, and obtains the rotation component around the z axis , constructs a rotation matrix around the z axis The correction matrix is calculated as follows: ； S6: check the calculation result, and use the determined semantics to assist the checking result; S7: correct the original panorama to obtain the target panorama, use the correction matrix to act on the pixels of the panorama in turn, obtain the coordinates of the original panorama, and then use the pixel interpolation method to obtain the RGB color value, complete the data filling, and obtain the corrected target panorama after traversing all the pixels.

2. A method of processing a panoramic skew as claimed in claim 1, characterized in that: The contrast, sharpness and definition are enhanced in S2.

3. A method of processing a panoramic skew as claimed in claim 1, characterized in that: The sub-process in S4 is as follows: S401: panorama cutting pinhole image, the horizontal direction of the panorama is 360 degrees, the vertical direction is 180 degrees, the horizontal direction is latitude, the vertical direction is longitude, the north and south poles are up and down, first determine the pinhole image view parameter, then cut the pinhole image along the 0 degree latitude circle evenly, ensure that there is a view overlap area between adjacent pinhole images, finally take one pinhole image at the north and south poles, determine the longitude and latitude parameters of the pinhole imaging center, and then use the pinhole imaging projection model to realize the projection transformation result from the local area of the panorama to the pinhole image; S402: extract straight line segment, use any one of the straight line segment extraction algorithms in the LSD algorithm and the LBD algorithm to extract the straight line segment of the pinhole image, and obtain the straight line segment set to be fused; S403: construct the normal of the straight line segment and the field of view angle, project the two end points Pstart2D (xStart, yStart), Pend2D (xEnd, yEnd) of the straight line segment extracted from the pinhole image onto the panorama sphere to obtain two three-dimensional end points Pstart3D (xStart, yStart, zStart), Pend3D (xEnd, yEnd, zEnd), combine the three points of the panorama sphere center O, Pstart3D and Pend3D to form a plane, calculate the normal of the plane, and record it as the normal of the straight line segment; Calculate the circular arc length corresponding to the straight line segment on the panorama sphere, and record it as the field of view angle of the straight line segment. The longer the circular arc length after projection is, the larger the field of view angle of the straight line segment is. S404: fusion of straight line segments, two straight line segments are taken out from the straight line segment set, it is checked whether the normal angle is less than a given threshold value, if it is less than the first threshold value, and there is an overlap between the two line segments, i.e. line segment fusion is performed, a new straight line segment is obtained, and the above method is used for multiple fusion to obtain the final straight line segment set .

4. The method of claim 1, wherein: The main shaft direction needs to be iterated, and each iteration is based on the last iteration to improve the effective search range by reducing the threshold, so that the result is more accurate.

5. The method of claim 1, wherein: The Lagrange multiplier method and the Newton iteration method of nonlinear equation are used for solving, and when the Newton iteration method of nonlinear equation is used for solving, initial values of the x-axis, the y-axis and the z-axis to be estimated need to be set , , .

6. A method of processing a panoramic skew as claimed in claim 5, characterized in that: The local optimization method is adopted to obtain , , , and the following objective function is constructed: The least square method is used to solve the equation Then, the following objective function is constructed The least square method is used to solve the equation , Finally .

7. A method of processing a panoramic skew as claimed in claim 5, characterized in that: The specific steps of using the determined semantics to assist the checking result in S6 are as follows: S601: Calculate The angle between the normal of each straight line segment of the straight line segment set and The result is not less than the fifth threshold value, and is recorded as the second sub-straight line segment set ; S602: acquire the semantic information of the panorama to be corrected; S603: judging The third sub-line segment set contained in the above semantics ; S604: Calculate If the sum of the field angles of the third set of straight line segments is lower than the sixth threshold value, it indicates that the correction result has failed.

Images