Omnidirectional image processing method and device and computer readable storage medium

Abstract

The invention provides an omnidirectional image processing method, an omnidirectional image processing device and a computer readable storage medium. The method comprises the following steps: acquiring an omnidirectional image; projecting the omnidirectional image to a pinhole camera model to obtain a pinhole image; setting a mask value of a pixel point with a corresponding projection point in the pinhole image in the omnidirectional image as a first preset value, and setting mask values of other pixel points of the omnidirectional image as a second preset value; and performing mask processing on the omni-directional image by using the mask value of each pixel point in the omni-directional image. Through the mode, the processing device uses the abstract pinhole camera model to describe the mapping relation of the omnidirectional camera, image information can be reserved as far as possible, and pixel distortion is reduced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an omnidirectional image processing method, an omnidirectional image processing device, and a computer-readable storage medium. Background Art

[0002] Most commercial cameras can be described as pinhole cameras, modeled by traditional perspective projection. However, the projection geometry of some cameras cannot be described using the traditional pinhole model, such as omnidirectional cameras, which introduce very high pixel distortion during imaging. Omnidirectional cameras have a 180-degree field of view in the horizontal plane, and a single camera can cover a hemisphere or even larger than a hemisphere.

[0003] There are two main solutions for traditional visual inertial tracking and positioning systems to process omnidirectional camera images:

[0004] Map part of the omnidirectional image into a pinhole perspective image, and discard the rest of the image that cannot be observed by the pinhole camera model. This solution has two obvious problems: 1. In terms of computational accuracy, the process of converting from an omnidirectional camera to a pinhole camera model significantly reduces the camera field of view, which will affect the algorithm accuracy of the visual inertial tracking and positioning system; 2. In terms of computational efficiency, converting from an omnidirectional image to a perspective image will involve a large amount of image calculation, which is very time-consuming for a lightweight tracking and positioning system.

[0005] The omnidirectional image is mapped to a unit sphere, and the projection relationship is described using a spherical coordinate system. At the same time, the traditional reprojection pixel error is changed to the angular error of the spherical coordinate system during the calculation process. This solution also has two obvious problems: 1. Most visual inertial tracking and positioning systems are based on the pinhole camera model. If it is changed to a spherical model, all the projection formula derivations of the system need to be modified, which is costly; 2. There is a large distortion in the mapping from the omnidirectional camera to the spherical coordinate system, especially in the area close to the boundary of the image; but the omnidirectional camera has a large pixel distortion, and the distortion in the boundary area is difficult to calibrate. In actual operation, it is difficult to map the omnidirectional image to a unit sphere. Summary of the invention

[0006] The present application provides an omnidirectional image processing method, an omnidirectional image processing device, and a computer-readable storage medium.

[0007] The present application provides a method for processing an omnidirectional image, the method comprising:

[0008] Acquire omnidirectional images;

[0009] The omnidirectional image is projected onto the pinhole camera model to obtain a pinhole image;

[0010] Setting the mask values ​​of the pixels in the omnidirectional image that have corresponding projection points in the pinhole image to a first preset value, and setting the mask values ​​of the remaining pixels in the omnidirectional image to a second preset value;

[0011] The omnidirectional image is subjected to mask processing by using the mask value of each pixel in the omnidirectional image.

[0012] In some embodiments, the method further comprises:

[0013] The mask value of the pixel points within a preset pixel distance from the edge to the center of the omnidirectional image is set to the second preset value.

[0014] In some embodiments, the method further comprises:

[0015] Acquire longitude position information and latitude position information of the omnidirectional image on the spherical model;

[0016] Divide the omnidirectional image after mask processing into spherical areas according to the longitude position information and the latitude position information, and then project the omnidirectional image onto the pinhole image to obtain a plurality of divided areas on the pinhole image corresponding to the result of the spherical area division;

[0017] Extracting a preset number of feature points in each divided area of ​​the pinhole image;

[0018] The omnidirectional image acquisition device is positioned and / or its posture is estimated based on the extracted feature points.

[0019] In some embodiments, after extracting a preset number of feature points in each divided area of ​​the pinhole image, the method further includes:

[0020] Determine adjacent pixel points of the feature point in the omnidirectional image;

[0021] Determine the adjacent pinhole pixel points corresponding to the adjacent pixel points in the pinhole image;

[0022] Determining a weight value of the feature point according to a positional relationship between the adjacent pinhole pixel points and the feature point;

[0023] The method of positioning and / or estimating the posture of the acquisition device of the omnidirectional image based on the extracted feature points includes:

[0024] The extracted feature points are weighted by weight values, and the weighted feature points are used for positioning and / or posture estimation.

[0025] In some embodiments, after determining the weight value of the feature point, the method further includes:

[0026] Get the reprojection error of each feature point;

[0027] A weighted sum of the reprojection errors of all feature points and the weight values ​​is obtained to determine the reprojection error of the pinhole image.

[0028] In some embodiments, determining the weight value of the feature point according to the positional relationship between the adjacent pinhole pixel points and the feature point includes:

[0029] Obtain the Euclidean distances from all adjacent pinhole pixels to the feature point;

[0030] The root mean square of all Euclidean distances is calculated, and the reciprocal of the root mean square is used as the weight value of the feature point.

[0031] In some embodiments, the omnidirectional image includes acquisition using a rolling shutter omnidirectional camera;

[0032] The method further comprises:

[0033] Obtaining the exposure time of the first row of pixels of the omnidirectional image and the current device posture of the omnidirectional camera;

[0034] Determine the calibrated device posture of the omnidirectional camera when the row where the feature points in the omnidirectional image are located is exposed according to the number of the row where the feature points in the omnidirectional image are located, the current device posture, and the device motion information of the omnidirectional camera when the frame where the omnidirectional image is located is collected by visually positioning the omnidirectional camera;

[0035] Projecting the three-dimensional points of the feature points of the omnidirectional image according to the position and posture of the calibration device to obtain calibration feature points;

[0036] The reprojection error of the feature point is determined by using the difference information between the feature point of the omnidirectional image and the calibrated feature point.

[0037] The present application also provides an omnidirectional image processing device, comprising:

[0038] an acquisition unit configured to acquire an omnidirectional image;

[0039] A projection unit, configured to project the omnidirectional image onto a pinhole camera model, thereby obtaining a pinhole image;

[0040] A setting unit, configured to set the mask values ​​of the pixel points in the omnidirectional image that have corresponding projection points in the pinhole image to a first preset value, and set the mask values ​​of the remaining pixel points in the omnidirectional image to a second preset value;

[0041] The mask processing unit is configured to perform mask processing on the omnidirectional image using the mask value of each pixel in the omnidirectional image.

[0042] The present application also provides an electronic device, which includes a processor and a memory, wherein the memory stores program data, and the processor is used to execute the program data to implement the processing method as described above.

[0043] The present application also provides a computer-readable storage medium, which is used to store program data. When the program data is executed by a processor, it is used to implement the above-mentioned processing method.

[0044] The beneficial effects of the present application are: the omnidirectional image processing device obtains the omnidirectional image; projects the omnidirectional image to a pinhole camera model to obtain a pinhole image; sets the mask value of the pixel points in the omnidirectional image that have corresponding projection points in the pinhole image to a first preset value, and sets the mask value of the remaining pixel points in the omnidirectional image to a second preset value; and performs mask processing on the omnidirectional image using the mask value of each pixel point in the omnidirectional image. In the above manner, the processing device uses an abstract pinhole camera model to describe the mapping relationship of the omnidirectional camera, which can retain image information as much as possible and reduce pixel distortion. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0046] Figure 1 It is a flowchart of the first embodiment of the omnidirectional image processing method provided by the present application;

[0047] Figure 2 is a flow chart of a second embodiment of the omnidirectional image processing method provided by the present application;

[0048] Figure 3 is a flowchart of a third embodiment of the omnidirectional image processing method provided by the present application;

[0049] Figure 4 is a flowchart of a fourth embodiment of the omnidirectional image processing method provided by the present application;

[0050] Figure 5 It is a structural schematic diagram of an embodiment of a processing device provided by the present application;

[0051] Figure 6 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application;

[0052] Figure 7It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] In order to solve the above-mentioned problems of the omnidirectional camera visual tracking and positioning system, this method proposes an omnidirectional camera visual inertial tracking and positioning algorithm without information loss based on the abstract pinhole camera projection model.

[0055] Traditional methods directly use spherical models to fit omnidirectional images, or intercept part of the omnidirectional image and map it into a perspective image. The former changes the camera model, and the latter loses some image information. This application uses an abstract pinhole camera model to describe the mapping relationship of the omnidirectional camera, and existing solutions such as fisheye projection model and isometric distortion model can be used for fitting.

[0056] Please refer to Figure 1 , Figure 1 It is a flowchart of the first embodiment of the omnidirectional image processing method provided by the present application.

[0057] The omnidirectional image processing method of the present application is applied to a processing device, wherein the processing device of the present application can be a server or a terminal device, etc., or a system in which a server and a terminal device cooperate with each other. Accordingly, the various parts included in the processing device, such as various units, sub-units, modules, and sub-modules, can all be set in the server, or can be set in the server and the terminal device respectively.

[0058] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, such as a distributed system, a cloud, etc., or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules used to provide distributed servers, or it can be implemented as a single software or software module, which is not specifically limited here. In some possible implementations, the omnidirectional image processing method of the embodiment of the present application can be implemented by a processor calling computer-readable instructions stored in a memory.

[0059] In a specific implementation, the processing device may also be a robot, an unmanned vehicle, a drone, etc., in which case the image processing result may be used for visual positioning and navigation. In another specific implementation, the processing device may also be an AR (Augmented Reality) device, a VR (Virtual Reality) device, etc., in which case the image processing result may be used for enhanced implementation technology, virtual reality technology, etc. The specific product types of the processing device are not listed one by one here.

[0060] Specifically, if Figure 1 As shown, the method for processing an omnidirectional image in an embodiment of the present application specifically includes the following steps:

[0061] Step S11: Acquire an omnidirectional image.

[0062] In the embodiment of the present application, the processing device uses a monocular omnidirectional camera or a binocular omnidirectional camera to collect an omnidirectional image of the target area. Among them, an omnidirectional camera (Omnidirectional Camera Model) is a camera with a 360-degree field of view on a horizontal plane, or a field of view covering a hemisphere or (approximately) an entire sphere.

[0063] Among them, the calibration internal parameters of a monocular omnidirectional camera or a binocular omnidirectional camera can use various forms of camera models and distortion models, including but not limited to: pinhole camera model, spherical camera model, fisheye camera model, equidistant distortion model, radial tangential distortion model, etc.

[0064] Step S12: Projecting the omnidirectional image onto the pinhole camera model to obtain a pinhole image.

[0065] In the embodiments of the present application, the traditional omnidirectional camera projection model uses a spherical model, but the projection formula of the spherical model is significantly different from the commonly used pinhole perspective model, and the visual inertial tracking and positioning system needs to make a lot of formula modifications to replace the projection model.

[0066] This application uses an abstract pinhole camera model, and introduces a nonlinear representation under the ideal linear pinhole camera model, which may include a combination of one or more camera models and distortion models in the visual field, such as a fisheye camera model, a pinhole camera model + an isometric distortion model, a pinhole camera model + a tangential radial distortion model, etc. The processing device represents the omnidirectional image using the pinhole camera model to obtain a pinhole image.

[0067] Step S13: setting the mask values ​​of the pixels in the omnidirectional image that have corresponding projection points in the pinhole image to the first preset value, and setting the mask values ​​of the remaining pixels in the omnidirectional image to the second preset value.

[0068] In the embodiment of the present application, step S12 uses an abstract pinhole camera model to describe the projection relationship of the omnidirectional camera, and step S13 accordingly proposes a mask mechanism based on the omnidirectional camera.

[0069] Specifically, the effective image collected by the omnidirectional camera is close to a circle, so there will be a lot of black invisible areas at the corners of the image. These areas need to be masked during feature extraction and matching. The mask is set as follows: according to the abstract pinhole camera model, you can try to project all the pixels on the image back to the pinhole camera plane. If the projection is successful, the mask at that position is set to 1, that is, the first preset value; otherwise, it is set to 0, that is, the second preset value.

[0070] Furthermore, considering that the camera model may have certain calibration errors that cause the image boundary to be inconsistent with the real boundary, and the boundary area is not suitable for extracting feature points, the mask area needs to be shrunk inward by a certain number of pixels.

[0071] Specifically, after step S13, the processing device further shrinks the mask area of ​​the omnidirectional image. The processing device sets the mask value of the pixel point within the preset pixel distance from the edge to the center of the omnidirectional image to the second preset value. Specifically, if the mask value of the pixel point within the preset pixel distance from the edge to the center of the omnidirectional image is set to the first preset value in the aforementioned step S13, the image processing device replaces it with the first preset value. If the mask value of the pixel point within the preset pixel distance from the edge to the center of the omnidirectional image is set to the second preset value in the aforementioned step S13, the mask value at the pixel point is maintained as the first preset value, thereby achieving the shrinkage of the mask area, thereby avoiding the introduction of inaccurate features at the edge of the omnidirectional image after the subsequent mask processing.

[0072] Step S14: performing mask processing on the omnidirectional image using the mask value of each pixel point in the omnidirectional image.

[0073] In the embodiment of the present application, after the processing device performs mask processing, the pixel information of the pixel points whose mask value is the first preset value in the omnidirectional image is retained, and the pixel information of the pixel points whose mask value is the second preset value is removed, so as to obtain the omnidirectional image after mask processing, and the omnidirectional image only retains the pixel points with corresponding projection points in the pinhole camera model. The omnidirectional image after setting the mask and the corresponding pinhole image can be used to realize image recognition, image segmentation, image tracking and other application scenarios, which are not listed here one by one.

[0074] In an embodiment of the present application, a processing device acquires an omnidirectional image; projects the omnidirectional image onto a pinhole camera model to obtain a pinhole image; sets the mask values ​​of the pixels in the omnidirectional image that have corresponding projection points in the pinhole image to a first preset value, and sets the mask values ​​of the remaining pixels in the omnidirectional image to a second preset value; and performs mask processing on the omnidirectional image using the mask values ​​of the pixels in the omnidirectional image. In the above manner, the processing device uses an abstract pinhole camera model to describe the mapping relationship of the omnidirectional camera, which can retain useful image information in the omnidirectional image as much as possible and reduce pixel distortion.

[0075] Furthermore, the present application describes the projection relationship of an omnidirectional camera based on an abstract pinhole camera model, and proposes a feature distribution mechanism based on an omnidirectional camera.

[0076] Please refer to Figure 2 , Figure 2 It is a flowchart diagram of another embodiment of the omnidirectional image processing method provided by the present application.

[0077] Specifically, based on the aforementioned steps S11 to S13, the omnidirectional image processing method of the embodiment of the present application further specifically includes the following steps:

[0078] Step S21: Acquire the longitude position information and latitude position information of the omnidirectional image on the spherical model.

[0079] There are two traditional feature point distribution schemes: 1. Suppressing other feature points within the radius of each feature point x pixels; 2. Octree and other spatial partitioning schemes, which divide the two-dimensional image into several regions of the same size and extract a specific number of points in each region. Traditional schemes do not take into account that pixel distortion causes image regions of the same size to actually contain large differences in the amount of information.

[0080] In the embodiment of the present application, the octree division used by the processing device is not a uniform division of the x and y axes of the two-dimensional image, but a division based on the longitude and latitude of the three-dimensional sphere. Please refer to the following steps for details:

[0081] Step S22: Divide the omnidirectional image after mask processing into spherical areas according to the longitude position information and the latitude position information, and then project the divided areas onto the pinhole image to obtain a plurality of divided areas on the pinhole image corresponding to the results of the spherical area division.

[0082] In the embodiment of the present application, the processing device divides the omnidirectional image after mask processing into spherical regions according to the longitude position information and the latitude position information, and then projects the spherical regions onto the pinhole image. Then, the processing device obtains the result of the spherical region division, and then projects the result of the spherical region division onto the pinhole image, thereby obtaining a plurality of divided regions on the pinhole image.

[0083] Step S23: extracting a preset number of feature points in each divided area of ​​the pinhole image.

[0084] In an embodiment of the present application, the processing device first constructs a virtual hemisphere, divides the longitude and latitude at equal angles on the hemisphere, projects the longitude and latitude onto the image through a pinhole camera model, and extracts a specified number of feature points in the divided area where the longitude and latitude intersect.

[0085] Step S24: positioning and / or posture estimation of the omnidirectional image acquisition device based on the extracted feature points.

[0086] In the embodiment of the present application, the processing device obtains accurate feature points in the omnidirectional image through the above steps. For the subsequent processing of the feature points, the processing device can implement application scenarios including but not limited to: target positioning, target recognition, target tracking, posture estimation, etc. For the above application scenarios, the processing device implements any application scenario according to the feature points and mature feature processing algorithms on the market.

[0087] The above-mentioned embodiment uses a three-dimensional spherical longitude and latitude division method to perform region segmentation on the omnidirectional image, and extracts a specified number of feature points in each segmented image area, thereby achieving a uniform distribution of feature points, which helps to extract the features of the omnidirectional image more comprehensively and accurately, thereby improving the accuracy of positioning and / or posture estimation.

[0088] In traditional technology, omnidirectional cameras have very high pixel distortion during the projection process. The pixel distortion in different areas of the image varies greatly, and it is impossible to use a unified reprojection error to represent points at different positions on the image. The embodiment of the present application applies weights to the distribution of feature points in different areas of the image based on the projection and distortion model, thereby reducing the impact of pixel distortion on system accuracy.

[0089] In this regard, an embodiment of the present application proposes a two-dimensional point weight model for an omnidirectional camera, which weights the distribution of feature points in different areas of the image based on projection and distortion models.

[0090] Specifically, for omnidirectional camera images, there is a large distortion of pixels, that is, in different areas of the image, the actual three-dimensional space area that can be described by a pixel varies greatly: in the center area of ​​the image, the actual three-dimensional space area that can be described by a pixel is smaller, and the pixel accuracy is higher; in the edge area of ​​the image, the actual three-dimensional space area that can be described by a pixel is larger, and the pixel accuracy is poor.

[0091] If the same weight is applied to all feature points, the weight of two-dimensional points with high description accuracy will be too low, while the weight of two-dimensional points with low description accuracy will be too high, which will cause the tracking and positioning system to have insufficient trust in high-quality points and too high trust in low-quality points, causing the system to be too sensitive to noise.

[0092] Therefore, the present application embodiment proposes a method for calculating a two-dimensional point weight model. For details, please refer to Figure 3 , Figure 3 FIG. 1 is a flow chart of another embodiment of the method for processing an omnidirectional image provided in an embodiment of the present application. In this embodiment, based on steps S11 to S14 and steps S21 to S24 in the above figure, the method for processing an omnidirectional image further includes:

[0093] Step S25: Determine the adjacent pixel points of the feature point in the omnidirectional image.

[0094] In the embodiment of the present application, the processing device maps the two-dimensional point A on the omnidirectional image to A' in an ideal pinhole camera coordinate system through an abstract pinhole camera model, and solves the pixel distortion problem by using the linear model of the pinhole camera.

[0095] The processing device draws a circle with a radius of 1 pixel and a 2D point on the omnidirectional image as the center, and samples N points at equal intervals on the circle. In other embodiments, other pixel distances may be set as the radius, for example, a circle may be drawn with a radius of 2 pixels.

[0096] Step S26: Determine the adjacent pinhole pixel points corresponding to the adjacent pixel points in the pinhole image.

[0097] In the embodiment of the present application, the processing device maps the N points to the ideal pinhole camera coordinate system through the abstract camera model, and calculates the Euclidean distances of these points, ie, the adjacent pinhole pixel points to the point A'.

[0098] In this embodiment, the aforementioned step S24 can be implemented by the following step S27:

[0099] Step S27: weighting the extracted feature points by weight values, and using the weighted feature points for positioning and / or posture estimation.

[0100] In the embodiment of the present application, the processing device calculates the root mean square of the Euclidean distances from all adjacent pinhole pixels to point A', and uses the reciprocal of the root mean square as the weight of point A.

[0101] The weights of the two-dimensional points calculated from step S25 to step S27 can be used in:

[0102] 1. The visual inertial tracking positioning system calculates the reprojection error of three-dimensional points. The points closer to the camera center contribute more to the visual error function, and the points closer to the image boundary contribute less to the visual error function.

[0103] 2. The feature tracking of the intrinsic matrix is ​​screened out to avoid excessive tracking errors of camera boundary points affecting the determination of external points.

[0104] 3. Image disparity calculation: feature points at the edge of a two-dimensional point displacement image with the same distance can provide more disparity than feature points at the center of the image. A larger disparity is more conducive to the triangulation of two-dimensional points.

[0105] Traditional tracking and positioning algorithms directly use the displacement distance of feature points on the original image when calculating feature point parallax. However, omnidirectional cameras have large pixel distortion. The displacement of the same pixel at the edge and center of the image may be several times or even dozens of times different in the perspective image. This method adds weight to the displacement, which can effectively increase the life cycle of feature points in the sliding window, thereby improving system accuracy.

[0106] Taking the first use as an example, please continue to refer to the following steps:

[0107] Step S28: Obtain the reprojection error of each feature point.

[0108] Step S29: Obtain the weighted sum of the reprojection errors of all feature points and the weight values ​​to determine the reprojection error of the pinhole image.

[0109] In the embodiment of the present application, the processing device calculates the reprojection error of each feature point in the omnidirectional image, and then weights and adds the reprojection errors of all feature points according to the weight of each feature point, thereby determining the reprojection error of the pinhole image as a whole. Using the weight value to adjust the feature points at different camera positions is conducive to improving the accuracy of the reprojection error and the visual error function, and improving the efficiency and effect of camera pose optimization.

[0110] Furthermore, the omnidirectional camera for collecting omnidirectional images of the present application may adopt a rolling shutter omnidirectional camera or a global shutter omnidirectional camera. For a rolling shutter camera, the camera will expose line by line when collecting images, that is, each line of the image corresponds to a different exposure time, and the exposure time of the first line and the last line of the image may differ by tens of milliseconds. If the camera device is moving, the time deviation of tens of milliseconds is enough for the camera to move several centimeters or degrees. Ignoring the jello effect of the camera will affect the accuracy of the system.

[0111] Therefore, this application also provides a rolling shutter effect compensation mechanism, adding online calibration of rolling shutter exposure time, please refer to Figure 4 , Figure 4It is a flowchart diagram of another embodiment of the omnidirectional image processing method provided by the present application.

[0112] Specifically, in the case where the omnidirectional image is collected using an omnidirectional camera with a rolling shutter, such as Figure 4 As shown, the process of the omnidirectional image processing method of this embodiment also includes:

[0113] Step S31: Obtain the exposure time of the first row of pixels of the omnidirectional image and the current device posture of the omnidirectional camera.

[0114] In an embodiment of the present application, the processing device can obtain the device posture of each frame of omnidirectional image during the operation of the visual inertial positioning system, specifically including the movement speed and rotation angle, and the exposure time of the first line of the frame.

[0115] Step S32: Determine the calibrated device posture of the omnidirectional camera when the row where the feature points in the omnidirectional image are located is exposed according to the row number where the feature points in the omnidirectional image are located, the current device posture, and the device motion information of the omnidirectional camera when the frame where the omnidirectional image is located is collected by visual positioning of the omnidirectional camera.

[0116] In an embodiment of the present application, the processing device assumes that each frame is in a state of uniform motion and uniform rotation at the time of acquisition. Based on the prior exposure time of the rolling shutter, the position information and angle information corresponding to each row of acquisition time, that is, the device posture, can be calculated.

[0117] Step S33: Project the three-dimensional points of the feature points of the omnidirectional image according to the position and posture of the calibration device to obtain calibration feature points.

[0118] In the embodiment of the present application, during the triangulation process, the processing device uses the position and angle corresponding to the row where the feature point is located as the camera's posture and triangulates each observation.

[0119] Step S34: using the difference information between the feature points of the omnidirectional image and the calibrated feature points, determine the reprojection errors of the feature points.

[0120] In an embodiment of the present application, in the process of solving the reprojection error, the processing device projects the three-dimensional point with the camera posture corresponding to the row where the feature point is located as the projection target, obtains the projected two-dimensional point, and the actual observation value of the two-dimensional point is used as the reprojection error.

[0121] This application proposes a visual inertial tracking and positioning system for an omnidirectional camera based on a pinhole projection model. Compared with the existing technology, this application does not use a spherical camera model. In response to the pixel distortion problem of the omnidirectional camera, this application proposes a distortion weight compensation mechanism that can significantly improve the accuracy and robustness of the tracking and positioning system. This application proposes a jelly effect compensation mechanism for rolling shutter omnidirectional cameras that can significantly improve the accuracy of the tracking and positioning system.

[0122] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0123] In order to implement the omnidirectional image processing method of the above embodiment, the present application also proposes a processing device, which can be found in Figure 5 , Figure 5 It is a structural schematic diagram of an embodiment of a processing device provided in the present application.

[0124] The processing device 400 of the embodiment of the present application includes an acquisition unit 41 , a projection unit 42 , a setting unit 43 and a mask processing unit 44 .

[0125] The acquisition unit 41 is configured to acquire an omnidirectional image.

[0126] The projection unit 42 is configured to project the omnidirectional image onto the pinhole camera model, thereby obtaining a pinhole image.

[0127] The setting unit 43 is configured to set the mask values ​​of the pixels in the omnidirectional image that have corresponding projection points in the pinhole image to the first preset value, and set the mask values ​​of the remaining pixels in the omnidirectional image to the second preset value.

[0128] The mask processing unit 44 is configured to perform mask processing on the omnidirectional image using the mask value of each pixel in the omnidirectional image.

[0129] Each module in the processing device 400 of this embodiment corresponds one-to-one to each step in the aforementioned method embodiment. The description of each step in the aforementioned embodiment of the method for processing omnidirectional images is also applicable to the corresponding module in the processing device 400, and will not be repeated here.

[0130] In order to implement the omnidirectional image processing method of the above embodiment, the present application also proposes another processing device, which can be found in Figure 6 , Figure 6 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application.

[0131] The electronic device 500 of the embodiment of the present application includes a memory 51 and a processor 52, wherein the memory 51 and the processor 52 are coupled.

[0132] The memory 51 is used to store program data, and the processor 52 is used to execute the program data to implement the omnidirectional image processing method described in the above embodiment.

[0133] In this embodiment, the processor 52 may also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip having the ability to process signals. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or the processor 52 may also be any conventional processor, etc.

[0134] In order to implement the omnidirectional image processing method of the above embodiment, the present application also provides a computer-readable storage medium, such as Figure 7 As shown, the computer-readable storage medium 600 is used to store program data 61. When the program data 61 is executed by the processor, it is used to implement the omnidirectional image processing method as described in the above embodiment.

[0135] The present application also provides a computer program product, wherein the computer program product includes a computer program, and the computer program is operable to enable a computer to execute the omnidirectional image processing method as described in the embodiment of the present application. The computer program product can be a software installation package.

[0136] The omnidirectional image processing method described in the above embodiment of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0137] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for processing an omnidirectional image, characterized in that: The method comprises: Acquire omnidirectional images; The omnidirectional image is projected onto the pinhole camera model to obtain a pinhole image; Setting the mask values ​​of the pixels in the omnidirectional image that have corresponding projection points in the pinhole image to a first preset value, and setting the mask values ​​of the remaining pixels in the omnidirectional image to a second preset value; The omnidirectional image is subjected to mask processing by using the mask value of each pixel in the omnidirectional image.

2. The method according to claim 1, characterized in that The method further comprises: The mask value of the pixel points within a preset pixel distance from the edge to the center of the omnidirectional image is set to the second preset value.

3. The method according to claim 1, characterized in that The method further comprises: Acquire longitude position information and latitude position information of the omnidirectional image on the spherical model; Divide the omnidirectional image after mask processing into spherical areas according to the longitude position information and the latitude position information, and then project the omnidirectional image onto the pinhole image to obtain a plurality of divided areas on the pinhole image corresponding to the result of the spherical area division; Extracting a preset number of feature points in each divided area of ​​the pinhole image; The omnidirectional image acquisition device is positioned and / or its posture is estimated based on the extracted feature points.

4. The method according to claim 3, characterized in that After extracting a preset number of feature points in each divided area of ​​the pinhole image, the method further includes: Determine adjacent pixel points of the feature point in the omnidirectional image; Determine the adjacent pinhole pixel points corresponding to the adjacent pixel points in the pinhole image; Determining a weight value of the feature point according to a positional relationship between the adjacent pinhole pixel points and the feature point; The method of positioning and / or estimating the posture of the acquisition device of the omnidirectional image based on the extracted feature points includes: The extracted feature points are weighted by weight values, and the weighted feature points are used for positioning and / or posture estimation.

5. The method according to claim 4, characterized in that After determining the weight value of the feature point, the method further includes: Get the reprojection error of each feature point; A weighted sum of the reprojection errors of all feature points and the weight values ​​is obtained to determine the reprojection error of the pinhole image.

6. The method according to claim 4 or 5, characterized in that: The step of determining the weight value of the feature point according to the positional relationship between the adjacent pinhole pixel points and the feature point comprises: Obtain the Euclidean distances from all adjacent pinhole pixels to the feature point; The root mean square of all Euclidean distances is calculated, and the reciprocal of the root mean square is used as the weight value of the feature point.

7. The method according to claim 1, characterized in that The omnidirectional image includes collecting the image using a rolling shutter omnidirectional camera; The method further comprises: Obtaining the exposure time of the first row of pixels of the omnidirectional image and the current device posture of the omnidirectional camera; Determine the calibrated device posture of the omnidirectional camera when the row where the feature points in the omnidirectional image are located is exposed according to the number of the row where the feature points in the omnidirectional image are located, the current device posture, and the device motion information of the omnidirectional camera when the frame where the omnidirectional image is located is collected by visually positioning the omnidirectional camera; Projecting the three-dimensional points of the feature points of the omnidirectional image according to the position and posture of the calibration device to obtain calibration feature points; The reprojection error of the feature point is determined by using the difference information between the feature point of the omnidirectional image and the calibrated feature point.

8. An omnidirectional image processing device, characterized in that: include: an acquisition unit, configured to acquire an omnidirectional image; A projection unit, configured to project the omnidirectional image onto a pinhole camera model, thereby obtaining a pinhole image; A setting unit, configured to set the mask values ​​of the pixel points in the omnidirectional image that have corresponding projection points in the pinhole image to a first preset value, and set the mask values ​​of the remaining pixel points in the omnidirectional image to a second preset value; The mask processing unit is configured to perform mask processing on the omnidirectional image using the mask value of each pixel in the omnidirectional image.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein program data is stored in the memory, and the processor is used to execute the program data to implement the processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program data, and when the program data is executed by a processor, it is used to implement the processing method according to any one of claims 1 to 7.

Images