Image processing method, image processing device and computer readable storage medium

Abstract

The invention provides an image processing method, an image processing device and a computer readable storage medium. The image processing method comprises the following steps: acquiring a panoramic image; splitting the panoramic image into a plurality of perspective images along a first spherical surface direction of the panoramic image; acquiring a projection range of the panoramic image along a second spherical surface direction of the panoramic image; setting a mask on each perspective view according to the projection range; and splicing the plurality of perspective images with the masks so as to obtain a perspective image spliced image corresponding to the panoramic image. Through the above mode, the image processing device provides a view angle splitting mask shielding scheme for the problem of overlarge distortion of south and north poles of the panoramic image, the stability of tracking and positioning can be improved, a panoramic image view angle splitting scheme is provided, on one hand, the full-amount information advantage of the panoramic image is utilized, and on the other hand, the view angle of the panoramic image is split; on one hand, complex camera models such as a fisheye camera and a spherical camera do not need to be processed.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, an image processing device, and a computer-readable storage medium. Background Art

[0002] Panoramic images can display the surrounding environment in 360 degrees without blind spots. Compared with traditional cameras, the images and videos captured are more immersive. They are widely used in various scenarios such as scene display, virtual reality, unmanned vehicles, and drones.

[0003] Panoramic images are usually stitched together from multiple cameras, and the original images from each camera are used directly to run a single-view or multi-view visual inertial tracking and positioning system. This method has many problems: 1. If only one of the cameras is used, 360-degree information cannot be used, and there is a lot of missing information, which affects the accuracy and stability of tracking and positioning; 2. If multiple cameras are used, it takes a lot of time and computing power to track and match multiple cameras in time and space; 3. Usually, a single camera in a panoramic camera has large distortion, and existing visual tracking methods cannot handle such images well. Summary of the invention

[0004] The present application provides an image processing method, an image processing device, and a computer-readable storage medium.

[0005] The present application provides an image processing method, which includes:

[0006] Acquire panoramic images;

[0007] Splitting the panoramic image into a plurality of perspective images along a first spherical direction of the panoramic image;

[0008] Acquire a projection range of the panoramic image along a second spherical direction of the panoramic image, wherein the first spherical direction is one of a longitude direction and a latitude direction of the panoramic image, and the second spherical direction is the other of the longitude direction and the latitude direction of the panoramic image;

[0009] Setting a mask on each of the perspective images according to the projection range;

[0010] The masked perspective images are stitched together to obtain a perspective image stitching image corresponding to the panoramic image.

[0011] In some embodiments, acquiring the projection range of the panoramic image along the second spherical direction of the panoramic image includes:

[0012] The preset angles are respectively extended in the south latitude direction and the north latitude direction along the equator of the panoramic image, and the coverage range is used as the projection range of the panoramic image.

[0013] In some embodiments, after acquiring the perspective stitched image corresponding to the panoramic image, the image processing method further includes:

[0014] Extracting features of the effective area of ​​the mask in the perspective stitched image to obtain target feature points in the perspective stitched image; and,

[0015] Visually-based target tracking and / or positioning is performed based on the target feature points of the perspective stitched image.

[0016] In some embodiments, after acquiring the perspective stitched image corresponding to the panoramic image, the image processing method further includes:

[0017] Performing feature extraction on a valid area of ​​the mask in the panoramic image to obtain target feature points in the panoramic image;

[0018] According to the projection relationship between the panoramic image and the perspective stitched image, projecting the target feature points of the panoramic image to the target feature points of the perspective stitched image; and,

[0019] Visually-based target tracking and / or positioning is performed based on the target feature points of the perspective stitched image.

[0020] In some embodiments, the image processing method further includes:

[0021] Acquire a plurality of continuous frames of perspective stitched images, and identify target feature points on the plurality of continuous frames of perspective stitched images;

[0022] After the target feature point moves from left to right to the right boundary of the perspective stitching image of the continuous multiple frames, continue searching for the target feature point from the left boundary of the next frame of the perspective stitching image;

[0023] Alternatively, after the target feature point moves from right to left to the left boundary of the perspective stitching image of the continuous multiple frames, the target feature point is continuously searched from the right boundary of the next frame of the perspective stitching image.

[0024] In some embodiments, the step of splicing the plurality of perspective views after setting the mask comprises:

[0025] Acquire a preset number of left pixel columns and a preset number of right pixel columns of each perspective image after masking;

[0026] Copy the left pixel column to the right boundary, and copy the right pixel column to the left boundary;

[0027] Join the right border of the left perspective image and the left border of the right perspective image in the adjacent perspective images.

[0028] In some embodiments, after acquiring the perspective stitching images of the plurality of perspective images, the image processing method further includes:

[0029] Obtaining perspective coordinates and world coordinates of the calibration points in the perspective stitched image;

[0030] Calibrate the relative rotation relationship between the omnidirectional camera and the inertial sensor by using the perspective coordinates, world coordinates, device posture and virtual rotation relationship between the virtual pinhole camera and the omnidirectional camera of the calibration point;

[0031] The relative position relationship between the omnidirectional camera and the inertial sensor is calibrated using the relative rotation relationship and the virtual rotation relationship.

[0032] The present application also provides an image processing device, which includes an acquisition module, a splitting module, a mask module, and a splicing module;

[0033] Wherein, the acquisition module is used to acquire a panoramic image;

[0034] The splitting module is used to split the panoramic image into a plurality of perspective images along a first spherical direction of the panoramic image;

[0035] The mask module is used to obtain a projection range of the panoramic image along a second spherical direction of the panoramic image, wherein the first spherical direction is one of a longitude direction and a latitude direction of the panoramic image, and the second spherical direction is the other of the longitude direction and the latitude direction of the panoramic image; and set a mask on each of the perspective images according to the projection range;

[0036] The stitching module is used to stitch a plurality of perspective images after setting the mask, so as to obtain a perspective stitching image corresponding to the panoramic image.

[0037] The present application also provides an image processing device, which includes a processor and a memory, wherein program data is stored in the memory, and the processor is used to execute the program data to implement the above-mentioned image processing method.

[0038] The present application also provides a computer-readable storage medium, which is used to store program data. When the program data is executed by a processor, it is used to implement the above-mentioned image processing method.

[0039] The beneficial effects of the present application are: the image processing device obtains a panoramic image; splits the panoramic image into several perspective views along the first spherical direction of the panoramic image; obtains the projection range of the panoramic image along the second spherical direction of the panoramic image, the first spherical direction is one of the longitude direction and the latitude direction of the panoramic image, and the second spherical direction is the other of the longitude direction and the latitude direction of the panoramic image; sets a mask on each of the perspective views according to the projection range; splices the several perspective views after setting the mask, so as to obtain the perspective view splicing image corresponding to the panoramic image. Through the above method, the image processing device proposes a perspective splitting masking scheme to solve the problem of excessive distortion of the north and south poles of the panoramic image, which can improve the stability of tracking and positioning, and proposes a panoramic view splitting scheme, which, on the one hand, utilizes the full information advantage of the panoramic image, and on the other hand, does not need to process complex camera models such as fisheye cameras and spherical cameras. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0041] Figure 1 It is a flowchart of an embodiment of an image processing method provided by the present application;

[0042] Figure 2 is a schematic diagram of an embodiment of a panoramic image provided by the present application;

[0043] Figure 3 is a schematic diagram of an embodiment of a perspective image provided by the present application;

[0044] Figure 4 is a schematic diagram of an embodiment of a mask image provided by the present application;

[0045] Figure 5 is a flowchart of another embodiment of the image processing method provided by the present application;

[0046] Figure 6 is a structural schematic diagram of an embodiment of an image processing device provided by the present application;

[0047] Figure 7 is a structural schematic diagram of another embodiment of the image processing device provided by the present application;

[0048] Figure 8 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] Visual inertial tracking and positioning systems are important underlying technologies in the fields of computer vision, robotics, unmanned vehicles, 3D reconstruction, and augmented reality. Robust and efficient visual inertial tracking and positioning solutions have important practical value in these fields. However, due to the problems of existing technologies, the stability of visual inertial tracking and positioning systems is poor. Running the visual inertial tracking and positioning system directly on the panoramic image requires changing the traditional pinhole camera projection model to a spherical camera model, and changing the traditional reprojection error function to a spherical angle error function. Traditional visual inertial tracking and positioning systems are based on pinhole cameras or abstract pinhole camera models. Changing them to spherical camera models requires re-deriving most of the formulas and parameter thresholds in the system, which greatly increases the difficulty of adaptation.

[0051] In order to solve the problems of the above-mentioned panoramic image visual tracking and positioning system, the present application proposes a perspective splitting tracking and positioning solution based on the stitched panoramic image without changing the camera model.

[0052] Please refer to Figure 1 , Figure 1 It is a flowchart of an embodiment of an image processing method provided by the present application.

[0053] The image processing method of the present application is applied to an image processing device, wherein the image processing device of the present application can be a server or a terminal device, etc., or a system in which a server and a terminal device cooperate with each other. Accordingly, the various parts included in the image processing device, such as various units, sub-units, modules, and sub-modules, can all be set in the server, or can be set in the server and the terminal device respectively.

[0054] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, such as a distributed system, a cloud, etc., or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules used to provide distributed servers, or it can be implemented as a single software or software module, which is not specifically limited here. In some possible implementations, the image processing method of the embodiment of the present application can be implemented by a processor calling computer-readable instructions stored in a memory.

[0055] In a specific implementation, the image processing device may also be a robot, an unmanned vehicle, a drone, etc., in which case the image processing result may be used for visual positioning and navigation. In another specific implementation, the image processing device may also be an AR device, a VR device, etc., in which case the image processing result may be used for enhanced implementation technology, virtual reality technology, etc. The specific product types of image processing devices are not listed here one by one.

[0056] This application splits the panoramic image into multiple perspective images: on the one hand, there is no need to use a spherical camera model while retaining the most complete information of the panoramic image; on the other hand, there is no need to process the original images of one or several cameras separately; at the same time, the perspective image is the most ideal pinhole projection model, and there is no need to process the distortion information in the camera.

[0057] A panoramic image is a spherical image unfolded into a rectangular image. This unfolding method inevitably results in huge distortion at the north and south poles of the sphere. In this application, each split view is less than 180 degrees, and there is no need to process the north and south poles with large distortion. This application also proposes a masking scheme for panoramic images, which can better avoid using the north and south pole image information of the panoramic image.

[0058] Specifically, if Figure 1 As shown, the image processing method of the embodiment of the present application specifically includes the following steps:

[0059] Step S11: Acquire a panoramic image.

[0060] In the embodiment of the present application, the image processing device uses a monocular omnidirectional camera or a binocular omnidirectional camera to collect a panoramic image of the target area. Among them, an omnidirectional camera (Omnidirectional Camera Model) is a camera with a 360-degree field of view on a horizontal plane, or a field of view covering a hemisphere or (approximately) an entire sphere.

[0061] Among them, the calibration internal parameters of a monocular omnidirectional camera or a binocular omnidirectional camera can use various forms of camera models and distortion models, including but not limited to: pinhole camera model, spherical camera model, fisheye camera model, equidistant distortion model, radial tangential distortion model, etc.

[0062] Step S12: splitting the panoramic image into a plurality of perspective images along a first spherical direction of the panoramic image.

[0063] In the embodiment of the present application, the panoramic image captured by the panoramic camera corresponds to a complete spherical camera model, and each point on the panoramic image can correspond one-to-one with a point on the spherical surface. In other words, if the horizontal and vertical coordinates of a point on the panoramic image are known, it can be corresponded to a point on a certain longitude and latitude on the spherical surface through a coordinate transformation.

[0064] The perspective splitting process can be understood as the binocular panoramic image unfolding along a spherical surface along a certain longitude or latitude cylinder, so that its left and right boundaries can be completely matched.

[0065] Specifically, the internal and external parameters of the binocular omnidirectional camera can be obtained by offline pre-calibration. The image processing device projects each feature point on the image onto a three-dimensional spherical surface through the internal and external parameters of the camera, and then obtains a projection coordinate of the three-dimensional spherical surface. It can be considered that each point on the binocular panoramic image can correspond to a point on the spherical surface one by one. Thus, the first correspondence between the image coordinate system of the panoramic image and the spherical coordinate system can be obtained.

[0066] Similarly, the image processing device can also construct a number of perspective viewpoints without a common view relationship, and project points on a unit sphere onto the perspective viewpoints respectively, wherein the points on the sphere and the points on the perspective view can also form a one-to-one correspondence. Thus, a second correspondence relationship between the spherical coordinate system and the image coordinate system of the perspective view can be obtained.

[0067] The first correspondence and the second correspondence can be presented in the form of a homography matrix or a projection matrix. The image processing device can construct a one-to-one correspondence from the binocular panoramic image to the split perspective image based on the first correspondence and the second correspondence, that is, a mapping relationship or a projection relationship between the image coordinate system of the panoramic image and the image coordinate system of the perspective image.

[0068] Furthermore, the image processing device may split the panoramic image into a plurality of perspective images along the spherical longitude or spherical latitude of the panoramic image according to the mapping relationship or projection relationship determined in the above process.

[0069] Specifically, the image processing device maps all pixel points in the binocular panoramic image to a spherical coordinate system through the camera internal and external parameters, and rotates the sphere so that the directions of its north and south poles are consistent with the Z-axis direction of the inertial coordinate system.

[0070] There are many ways to select the perspective of image splitting, for example, splitting a 360-degree panoramic image into three 120-degree perspective images, or four 90-degree perspective images, or five 72-degree perspective images, or six 60-degree perspective images, etc. In the following embodiments, the perspective splitting scheme is continued by taking the example of splitting a 360-degree panoramic image into four 90-degree perspective images. Among them, the 360-degree panoramic image provided in this application can be seen in Figure 2 , Figure 2It is a schematic diagram of an embodiment of a panoramic image provided by the present application.

[0071] Specifically, the split perspective image has a 90-degree viewing angle in the horizontal direction. To this end, the image processing device can divide the longitude of the sphere into four equal parts, with 45° east longitude, 45° west longitude, 135° east longitude and 135° west longitude as the horizontal optical center, 90° as the field of view, and four spherical areas. It should be noted that the image processing device can also divide the sphere into four equal parts according to the latitude, which will not be repeated here.

[0072] The perspective view has a 90-degree viewing angle in the vertical direction, so the 45-degree viewing angle needs to be mapped to the corresponding areas at the south and north latitudes respectively.

[0073] The image processing device splits the 360-degree panoramic image into the following: Figure 3 The perspective view shown, Figure 3 It is a schematic diagram of an embodiment of a perspective image provided by the present application.

[0074] in, Figure 3 The perspective images in the Figure 1 ,perspective Figure 2 ,perspective Figure 3 And perspective Figure 4 .

[0075] For perspective Figure 1 , whose image area is a panoramic image within the spherical surface from 0° east longitude to 90° east longitude, and from 45° north of the equator to 45° south of the equator. The image feature points in this area can be projected onto the normalized camera plane of the pinhole camera, and mapped to a virtual perspective image through a certain established distortion-free camera intrinsic parameter. Similarly, perspective Figure 2 , the image area is 90°E to 180°E on the spherical surface; perspective Figure 3 , the image area is from 180°W to 90°W; perspective Figure 4 , and its image area is from 90°W to 0°W.

[0076] Given the first correspondence between the image coordinate system of the panoramic image and the spherical coordinate system, and the second correspondence between the spherical coordinate system and the image coordinate system of the perspective view, a mapping relationship or projection relationship between the image coordinate system of the panoramic image and the image coordinate system of the perspective view can be established. Through the mapping relationship or projection relationship, the binocular panoramic images are respectively mapped to Figure 3 In the 4 perspective views shown.

[0077] In other embodiments, the image processing device may obtain the perspective image by first mapping and then splitting; or may obtain the perspective image by first splitting and then mapping, and the process will not be described in detail here.

[0078] The first spherical direction is the spherical longitude direction of the panoramic image, and the second spherical direction is the spherical latitude direction of the panoramic image. The first spherical direction is the spherical latitude direction of the panoramic image, and the second spherical direction is the spherical longitude direction of the panoramic image.

[0079] Step S13: Acquire the projection range of the panoramic image along the second spherical direction of the panoramic image, the first spherical direction is one of the longitude direction and the latitude direction of the panoramic image, and the second spherical direction is the other of the longitude direction and the latitude direction of the panoramic image.

[0080] Step S14: Set a mask on each perspective image according to the projection range.

[0081] This application splits the panorama into perspectives, and each perspective is less than 180°. It is inevitable that some pixels on the panorama cannot be projected to the perspective image after splitting, mainly pixels close to the north and south poles of the spherical surface. On the one hand, the pixels at the north and south poles are too distorted on the panoramic image, and participating in tracking and positioning will affect the accuracy and stability of the system; on the other hand, the points at the north and south poles cannot be observed by the perspective image after splitting, so masks need to be set for these areas in the visual inertial tracking and positioning system.

[0082] The viewing angle size of each virtual camera is split in the known viewing angle. The effective latitude range can be obtained based on the information. The latitude range is projected onto the panoramic image to obtain a mask of the effective area of ​​the panoramic image.

[0083] Specifically, in the embodiments of the present application, Figure 3 Take the scene in the figure as an example. Each perspective image has a 90° viewing angle. The effective latitude range of the corresponding spherical camera is 45 degrees from the equator to the north and south respectively. The areas within this range are projected onto the panoramic image, and the following can be obtained: Figure 4 The mask image of Figure 4 It is a schematic diagram of an embodiment of a mask image provided by the present application.

[0084] Step S15: stitching the masked perspective images together to obtain a stitched perspective image corresponding to the panoramic image.

[0085] In the embodiment of the present application, the split perspective images are obtained by sampling the sphere at equal distances along the meridian direction. The left border of one perspective image is seamlessly connected to the right border of another image, so it can be used Figure 3 The method shown is to separate the several perspective drawings and Figure 4 The mask images in are stitched together to form a complete perspective stitching image.

[0086] The image processing method of the present application can be widely used as a complete visual inertial tracking and positioning system in various scenarios that require visual positioning and navigation, including: autonomous driving, mobile robots, drones, smart phones, virtual reality helmets, augmented reality glasses, and other devices that have both binocular omnidirectional camera sensors and inertial sensor modules. Specifically, the image processing device can construct a map using perspective stitching images, and locate and track the target on the map, or perform path planning on the map, etc.

[0087] A panorama is a sphere unfolded along a certain meridian cylinder, so its left and right borders can fit perfectly. Traditional solutions do not consider the problem of feature points moving from one border to another during the temporal tracking of the panorama. This application proposes a cylindrical feature tracking solution that can significantly improve the life cycle of feature points.

[0088] Furthermore, based on the above perspective image stitching solution, the present application provides a panoramic image cylindrical feature tracking solution.

[0089] The traditional image feature tracking scheme assumes that the image has boundaries. It assumes that the left boundary of the image is 1 and the right boundary is w. Feature tracking can only be performed between 1-w. If it exceeds the image boundary, the feature point tracking is considered to have failed.

[0090] For a panorama, the image is obtained by unfolding the spherical model along a meridian. By folding and aligning the left and right borders of the panorama, a cylindrical image can be obtained. In this cylindrical image, column w+1 corresponds to column 1 of the original image, and column 0 corresponds to column w of the original image. When the feature point exceeds the left and right borders of the original panorama, it can be quickly mapped to the image area on the other side of the border.

[0091] In an embodiment of the present application, the multiple split images are pieced together into a cylindrical image, and based on this, a cylindrical surface feature tracking solution is proposed, which can significantly improve the accuracy of feature tracking and the life cycle of feature points.

[0092] It should be noted that since a mask is set on the perspective stitching image, the mask is used to eliminate the impact of the north and south pole distortion of the panoramic image on the stability of the system. The image processing device can extract feature points of the perspective stitching image in the effective mask area according to the coverage position of the mask, and no feature points are extracted for the perspective stitching image in the invalid mask area.

[0093] Specifically, the image processing device can extract features from the effective area of ​​the mask in the perspective stitched image, obtain target feature points in the perspective stitched image, and realize visual tasks such as vision-based target recognition, target tracking, and target positioning through the target feature points. For the target recognition algorithm, target tracking algorithm, and target positioning algorithm, a general algorithm can be used, and the input of the algorithm is the target feature points in the perspective stitched image extracted by this application.

[0094] Specifically, the image processing device can also perform feature extraction on the effective area of ​​the mask in the panoramic image to obtain the target feature points in the panoramic image. The image processing device projects the target feature points of the panoramic image to the perspective stitching image, thereby avoiding the error caused by directly extracting the feature points from the perspective stitching image. Regarding the projection relationship between the panoramic image and the perspective stitching image, the projection relationship can be calibrated in the process of stitching the panoramic image to obtain the perspective stitching image.

[0095] Specifically, the image processing device may copy the left pixel columns of n columns of the leftmost perspective image and splice them to the right edge of the rightmost perspective image; copy the right pixel columns of n columns of the rightmost perspective image and splice them to the left edge of the leftmost perspective image. Thus, n columns of pixel columns are added to the leftmost perspective image, and n columns of pixel columns are added to the rightmost perspective image.

[0096] By expanding the pixel columns of the leftmost and rightmost perspective images, when feature points exceed the left and right boundaries of the original panorama, they can be quickly mapped to the image area on the other side of the boundary, which can effectively improve the accuracy of feature point extraction.

[0097] When stitching perspective images, the image processing device can stitch the leftmost perspective image after copying the pixel column, the rightmost perspective image after copying the pixel column, and other perspective images according to the perspective direction to obtain a perspective stitching image corresponding to the panoramic image.

[0098] In an embodiment of the present application, an image processing device acquires a panoramic image; splits the panoramic image into several perspective views along a first spherical direction of the panoramic image; acquires a projection range of the panoramic image along a second spherical direction of the panoramic image, wherein the first spherical direction is one of the longitude direction and the latitude direction of the panoramic image, and the second spherical direction is the other of the longitude direction and the latitude direction of the panoramic image; sets a mask on each of the perspective views according to the projection range; and splices the several perspective views after setting the mask, thereby acquiring a perspective view splicing image corresponding to the panoramic image. In the above manner, the image processing device proposes a perspective splitting masking scheme for the problem of excessive distortion of the north and south poles of the panoramic image, which can improve the stability of tracking and positioning, and proposes a panoramic view splitting scheme, which, on the one hand, utilizes the full information advantage of the panoramic image, and on the other hand, does not need to process complex camera models such as fisheye cameras and spherical cameras; can better process panoramic images, eliminate the distortion of the north and south poles of the panoramic image, and can perform feature tracking across image boundaries.

[0099] The traditional panoramic visual inertial tracking and positioning system does not take into account that the external parameters between the panoramic image and the inertial sensor may be different for each device, or that the external parameters may change after the device is subjected to external force. This method proposes an online calibration scheme for the external parameters of the panoramic camera, which can optimize the external parameter information of the panoramic camera online in real time.

[0100] Please refer to Figure 5 , Figure 5 It is a flowchart of another embodiment of the image processing method provided by the present application.

[0101] Specifically, if Figure 5 As shown, the image processing method of the embodiment of the present application specifically includes the following steps:

[0102] Step S16: Obtain the perspective coordinates and world coordinates of the calibration points in the perspective stitched image.

[0103] In the embodiment of the present application, the symbols used in the online calibration scheme of the external parameters of the panoramic camera are first introduced below:

[0104] In this scheme, the inertial navigation coordinate system is taken as the device's own coordinate system, and the displacement of the device can be expressed as G p B , the rotation can be expressed as G R B .

[0105] In the scheme, the relative position between the panoramic camera and the inertial sensor is B p C , the relative rotation is B R C .

[0106] The scheme assumes that the panorama is split into four virtual pinhole camera perspectives: A, B, C, D. Without loss of generality, assume that one of the cameras {p 0 The rotation relationship between} and the panoramic camera is

[0107] Virtual Pinhole Camera 0 The pose of} is:

[0108]

[0109] in, is the relative position of the virtual pinhole camera in the world coordinate system, is the relative rotation of the virtual pinhole camera in the world coordinate system.

[0110] Assume the position of the 3D point in the world coordinate system G p f , projecting it onto the virtual pinhole {C 0}, the projection relationship is:

[0111]

[0112] in, is the position of the 3D point in the virtual pinhole camera coordinate system, is the relative rotation between the virtual pinhole camera and the world coordinate system, is the relative displacement between the world coordinate system and the virtual pinhole camera coordinate system, is the relative displacement between the panoramic camera and the virtual pinhole camera, G p B It is the displacement of the device when the inertial navigation coordinate system is taken as the device's own coordinate system.

[0113] Step S17: Calibrate the relative rotation relationship between the omnidirectional camera and the inertial sensor using the perspective coordinates, world coordinates, device posture, and virtual rotation relationship between the virtual pinhole camera and the omnidirectional camera of the calibration point.

[0114] In the embodiment of the present application, the Jacobian matrix of the online calibration external parameter is calculated:

[0115] B R C The Jacobian matrix of :

[0116]

[0117] in, is the position of the 3D point in the virtual pinhole camera coordinate system, G p f is the position of the three-dimensional point in the world coordinate system, is the relative displacement between the world coordinate system and the virtual pinhole camera coordinate system.

[0118] Step S18: calibrate the relative position relationship between the omnidirectional camera and the inertial sensor using the relative rotation relationship and the virtual rotation relationship.

[0119] In the embodiment of the present application, the Jacobian matrix of the online calibration external parameter is calculated:

[0120] B p C The Jacobian matrix of :

[0121]

[0122] Among them, π is the camera projection model, is the relative rotation between the virtual pinhole camera and the world coordinate system.

[0123] Among them, the Jacobian matrix of the calibration parameters represents its linearized gradient direction. Currently, common visual inertial tracking and positioning systems are optimized based on the gradient direction, such as extended Kalman filter, gradient descent, Gauss-Newton and other optimization methods.

[0124] In the embodiment of the present application, the image processing device takes into account the difference between the actual external parameters of the panoramic camera and the inertial sensor and the offline calibration external parameters, and performs real-time online compensation on the external parameters, which can improve the accuracy of the tracking and positioning algorithm.

[0125] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0126] In order to implement the image processing method of the above embodiment, the present application also proposes an image processing device, for details, please refer to Figure 6 , Figure 6 It is a structural schematic diagram of an embodiment of an image processing device provided by the present application.

[0127] The image processing device 400 of the embodiment of the present application includes an acquisition module 41 , a splitting module 42 , a mask module 43 and a stitching module 44 .

[0128] Wherein, the acquisition module 41 is used to acquire a panoramic image.

[0129] The splitting module 42 is used to split the panoramic image into a plurality of perspective images along a first spherical direction of the panoramic image.

[0130] The mask module 43 is used to obtain the projection range of the panoramic image along the second spherical direction of the panoramic image, where the first spherical direction is one of the longitude direction and the latitude direction of the panoramic image, and the second spherical direction is the other of the longitude direction and the latitude direction of the panoramic image; and set a mask on each of the perspective views according to the projection range.

[0131] The stitching module 44 is used to stitch the plurality of perspective images after setting the mask, so as to obtain a perspective stitching image of the panoramic image.

[0132] In order to implement the image processing method of the above embodiment, the present application also proposes another image processing device, which can be found in detail. Figure 7 , Figure 7 It is a structural schematic diagram of another embodiment of the image processing device provided by the present application.

[0133] The image processing device 500 of the embodiment of the present application includes a memory 51 and a processor 52 , wherein the memory 51 and the processor 52 are coupled.

[0134] The memory 51 is used to store program data, and the processor 52 is used to execute the program data to implement the image processing method described in the above embodiment.

[0135] In this embodiment, the processor 52 may also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip having the ability to process signals. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or the processor 52 may also be any conventional processor, etc.

[0136] To implement the image processing method of the above embodiment, the present application also provides a computer-readable storage medium, such as Figure 8 As shown, the computer-readable storage medium 600 is used to store program data 61. When the program data 61 is executed by a processor, it is used to implement the image processing method described in the above embodiment.

[0137] The present application also provides a computer program product, wherein the computer program product includes a computer program, and the computer program is operable to enable a computer to execute the image processing method as described in the embodiment of the present application. The computer program product may be a software installation package.

[0138] The image processing method described in the above embodiment of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0139] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image processing method, characterized in that: The image processing method comprises: Acquire panoramic images; Splitting the panoramic image into a plurality of perspective images along a first spherical direction of the panoramic image; Acquire a projection range of the panoramic image along a second spherical direction of the panoramic image, wherein the first spherical direction is one of a longitude direction and a latitude direction of the panoramic image, and the second spherical direction is the other of the longitude direction and the latitude direction of the panoramic image; Setting a mask on each of the perspective images according to the projection range; The masked perspective images are stitched together to obtain a perspective image stitching image corresponding to the panoramic image.

2. The image processing method according to claim 1, characterized in that: The acquiring the projection range of the panoramic image along the second spherical direction of the panoramic image comprises: The preset angles are respectively extended in the south latitude direction and the north latitude direction along the equator of the panoramic image, and the coverage range is used as the projection range of the panoramic image.

3. The image processing method according to claim 1, characterized in that: After acquiring the perspective stitching image corresponding to the panoramic image, the image processing method further includes: Extracting features of the effective area of ​​the mask in the perspective stitched image to obtain target feature points in the perspective stitched image; and, Visually-based target tracking and / or positioning is performed based on the target feature points of the perspective stitched image.

4. The image processing method according to claim 1, characterized in that: After acquiring the perspective stitching image corresponding to the panoramic image, the image processing method further includes: Performing feature extraction on a valid area of ​​the mask in the panoramic image to obtain target feature points in the panoramic image; According to the projection relationship between the panoramic image and the perspective stitched image, projecting the target feature points of the panoramic image to the target feature points of the perspective stitched image; and, Visually-based target tracking and / or positioning is performed based on the target feature points of the perspective stitched image.

5. The image processing method according to claim 4 or 5, characterized in that: The image processing method further comprises: Acquire a plurality of continuous frames of perspective stitched images, and identify target feature points on the plurality of continuous frames of perspective stitched images; After the target feature point moves from left to right to the right boundary of the perspective stitching image of the continuous multiple frames, continue searching for the target feature point from the left boundary of the next frame of the perspective stitching image; Alternatively, after the target feature point moves from right to left to the left boundary of the perspective stitching image of the continuous multiple frames, the target feature point is continuously searched from the right boundary of the next frame of the perspective stitching image.

6. The image processing method according to claim 1, characterized in that: The step of splicing the plurality of perspective views after setting the mask comprises: Obtaining a preset number of left pixel columns of the leftmost perspective image after masking, and a preset number of right pixel columns of the rightmost perspective image; Copy the left pixel column to the right border of the rightmost perspective image, and copy the right pixel column to the left border of the leftmost perspective image; Join the right border of the left perspective image and the left border of the right perspective image in the adjacent perspective images.

7. The image processing method according to claim 1, characterized in that: After obtaining the perspective stitching images of the plurality of perspective images, the image processing method further includes: Obtaining perspective coordinates and world coordinates of the calibration points in the perspective stitched image; Calibrate the relative rotation relationship between the omnidirectional camera and the inertial sensor by using the perspective coordinates, world coordinates, device posture, and virtual rotation relationship between the virtual pinhole camera and the omnidirectional camera of the calibration point; The relative position relationship between the omnidirectional camera and the inertial sensor is calibrated using the relative rotation relationship and the virtual rotation relationship.

8. An image processing device, characterized in that: The image processing device comprises an acquisition module, a splitting module, a mask module and a splicing module; Wherein, the acquisition module is used to acquire a panoramic image; The splitting module is used to split the panoramic image into a plurality of perspective images along a first spherical direction of the panoramic image; The mask module is used to obtain a projection range of the panoramic image along a second spherical direction of the panoramic image, wherein the first spherical direction is one of a longitude direction and a latitude direction of the panoramic image, and the second spherical direction is the other of the longitude direction and the latitude direction of the panoramic image; and set a mask on each of the perspective images according to the projection range; The stitching module is used to stitch a plurality of perspective images after setting the mask, so as to obtain a perspective stitching image corresponding to the panoramic image.

9. An image processing device, characterized in that: The image processing device comprises a processor and a memory, wherein program data is stored in the memory, and the processor is used to execute the program data to implement the image processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program data, and when the program data is executed by a processor, it is used to implement the image processing method according to any one of claims 1 to 7.

Images