Image processing method, image processing device and computer readable storage medium

Abstract

The invention provides an image processing method, an image processing device and a computer readable storage medium. The image processing method comprises the following steps: acquiring an omnidirectional image; splitting the omni-directional image into a plurality of perspective views which are sequentially arranged according to a preset view angle change direction along the spherical longitude or the spherical latitude of the omni-directional image; and sequentially splicing the right boundary of the left perspective view and the left boundary of the right perspective view in the adjacent perspective views so as to obtain a spliced image of the plurality of perspective views. Through the above mode, the image processing device better solves the problem of serious edge distortion of the omnidirectional image through a visual angle splitting and image splicing scheme, does not need to change a camera model into a spherical camera model, and supports the omnidirectional camera to output a high-precision panoramic image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, an image processing device, and a computer-readable storage medium. Background Art

[0002] There are many types of panoramic cameras on the market. Among them, the most common solution for consumer-level panoramic cameras is to use a front- and rear-view binocular omnidirectional camera. How to perform high-precision image acquisition based on a binocular omnidirectional camera has also become a current research direction.

[0003] However, there are two main solutions for the traditional binocular omnidirectional camera image acquisition system: First, directly use the original image of the camera to run the binocular image acquisition system, which has the problem of serious omnidirectional image distortion, especially at the pixel points at the edge of the image. The distortion has a great impact on the stability of the image acquisition system. Second, stitch the binocular omnidirectional images into a panoramic image and run the image acquisition system directly on the panoramic image; this method requires changing the traditional pinhole camera projection model to a spherical camera model, and at the same time, changing the traditional reprojection error function to a spherical angle error function. Traditional image acquisition systems are all based on pinhole cameras or abstract pinhole camera models. Changing them to spherical camera models requires re-deriving most of the formulas and parameter thresholds in the system, which greatly increases the difficulty of adaptation. Summary of the invention

[0004] The present application provides an image processing method, an image processing device, and a computer-readable storage medium.

[0005] The present application provides an image processing method, which includes:

[0006] Acquire omnidirectional images;

[0007] Splitting the omnidirectional image into a plurality of perspective images arranged in sequence according to a predetermined viewing angle change direction along the spherical longitude or spherical latitude of the omnidirectional image;

[0008] The right border of the left perspective image and the left border of the right perspective image in the adjacent perspective images are sequentially spliced ​​together to obtain a spliced ​​image of the plurality of perspective images.

[0009] In some embodiments, splitting the omnidirectional image into a plurality of perspective views along the spherical longitude or spherical latitude of the omnidirectional image comprises:

[0010] Acquire a first correspondence between a first image coordinate system of the omnidirectional image and a spherical coordinate system;

[0011] Acquire a second correspondence between the spherical coordinate system and a second image coordinate system of the perspective image;

[0012] Determine a mapping relationship between the first image coordinate system and the second image coordinate system based on the first corresponding relationship and the second corresponding relationship;

[0013] The omnidirectional image is split into a plurality of perspective images along the spherical longitude or spherical latitude of the omnidirectional image according to the mapping relationship.

[0014] In some embodiments, splitting the omnidirectional image into a plurality of perspective images along the spherical longitude or spherical latitude of the omnidirectional image according to the mapping relationship comprises:

[0015] Splitting the omnidirectional image into a plurality of omnidirectional sub-images according to a preset viewing angle along the spherical longitude of the omnidirectional image;

[0016] Each omnidirectional sub-image is converted into a perspective image according to the mapping relationship.

[0017] In some embodiments, the splitting of the omnidirectional image into a plurality of omnidirectional sub-images according to a preset viewing angle along the spherical longitude of the omnidirectional image includes:

[0018] The omnidirectional image is split into a plurality of omnidirectional sub-images along the spherical longitude of the omnidirectional image according to a first preset viewing angle, and along the spherical latitude of the omnidirectional image according to a second preset viewing angle.

[0019] In some embodiments, after splicing the right border of the left perspective image and the left border of the right perspective image in the adjacent perspective images, the image processing method further includes:

[0020] Acquire a preset number of left pixel columns of the leftmost perspective image and a preset number of right pixel columns of the rightmost perspective image;

[0021] A preset number of left pixel columns of the leftmost perspective view are copied to the outside of the right boundary of the rightmost perspective view, and a preset number of right pixel columns of the rightmost perspective view are copied to the outside of the left boundary of the leftmost perspective view, and the obtained image is used as a stitched image of several perspective views.

[0022] In some embodiments, the omnidirectional image is collected by a rolling shutter omnidirectional camera;

[0023] After obtaining the stitched images of the plurality of perspective images, the image processing method further comprises:

[0024] Obtaining a time deviation of the rolling shutter omnidirectional camera;

[0025] Based on the time deviation, obtaining a compensation device posture of the rolling shutter omnidirectional camera;

[0026] Acquire a third corresponding relationship between the first image coordinate system and the camera coordinate system of the rolling shutter omnidirectional camera;

[0027] Obtaining a reprojection error term using the compensation device posture and the third corresponding relationship;

[0028] The stitched image is compensated using the reprojection error term.

[0029] In some embodiments, obtaining the time deviation of the rolling shutter omnidirectional camera includes:

[0030] Obtain the row number of the current feature point in the omnidirectional image;

[0031] Get the exposure time of each row of the rolling shutter omnidirectional camera:

[0032] Obtaining a sensor time deviation between the rolling shutter omnidirectional camera and the inertial sensor;

[0033] The time deviation of the sensor, the number of rows of the current feature point, and the exposure time of each row are used to determine the time deviation of the exposure time point of the current feature point and the inertial sensor.

[0034] In some embodiments, after acquiring the stitched images of the plurality of perspective views, the image processing method further includes:

[0035] Acquire a plurality of continuous frames of stitched images, and identify target feature points on the plurality of continuous frames of stitched images;

[0036] After the target feature point moves from left to right to the right boundary of the stitched image of the continuous multiple frames, continue searching for the target feature point from the left boundary of the next stitched image;

[0037] Alternatively, after the target feature point moves from right to left to the left boundary of the stitched image of the continuous multiple frames, the target feature point is continuously searched from the right boundary of the next stitched image.

[0038] In some embodiments, after acquiring the stitched images of the plurality of perspective views, the image processing method further includes:

[0039] constructing a map based on the stitched image;

[0040] The position of the target is located on the map, or path planning is performed.

[0041] The present application also provides an image processing device, which includes a processor and a memory, wherein program data is stored in the memory, and the processor is used to execute the program data to implement the above-mentioned image processing method.

[0042] The present application also provides a computer-readable storage medium, which is used to store program data. When the program data is executed by a processor, it is used to implement the above-mentioned image processing method.

[0043] The beneficial effects of the present application are: the image processing device acquires an omnidirectional image; splits the omnidirectional image into a plurality of perspective images arranged in sequence according to a predetermined perspective change direction along the spherical longitude or spherical latitude of the omnidirectional image; sequentially splices the right border of the left perspective image and the left border of the right perspective image in adjacent perspective images, thereby acquiring a spliced ​​image of the plurality of perspective images. In the above manner, the image processing device better handles the problem of severe edge distortion of the omnidirectional image through a perspective splitting and image splicing scheme, without changing the camera model to a spherical camera model, and supports the omnidirectional camera to output high-precision panoramic images. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0045] Figure 1 It is a flowchart of an embodiment of an image processing method provided by the present application;

[0046] Figure 2 is a schematic diagram of a forward-looking omnidirectional camera image provided by this application;

[0047] Figure 3 is a schematic diagram of a rear-view omnidirectional camera image provided by this application;

[0048] Figure 4 is a schematic diagram of an embodiment of a perspective image provided by the present application;

[0049] Figure 5 yes Figure 1 The specific flow chart of step S14 of the image processing method shown;

[0050] Figure 6 is a flowchart of another embodiment of the image processing method provided by the present application;

[0051] Figure 7 is a structural schematic diagram of an embodiment of an image processing device provided by the present application;

[0052] Figure 8 is a structural schematic diagram of another embodiment of the image processing device provided by the present application;

[0053] Fig. 9 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] Visual inertial tracking and positioning systems are important underlying technologies in computer vision, robotics, unmanned vehicles, 3D reconstruction, and augmented reality. Robust and efficient visual inertial tracking and positioning solutions have important practical value in these fields. However, due to problems with existing technologies, the stability of visual inertial tracking and positioning systems is poor.

[0056] In order to solve the problems of the above-mentioned binocular omnidirectional camera visual inertial tracking and positioning system, the present application proposes a tracking and positioning solution based on splitting the perspective of the binocular omnidirectional image into several perspective images.

[0057] Please refer to Figure 1 , Figure 1 It is a flowchart of an embodiment of an image processing method provided by the present application.

[0058] The image processing method of the present application is applied to an image processing device, wherein the image processing device of the present application can be a server or a terminal device, etc., or a system in which a server and a terminal device cooperate with each other. Accordingly, the various parts included in the image processing device, such as various units, sub-units, modules, and sub-modules, can all be set in the server, or can be set in the server and the terminal device respectively.

[0059] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, such as a distributed system, a cloud, etc., or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules used to provide distributed servers, or it can be implemented as a single software or software module, which is not specifically limited here. In some possible implementations, the image processing method of the embodiment of the present application can be implemented by a processor calling computer-readable instructions stored in a memory.

[0060] In a specific implementation, the image processing device may also be a robot, an unmanned vehicle, a drone, etc., in which case the image processing result may be used for visual positioning and navigation. In another specific implementation, the image processing device may also be an AR (Augmented Reality) device, a VR (Virtual Reality) device, etc., in which case the image processing result may be used for enhanced implementation technology, virtual reality technology, etc. The specific product types of image processing devices are not listed here one by one.

[0061] This application splits the binocular omnidirectional image into multiple perspective images along the spherical longitude direction: on the one hand, there is no need to use a spherical camera model while retaining the most complete information of the binocular omnidirectional image; on the other hand, there is no need to perform feature extraction and tracking on the omnidirectional image, which can significantly improve the accuracy and robustness of feature tracking; at the same time, the perspective image is the most ideal pinhole projection model, and there is no need to process the distortion information in the camera; the viewing angle of the perspective camera is smaller than that of the omnidirectional camera, and feature processing only on the perspective image can avoid the severely distorted areas in the omnidirectional image, thereby improving the accuracy and stability of the features.

[0062] Specifically, if Figure 1 As shown, the image processing method of the embodiment of the present application specifically includes the following steps:

[0063] Step S11: Acquire an omnidirectional image.

[0064] In the embodiment of the present application, the image processing device uses a monocular omnidirectional camera or a binocular omnidirectional camera to collect an omnidirectional image of the target area. Among them, the omnidirectional camera (Omnidirectional Camera Model) is a camera with a 360-degree field of view on the horizontal plane, or a field of view covering a hemisphere or (approximately) the entire sphere.

[0065] Among them, the calibration internal parameters of a monocular omnidirectional camera, a binocular omnidirectional camera or a multi-omnidirectional camera including at least three imaging units can use various forms of camera models and distortion models, including but not limited to: pinhole camera model, spherical camera model, fisheye camera model, equidistant distortion model, radial tangential distortion model, etc.

[0066] Step S12: splitting the omnidirectional image into a plurality of perspective images arranged in sequence according to a predetermined viewing angle change direction along the spherical longitude or spherical latitude of the omnidirectional image.

[0067] In the embodiment of the present application, the perspective splitting process can be understood as the binocular omnidirectional image is unfolded along a spherical surface along a certain longitude or latitude cylinder, so that its left and right boundaries can be completely fitted. When the cylinder is unfolded, the perspective image needs to be arranged according to a certain predetermined perspective change direction. For example, the present application can split the omnidirectional image according to the perspective change scheme that continuously changes from the left perspective to the right perspective. In other embodiments, other perspective change schemes can also be adopted, such as continuously changing from the right perspective to the left perspective.

[0068] Specifically, the internal and external parameters of the binocular omnidirectional camera can be obtained by offline pre-calibration. The image processing device projects each feature point on the image onto a three-dimensional spherical surface through the internal and external parameters of the camera, and then obtains the projection coordinates of a three-dimensional spherical surface. It can be considered that each point on the binocular omnidirectional image can correspond to a point on the spherical surface one by one. Thus, the first correspondence between the image coordinate system of the omnidirectional image and the spherical coordinate system can be obtained.

[0069] Similarly, the image processing device can also construct a number of perspective viewpoints without a common view relationship, and project points on a unit sphere onto the perspective viewpoints respectively, wherein the points on the sphere and the points on the perspective view can also form a one-to-one correspondence. Thus, a second correspondence relationship between the spherical coordinate system and the image coordinate system of the perspective view can be obtained.

[0070] The first correspondence and the second correspondence can be presented in the form of a homography matrix or a projection matrix. The image processing device can construct a one-to-one correspondence from the binocular omnidirectional image to the split perspective image based on the first correspondence and the second correspondence, that is, a mapping relationship or a projection relationship between the image coordinate system of the omnidirectional image and the image coordinate system of the perspective image.

[0071] Furthermore, the image processing device may split the omnidirectional image into a plurality of perspective images along the spherical longitude or spherical latitude of the omnidirectional image according to the mapping relationship or projection relationship determined in the above process.

[0072] Specifically, the image processing device maps all pixel points in the binocular omnidirectional image to a spherical coordinate system through the camera's internal and external parameters, and rotates the sphere so that the directions of its north and south poles are consistent with the Z-axis direction of the inertial coordinate system.

[0073] There are many ways to select the perspective of image splitting, for example, splitting a 360-degree panoramic image into three 120-degree perspective images, or four 90-degree perspective images, or five 72-degree perspective images, or six 60-degree perspective images, etc. In the following embodiments, the perspective splitting scheme is continued by taking the example of splitting a 360-degree panoramic image into four 90-degree perspective images. Among them, the 360-degree panoramic image provided in this application can be seen in FIG. Figure 2 and Figure 3 , Figure 2is a schematic diagram of a forward-looking omnidirectional camera image provided in this application, Figure 3 It is a schematic diagram of a rear-view omnidirectional camera image provided in this application.

[0074] Specifically, the split perspective image has a 90-degree viewing angle in the horizontal direction. To this end, the image processing device can divide the longitude of the sphere into four equal parts, with 45° east longitude, 45° west longitude, 135° east longitude and 135° west longitude as the horizontal optical center, 90° as the field of view, and four spherical areas. It should be noted that the image processing device can also divide the sphere into four equal parts according to the latitude, which will not be repeated here.

[0075] The perspective view has a 90-degree viewing angle in the vertical direction, so the 45-degree viewing angle needs to be mapped to the corresponding areas at the south and north latitudes respectively.

[0076] The image processing device uses the above perspective splitting scheme to Figure 2 and Figure 3 The 360-degree panoramic image shown is split into Figure 4 The perspective view shown, Figure 4 It is a schematic diagram of an embodiment of a perspective image provided by the present application.

[0077] in, Figure 4 The perspective images in the Figure 1 ,perspective Figure 2 ,perspective Figure 3 And perspective Figure 4 .

[0078] For perspective Figure 1 , whose image area is a panoramic image within the spherical surface from 0° east longitude to 90° east longitude, and from 45° north of the equator to 45° south of the equator. The image feature points in this area can be projected onto the normalized camera plane of the pinhole camera, and mapped to a virtual perspective image through a certain established distortion-free camera intrinsic parameter. Similarly, perspective Figure 2 , the image area is spherical longitude 90° to east longitude 180°; perspective Figure 3 , the image area is from 180°W to 90°W; perspective Figure 4 , and its image area is from 90°W to 0°W.

[0079] Given the first correspondence between the image coordinate system of the omnidirectional image and the spherical coordinate system, and the second correspondence between the spherical coordinate system and the image coordinate system of the perspective view, a mapping relationship or projection relationship between the image coordinate system of the omnidirectional image and the image coordinate system of the perspective view can be established. Through this mapping relationship or projection relationship, the binocular omnidirectional image is mapped to Figure 4 In the 4 perspective views shown.

[0080] In other embodiments, the image processing device may obtain the perspective image by first mapping and then splitting; or may obtain the perspective image by first splitting and then mapping, and the process will not be described in detail here.

[0081] Step S13: sequentially splicing the right border of the left perspective image and the left border of the right perspective image in adjacent perspective images, thereby obtaining a spliced ​​image of several perspective images.

[0082] In the embodiment of the present application, the split perspective images are obtained by sampling the sphere at equal distances along the meridian direction. The left border of one perspective image is seamlessly connected to the right border of another image, so it can be used Figure 4 The method shown is to stitch together several split perspective drawings to form a complete perspective stitching drawing.

[0083] The image processing method of the present application can be widely used as a complete visual inertial tracking and positioning system in various scenarios that require visual positioning and navigation, including: autonomous driving, mobile robots, drones, smart phones, virtual reality helmets, augmented reality glasses, and other devices that have both binocular omnidirectional camera sensors and inertial sensor modules. Specifically, the image processing device can use the spliced ​​images to build a map, and locate and track the target on the map, or perform path planning on the map, etc.

[0084] Furthermore, based on the above perspective image stitching solution, this application provides a cylindrical feature tracking solution. For details, please refer to Figure 5 , Figure 5 yes Figure 1 The specific flow chart of step S13 of the image processing method is shown.

[0085] like Figure 5 As shown, step S13 specifically includes the following steps:

[0086] Step S131: acquiring a plurality of consecutive frames of stitched images, and identifying target feature points on the plurality of consecutive frames of stitched images.

[0087] In the embodiment of the present application, the image processing device directly extracts feature points or corner points on the perspective mosaic image.

[0088] Step S132: After the target feature point moves from left to right to the right boundary of the stitched image of the continuous multiple frames, the target feature point is continuously searched from the left boundary of the next stitched image.

[0089] Step S133: after the target feature point moves from right to left to the left boundary of the stitched image of the continuous multiple frames, the target feature point is continuously searched from the right boundary of the next stitched image.

[0090] In an embodiment of the present application, the multiple split images are pieced together into a cylindrical image, and based on this, a cylindrical surface feature tracking solution is proposed, which can significantly improve the accuracy of feature tracking and the life cycle of feature points.

[0091] Specifically, for the perspective mosaic, let the leftmost column of the first perspective image be the first column, and the rightmost column of the last perspective image be the wth column, then the effective image area of ​​the perspective mosaic is columns 1-w. The perspective mosaic image is obtained by unfolding the spherical model along a certain meridian. By folding and aligning its left and right boundaries, a cylindrical image can be obtained. On this cylindrical image, column w+1 corresponds to column 1 of the original image, and column 0 corresponds to column w of the original image. When the feature point exceeds the left and right boundaries of the original panoramic image, it can be quickly mapped to the image area on the other side of the boundary.

[0092] Furthermore, the image processing device can also copy columns w-n+1 to w in the rightmost perspective image and splice them to the leftmost perspective image. Figure 1 Column 1 to column n; copy the leftmost perspective image and splice it to the right of column w of the rightmost image. Thus, all perspective images change from column w of the original image to column w+2n of the new image, and both sides of the boundary are filled by the other side boundary image. Among them, 2n is the number of new copy and pixel columns.

[0093] In other embodiments, the number of duplicate pixel columns of the leftmost perspective image and the number of duplicate pixel columns of the rightmost perspective image may also be different.

[0094] The image processing device maps the feature points in the original image to a new image, runs a feature tracking algorithm on the new image, obtains the result of feature tracking on the new image, and maps the tracked feature points and the feature points in the left and right padded parts back to the original perspective image to obtain the cylindrical feature tracking result.

[0095] It should be noted that the number of columns in the above process is discussed for all perspective views, and the effect achieved is to copy several columns of pixels on the left side of the leftmost perspective view in all perspective views to the right side of the rightmost perspective view, and to copy several columns of pixels on the right side of the rightmost perspective view to the left side of the leftmost perspective view, thereby increasing the range of cylindrical feature recognition and tracking, thereby avoiding the situation in which part of the cylindrical feature in all perspective views is on the rightmost perspective view and the other part is on the leftmost perspective view, resulting in cylindrical feature recognition errors.

[0096] In other embodiments, the image processing device may also directly stitch together partial images of the left and right boundaries of adjacent perspective images, and perform feature tracking again on the stitched image for untracked points near the boundaries.

[0097] In an embodiment of the present application, an image processing device obtains an omnidirectional image; splits the omnidirectional image into a plurality of perspective images along the spherical longitude or spherical latitude of the omnidirectional image; obtains the left and right boundaries of each perspective image; and splices the right boundary of the left perspective image and the left boundary of the right perspective image in adjacent perspective images, thereby obtaining a spliced ​​image of the plurality of perspective images. In the above manner, the image processing device better handles the problem of severe edge distortion of the omnidirectional image through a perspective splitting and image splicing scheme, without changing the camera model to a spherical camera model, and supports the omnidirectional camera to output high-precision panoramic images.

[0098] Using the above image processing method, the tracking and positioning system can use the traditional pinhole camera model, and after simple modification, it can quickly support omnidirectional cameras. This application proposes a binocular omnidirectional perspective splitting solution, which on the one hand takes advantage of the full information of the omnidirectional image, and on the other hand does not need to process complex camera models such as fisheye cameras and spherical cameras. Moreover, in the perspective splitting process, the omnidirectional image can be better processed, the area with large distortion of the omnidirectional image is eliminated, and feature tracking across the image boundary can be performed.

[0099] Furthermore, for rolling shutter omnidirectional cameras, the camera images line by line during exposure. Since the camera moves during use, the acquisition time of each line will have microsecond differences. If this time difference is not taken into account, it will affect the accuracy of the algorithm, especially the system accuracy when the device is rotating. Figure 1 The image processing method shown performs perspective splitting, and the final feature processing is performed on the spliced ​​perspective image, which inevitably loses the original information. To address this problem, the present application proposes a rolling shutter time offset compensation solution.

[0100] Please refer to Figure 6 , Figure 6 It is a flowchart of another embodiment of the image processing method provided by the present application.

[0101] Specifically, if Figure 6 As shown, the image processing method of the embodiment of the present application specifically includes the following steps:

[0102] Step S21: obtaining the time deviation of the rolling shutter omnidirectional camera.

[0103] In the embodiment of the present application, the symbols used in the rolling shutter time offset compensation scheme are first introduced below:

[0104] In this scheme, the inertial navigation coordinate system is taken as the device's own coordinate system, and the displacement of the device can be expressed as G p B , the rotation can be expressed as G R B .

[0105] In the scheme, the relative position between a split pinhole camera and the inertial sensor is B p C , the relative rotation is B R C .

[0106] In the scheme, the pinhole camera model is assumed to be an ideal pinhole camera model, and its optical centers are c x and c y , the focal lengths are f x and f y .

[0107] In the scheme, let a pixel x on the perspective image p , through the mapping relationship of perspective splitting, it is mapped to a point x on an omnidirectional camera o =(x,y) T .

[0108] The exposure time of each row of the rolling shutter is set to t r , the time deviation between the camera and the inertial sensor is t d , a single frame image has N rows, the current feature point x o At row y.

[0109] The image processing device can obtain the current feature point x according to the above o The time deviation t between the exposure time point and the inertial system δ for:

[0110]

[0111] Step S22: Based on the time deviation, the compensated device posture of the rolling shutter omnidirectional camera is obtained.

[0112] In the embodiment of the present application, assuming that the device motion speed in the current inertial sensor coordinate system is v, and the angular velocity is ω, then the corresponding feature point p f For , the corresponding device posture is:

[0113]

[0114] in, is the rotation of the device in the current inertial sensor coordinate system, It is the displacement of the current inertial sensor coordinate system device.

[0115] Step S23: obtaining a third corresponding relationship between the first image coordinate system and the camera coordinate system of the rolling shutter omnidirectional camera.

[0116] In the embodiment of the present application, let point m = (u, v)T is the normalized two-dimensional point in the camera coordinate system, I=(i x ,i y ) T For a two-dimensional point on the image, there is an image coordinate calculation formula:

[0117]

[0118] set up C p f =(x, y, z) T is a three-dimensional point in the camera coordinate system, and there is a normalization formula:

[0119]

[0120] The position of the 3D point in the world coordinate system is known G p f , we can get C p f The transformation formula is:

[0121]

[0122] in, C R G is the relative rotation between the pinhole camera and the world coordinate system, is the rotation of the device in the current inertial sensor coordinate system, B R C is the relative rotation between a split pinhole camera and the inertial sensor, To calculate the displacement of the device when the inertial navigation coordinate system is taken as the device's own coordinate system, G p C is the relative displacement between the world coordinate system and the camera coordinate system.

[0123] Step S24: using the compensation device posture and the third correspondence to obtain a reprojection error term.

[0124] In the embodiment of the present application, the above formulas can be combined to obtain the reprojection error term formula of the three-dimensional point:

[0125]

[0126] Among them, π is the camera projection model, G p f is the position of the three-dimensional point in the world coordinate system, C R G is the relative rotation between the pinhole camera and the ground coordinate system, G p C is the relative displacement between the world coordinate system and the camera coordinate system, is the rotation of the device in the current inertial sensor coordinate system, B R C is the relative rotation between a split pinhole camera and the inertial sensor.

[0127] Step S25: compensating the stitched image using the reprojection error term.

[0128] In the embodiment of the present application, the time deviation of the rolling shutter can be added to the reprojection error term according to formula (7) in step S24, so that the system can compensate for the time deviation of the rolling shutter online in real time.

[0129] Among them, the reprojection error refers to the difference between the projection of the real 3D space point on the image plane (that is, the pixel point on the image) and the reprojection (actually the virtual pixel point obtained by our calculation value). Due to various reasons, the calculated value will not be completely consistent with the actual situation, that is, this difference cannot be exactly 0. At this time, it is necessary to minimize the sum of these differences to obtain the optimal camera pose parameters and the coordinates of the 3D space point.

[0130] Reprojection error is often used in computer vision. For example, when calculating the homography matrix and projection matrix, the cost function is often constructed using the reprojection error, and then the cost function is minimized to optimize the homography matrix or projection matrix. The reason for using the reprojection error is that it not only takes into account the calculation error of the homography matrix, but also the measurement error of the image points, so its accuracy will be higher.

[0131] In the embodiment of the present application, the image processing device takes into account the influence of the rolling shutter on the camera imaging, and adds a time deviation in the omnidirectional image segmentation process, and compensates for the rolling shutter online, which can improve the accuracy and stability of the system.

[0132] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0133] In order to implement the image processing method of the above embodiment, the present application also proposes an image processing device, for details, please refer to Figure 7 , Figure 7 It is a structural schematic diagram of an embodiment of an image processing device provided by the present application.

[0134] The image processing device 400 of the embodiment of the present application includes an acquisition module 41 , a splitting module 42 and a stitching module 43 .

[0135] Wherein, the acquisition module 41 is used to acquire omnidirectional images.

[0136] The splitting module 42 is used to split the omnidirectional image into a plurality of perspective images arranged in sequence according to a predetermined viewing angle change direction along the spherical longitude or spherical latitude of the omnidirectional image.

[0137] The stitching module 43 is used to stitch the right border of the left perspective image and the left border of the right perspective image in sequence among the adjacent perspective images, so as to obtain a stitched image of the plurality of perspective images.

[0138] It can be understood that each module in the device 400 corresponds one-to-one to the steps in the aforementioned method embodiment. The specific implementation of each module can refer to the description of the corresponding steps in the aforementioned method embodiment, and will not be repeated here.

[0139] In order to implement the image processing method of the above embodiment, the present application also proposes another image processing device, which can be found in detail. Figure 8 , Figure 8 It is a structural schematic diagram of another embodiment of the image processing device provided by the present application.

[0140] The image processing device 500 of the embodiment of the present application includes a memory 51 and a processor 52 , wherein the memory 51 and the processor 52 are coupled.

[0141] The memory 51 is used to store program data, and the processor 52 is used to execute the program data to implement the image processing method described in the above embodiment.

[0142] In this embodiment, the processor 52 may also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip having the ability to process signals. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or the processor 52 may also be any conventional processor, etc.

[0143] To implement the image processing method of the above embodiment, the present application also provides a computer-readable storage medium, such as Fig. 9As shown, the computer-readable storage medium 600 is used to store program data 61. When the program data 61 is executed by a processor, it is used to implement the image processing method described in the above embodiment.

[0144] The present application also provides a computer program product, wherein the computer program product includes a computer program, and the computer program is operable to enable a computer to execute the image processing method as described in the embodiment of the present application. The computer program product may be a software installation package.

[0145] The image processing method described in the above embodiment of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0146] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image processing method, characterized in that: The image processing method comprises: Acquire omnidirectional images; Splitting the omnidirectional image into a plurality of perspective images arranged in sequence according to a predetermined viewing angle change direction along the spherical longitude or spherical latitude of the omnidirectional image; The right border of the left perspective image and the left border of the right perspective image in the adjacent perspective images are sequentially spliced ​​together to obtain a spliced ​​image of the plurality of perspective images.

2. The image processing method according to claim 1, characterized in that: The omnidirectional image is split into a plurality of perspective images along the spherical longitude or spherical latitude of the omnidirectional image, including: Acquire a first correspondence between a first image coordinate system of the omnidirectional image and a spherical coordinate system; Acquire a second correspondence between the spherical coordinate system and a second image coordinate system of the perspective image; Determine a mapping relationship between the first image coordinate system and the second image coordinate system based on the first corresponding relationship and the second corresponding relationship; The omnidirectional image is split into a plurality of perspective images along the spherical longitude or spherical latitude of the omnidirectional image according to the mapping relationship.

3. The image processing method according to claim 2, characterized in that: The step of splitting the omnidirectional image into a plurality of perspective images along the spherical longitude or spherical latitude of the omnidirectional image according to the mapping relationship comprises: Splitting the omnidirectional image into a plurality of omnidirectional sub-images according to a preset viewing angle along the spherical longitude of the omnidirectional image; Each omnidirectional sub-image is converted into a perspective image according to the mapping relationship.

4. The image processing method according to claim 3, characterized in that: The omnidirectional image is split into a plurality of omnidirectional sub-images according to a preset viewing angle along the spherical longitude of the omnidirectional image, including: The omnidirectional image is split into a plurality of omnidirectional sub-images along the spherical longitude of the omnidirectional image according to a first preset viewing angle, and along the spherical latitude of the omnidirectional image according to a second preset viewing angle.

5. The image processing method according to claim 1, characterized in that: After splicing the right border of the left perspective image and the left border of the right perspective image in the adjacent perspective images, the image processing method further includes: Acquire a preset number of left pixel columns of the leftmost perspective image and a preset number of right pixel columns of the rightmost perspective image; A preset number of left pixel columns of the leftmost perspective view are copied to the outside of the right boundary of the rightmost perspective view, and a preset number of right pixel columns of the rightmost perspective view are copied to the outside of the left boundary of the leftmost perspective view, and the obtained image is used as a stitched image of several perspective views.

6. The image processing method according to any one of claims 1 to 5, characterized in that: The omnidirectional image is collected by a rolling shutter omnidirectional camera; After obtaining the stitched images of the plurality of perspective images, the image processing method further comprises: Obtaining a time deviation of the rolling shutter omnidirectional camera; Based on the time deviation, obtaining a compensation device posture of the rolling shutter omnidirectional camera; Acquire a third corresponding relationship between the first image coordinate system and the camera coordinate system of the rolling shutter omnidirectional camera; Obtaining a reprojection error term using the compensation device posture and the third corresponding relationship; The stitched image is compensated using the reprojection error term.

7. The image processing method according to claim 6, characterized in that: The step of obtaining the time deviation of the rolling shutter omnidirectional camera includes: Obtain the row number of the current feature point in the omnidirectional image; Get the exposure time of each row of the rolling shutter omnidirectional camera: Obtaining a sensor time deviation between the rolling shutter omnidirectional camera and the inertial sensor; The time deviation of the sensor, the number of rows of the current feature point, and the exposure time of each row are used to determine the time deviation of the exposure time point of the current feature point and the inertial sensor.

8. The image processing method according to any one of claims 1 to 5, characterized in that: After obtaining the stitched images of the plurality of perspective images, the image processing method further includes: Acquire a plurality of continuous frames of stitched images, and identify target feature points on the plurality of continuous frames of stitched images; After the target feature point moves from left to right to the right boundary of the stitched image of the continuous multiple frames, continue searching for the target feature point from the left boundary of the next stitched image; Alternatively, after the target feature point moves from right to left to the left boundary of the stitched image of the continuous multiple frames, the target feature point is continuously searched from the right boundary of the next stitched image.

9. The image processing method according to any one of claims 1 to 5, characterized in that: After obtaining the stitched images of the plurality of perspective images, the image processing method further includes: constructing a map based on the stitched image; The position of the target is located on the map, or path planning is performed.

10. An image processing device, characterized in that: The image processing device comprises a processor and a memory, wherein program data is stored in the memory, and the processor is used to execute the program data to implement the image processing method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program data, and when the program data is executed by a processor, it is used to implement the image processing method according to any one of claims 1 to 9.

Images