Camera system pose determination method and device, movable platform and related product

Abstract

The invention relates to a pose determination method and device of a camera system, equipment, a medium and a product. The method comprises the steps of obtaining a current frame fisheye image collected by a fisheye camera in a camera system and a current frame common image collected by a preset type camera, and performing feature point tracking on the current frame fisheye image to obtain current frame fisheye feature points of the current frame fisheye image, and performing feature point tracking on the current frame common image to obtain current frame common feature points of the current frame common image, and determining current pose information of the camera system according to the current frame fisheye feature points and the current frame common feature points. According to the method, the fisheye camera with the large visual angle range in the camera system can make up for the small visual angle range of the preset type of camera, and the fisheye image collected by the fisheye camera makes up for the defect that common images collected by the preset type of camera have weak textures, so that the accuracy of the determined pose of the camera system can be improved.

Description

Technical Field

[0001] The present application relates to the field of positioning technology, and in particular to a method, device, movable platform and related products for determining the position and posture of a camera system. Background Art

[0002] With the development of image processing technology, visual positioning technology has been widely used.

[0003] Taking the positioning of a camera system as an example, in the related technology, visual positioning processing is mainly performed on the images collected by each camera in the camera system to determine the position information of the camera system.

[0004] However, in the related art, there is a problem that the determined pose information of the camera system is inaccurate. Summary of the invention

[0005] Based on this, it is necessary to provide a method, device, movable platform and related products for determining the posture of a camera system in order to address the above-mentioned technical problems, so as to improve the accuracy of the determined posture information of the camera system.

[0006] In a first aspect, an embodiment of the present application provides a method for determining a position and posture of a camera system, the method comprising:

[0007] Obtain a fisheye image of a current frame captured by a fisheye camera and a normal image of a current frame captured by a preset type camera in the camera system;

[0008] Tracking feature points of the current frame fisheye image to obtain current frame fisheye feature points of the current frame fisheye image, and tracking feature points of the current frame normal image to obtain current frame normal feature points of the current frame normal image;

[0009] The current position information of the camera system is determined based on the fisheye feature points of the current frame and the common feature points of the current frame.

[0010] In one embodiment, tracking feature points of the current frame fisheye image to obtain the current frame fisheye feature points of the current frame fisheye image includes:

[0011] Obtain historical frame fisheye feature points of historical frame fisheye images captured by the fisheye camera;

[0012] According to the fisheye feature points of the historical frames, the inter-frame feature points of the current frame fisheye image are tracked to obtain the current frame fisheye feature points of the current frame fisheye image.

[0013] In one of the embodiments, the camera system includes at least one fisheye camera; if the historical frame fisheye image includes a previous frame fisheye image of the current frame fisheye image, and the previous frame fisheye image is an initial frame fisheye image, obtaining the historical frame fisheye feature points of the historical frame fisheye image acquired by the fisheye camera includes:

[0014] Obtain the previous frame of fisheye image captured by each fisheye camera;

[0015] Extracting a preset number of feature points of fisheye features from the target previous frame fisheye image in each previous frame fisheye image; the target previous frame fisheye image represents any one of the previous frame fisheye images;

[0016] According to the feature points extracted from the previous frame fisheye image of the target, the feature points on other previous frame fisheye images in each previous frame fisheye image are extracted in sequence to obtain the previous frame fisheye feature points of the previous frame fisheye image collected by each fisheye camera.

[0017] In one embodiment, based on the feature points extracted from the previous fisheye image of the target, the feature points on other previous fisheye images in each previous fisheye image are sequentially extracted, including:

[0018] Generate an image sequence from each previous frame of fisheye images; the previous frame of fisheye image of the target in the image sequence is the starting fisheye image;

[0019] For any previous fisheye image in the image sequence, based on the feature points extracted from all previous fisheye images before the previous fisheye image, the feature points tracked from the previous fisheye image;

[0020] If the number of tracked feature points on the previous frame of fisheye image is equal to the preset number of fisheye features, the tracked feature points are used as the previous frame of fisheye feature points of the previous frame of fisheye image.

[0021] In one embodiment, the method further comprises:

[0022] If the number of feature points tracked on the previous frame of fisheye image is less than the preset number of fisheye features, the number of compensating feature points of the previous frame of fisheye image is determined according to the number of feature points tracked on the previous frame of fisheye image and the preset number of fisheye features;

[0023] According to the number of compensated feature points of the previous frame of fisheye image, feature point compensation extraction is performed on the previous frame of fisheye image to obtain the previous frame of fisheye feature points of the previous frame of fisheye image.

[0024] In one embodiment, tracking feature points of a current frame normal image to obtain current frame normal feature points of the current frame normal image includes:

[0025] Obtaining common feature points of common historical frames of common images collected by cameras of a preset type;

[0026] According to the common feature points of the common images of the historical frames, the common feature points of the current frame are tracked to obtain the candidate common feature points of the common images of the current frame;

[0027] If the number of candidate common feature points of the current frame common image is equal to the preset number of common features, the candidate common feature points of the current frame common image are determined as the current frame common feature points of the current frame common image.

[0028] In one embodiment, the camera system includes at least one fisheye camera; if the historical frame fisheye image includes a previous frame fisheye image of the current frame fisheye image, the historical frame normal image includes a previous frame normal image of the current frame normal image, and the previous frame fisheye image is an initial frame fisheye image, and the previous frame normal image is an initial frame normal image, then obtaining the historical frame normal feature points of the historical frame normal image acquired by the preset type camera includes:

[0029] Obtain the previous frame of fisheye feature points of the previous frame of fisheye image captured by each fisheye camera;

[0030] According to the fisheye feature points of each previous frame, the feature points of the previous frame of ordinary image are tracked to obtain candidate ordinary feature points of the previous frame of ordinary image;

[0031] If the number of candidate common feature points of the previous frame of common image is equal to the preset number of common features, the candidate common feature points of the previous frame of common image are determined as the previous frame of common feature points of the previous frame of common image.

[0032] In one embodiment, the method further comprises:

[0033] If the number of candidate common feature points of the previous frame of common image is less than the preset number of common features, the number of compensating feature points of the previous frame of common image is determined according to the total number of candidate common feature points of the previous frame of common image and the preset number of common features;

[0034] According to the number of compensated feature points of the previous frame of ordinary image, feature point compensation extraction is performed on the previous frame of ordinary image;

[0035] The common feature points of the previous frame are determined according to the feature point compensation extraction result of the previous frame common image and the candidate common feature points.

[0036] In one embodiment, the method further comprises:

[0037] If the number of candidate ordinary feature points of the current frame ordinary image is less than the preset number of ordinary features, the feature points of the current frame ordinary image are tracked according to the historical frame ordinary feature points of the historical frame ordinary images collected by the preset type of camera to determine the current frame ordinary feature points of the current frame ordinary image.

[0038] In one embodiment, determining the current pose information of the camera system according to the fisheye feature points of the current frame and the common feature points of the current frame includes:

[0039] Obtain the previous frame of fisheye feature points of the previous frame of fisheye image captured by the fisheye camera, and the previous frame of common feature points of the previous frame of common image captured by the preset type camera;

[0040] According to the fisheye feature points of the current frame, the fisheye feature points of the previous frame, the common feature points of the current frame and the common feature points of the previous frame, a current posture observation model of the camera system is constructed;

[0041] According to the current posture observation model, the current posture optimization function of the camera system is constructed;

[0042] Based on the current pose optimization function, the current pose information of the camera system is determined.

[0043] In one embodiment, the current posture observation model includes a depth information observation model and a non-depth information observation model; and the current posture observation model of the camera system is constructed according to the fisheye feature points of the current frame, the fisheye feature points of the previous frame, the common feature points of the current frame, and the common feature points of the previous frame, including:

[0044] Determine the depth information observation model according to the fisheye feature points of the current frame and the common feature points of the current frame;

[0045] A non-depth information observation model is constructed based on the fisheye feature points of the current frame, the fisheye feature points of the previous frame, the common feature points of the current frame and the common feature points of the previous frame.

[0046] In one embodiment, determining a depth information observation model according to fisheye feature points of a current frame and common feature points of a current frame includes:

[0047] Obtaining depth feature points with depth information from the fisheye feature points of the current frame and the common feature points of the current frame; the depth feature points include the three-dimensional position coordinates of the feature points;

[0048] Determine the unit sphere coordinates of each depth feature point according to the current position coordinates of each depth feature point in the current frame image;

[0049] Determine the depth residual function of each depth feature point according to the current position coordinates of each depth feature point in the world coordinate system and the unit sphere coordinates of each depth feature point, as well as the external parameters between the current camera system coordinate system and the camera coordinate system corresponding to the current frame image to which each depth feature point belongs;

[0050] The depth information observation model is determined according to the depth residual function of each depth feature point.

[0051] In one embodiment, a non-depth information observation model is constructed based on fisheye feature points of a current frame, fisheye feature points of a previous frame, common feature points of a current frame, and common feature points of a previous frame, including:

[0052] Acquire a plurality of first non-depth feature points, a plurality of second non-depth feature points, and a plurality of third non-depth feature points from the fisheye feature points of the current frame and the common feature points of the current frame;

[0053] Constructing a non-depth residual function of each first non-depth feature point, a non-depth residual function of each second non-depth feature point, and a non-depth residual function of each third non-depth feature point;

[0054] A non-depth information observation model is determined according to the non-depth residual function of each first non-depth feature point, the non-depth residual function of each second non-depth feature point, and the non-depth residual function of each third non-depth feature point.

[0055] In one embodiment,

[0056] Each first non-depth feature point represents a current frame feature point without depth information that is in at least two current frame images and in at least two previous frame images;

[0057] Each second non-depth feature point represents a current frame feature point without depth information that is only in one current frame image and in at least two previous frame images;

[0058] Each third non-depth feature point represents a current frame feature point without depth information that is in at least two current frame images and in only one previous frame image;

[0059] The current frame image is either a current frame fisheye image or a current frame ordinary image; the previous frame image is either a previous frame fisheye image or a previous frame ordinary image.

[0060] In one embodiment, constructing a non-depth residual function of each first non-depth feature point includes:

[0061] Obtaining the previous position coordinates of each first non-depth feature point on the corresponding previous frame image;

[0062] Triangulate the front position coordinates of each first non-depth feature point to obtain the three-dimensional front position coordinates of each first non-depth feature point in the world coordinate system; and triangulate the current position coordinates of each first non-depth feature point to obtain the three-dimensional current position coordinates of each first non-depth feature point in the current camera system coordinate system;

[0063] A non-depth residual function of each first non-depth feature point is constructed according to the three-dimensional front position coordinates of each first non-depth feature point in the world coordinate system and the three-dimensional current position coordinates of each first non-depth feature point in the current camera system coordinate system.

[0064] In one embodiment, constructing a non-depth residual function of each second non-depth feature point includes:

[0065] Obtaining the previous position coordinates of each second non-depth feature point on the corresponding previous frame image;

[0066] Triangulating the front position coordinates of each second non-depth feature point to obtain the three-dimensional front position coordinates of each second non-depth feature point in the world coordinate system;

[0067] Determine the unit sphere coordinates of each second non-depth feature point according to the current position coordinates of each second non-depth feature point in the current frame image to which it belongs;

[0068] A non-depth residual function of each second non-depth feature point is constructed according to the three-dimensional front position coordinates of each second non-depth feature point in the world coordinate system, the unit sphere coordinates of each second non-depth feature point, and the external parameters between the current camera system coordinate system and the camera coordinate system corresponding to the current frame image to which each second non-depth feature point belongs.

[0069] In one embodiment, constructing a non-depth residual function of each third non-depth feature point includes:

[0070] Triangulate the current position coordinates of each third non-depth feature point on the current frame image to obtain the three-dimensional current position coordinates of each third non-depth feature point in the current camera coordinate system;

[0071] Determine the unit sphere coordinates of each third non-depth feature point according to the previous position coordinates of each third non-depth feature point in the corresponding previous frame image;

[0072] According to the three-dimensional current position coordinates of each third non-depth feature point in the current camera coordinate system, the unit sphere coordinates of each third non-depth feature point, the transformation relationship between the world coordinate system and the front camera system coordinate system, and the external parameters between the current camera system coordinate system and the camera coordinate system corresponding to the previous frame image to which each third non-depth feature point belongs, a non-depth residual function of each third non-depth feature point is constructed.

[0073] In a second aspect, an embodiment of the present application provides a device for determining a position and posture of a camera system, the device comprising:

[0074] An image acquisition module, used to acquire a fisheye image of a current frame acquired by a fisheye camera in a camera system and a normal image of a current frame acquired by a preset type camera;

[0075] A feature point tracking module is used to track the feature points of the current frame fisheye image to obtain the current frame fisheye feature points of the current frame fisheye image, and to track the feature points of the current frame ordinary image to obtain the current frame ordinary feature points of the current frame ordinary image;

[0076] The determination module is used to determine the current posture information of the camera system according to the fisheye feature points of the current frame and the common feature points of the current frame.

[0077] In a third aspect, an embodiment of the present application further provides a movable platform, the movable platform comprising:

[0078] Camera system;

[0079] processor;

[0080] a memory for storing a computer program for execution by a processor;

[0081] In which, when the processor executes the computer program, the steps of the method of any embodiment of the first aspect above are implemented.

[0082] In a fourth aspect, an embodiment of the present application further provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the method of any embodiment of the first aspect when executing the computer program.

[0083] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method of any embodiment in the above-mentioned first aspect are implemented.

[0084] In a sixth aspect, an embodiment of the present application further provides a computer program product, the computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method of any embodiment in the first aspect above.

[0085] The camera system posture determination method, device, movable platform and related products provided by the embodiments of the present application include: obtaining a current frame fisheye image captured by a fisheye camera in the camera system and a current frame ordinary image captured by a preset type camera, tracking feature points of the current frame fisheye image to obtain current frame fisheye feature points of the current frame fisheye image, and tracking feature points of the current frame ordinary image to obtain current frame ordinary feature points of the current frame ordinary image, and determining the current posture information of the camera system based on the current frame fisheye feature points and the current frame ordinary feature points; the above method can compensate for the preset type by using a fisheye camera with a large viewing angle in the camera system. The camera's viewing angle is small, and the fisheye image collected by the fisheye camera makes up for the weak texture defect of ordinary images collected by the preset type of camera, so that more feature points from the surrounding environment can be obtained in the process of determining the position and posture of the camera system. Further determining the position and posture of the camera system by more feature points can improve the accuracy of the determined camera system position and posture; at the same time, since the fisheye camera has a unique imaging model, the movement of the same feature point in the inter-frame image is small, stable and not easy to lose, thereby improving the accuracy of feature point tracking. On this basis, the accuracy of the determined camera system position and posture can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 A diagram showing an application environment of a method for determining a position and posture of a camera system in one embodiment;

[0087] Figure 2 is a schematic flow chart of a method for determining the position and posture of a camera system in one embodiment;

[0088] Figure 3 is a schematic flow chart of a method for determining a position and posture of a camera system in another embodiment;

[0089] Figure 4 is a schematic flow chart of a method for determining a position and posture of a camera system in another embodiment;

[0090] Figure 5 is a schematic flow chart of a method for determining a position and posture of a camera system in another embodiment;

[0091] Figure 6 is a schematic flow chart of a method for determining a position and posture of a camera system in another embodiment;

[0092] Figure 7 is a schematic flow chart of a method for determining a position and posture of a camera system in another embodiment;

[0093] Figure 8 is a schematic flow chart of a method for determining a position and posture of a camera system in another embodiment;

[0094] Fig. 9is a schematic flow chart of a method for determining a position and posture of a camera system in another embodiment;

[0095] Fig.10 is a schematic flow chart of a method for determining a position and posture of a camera system in another embodiment;

[0096] Fig.11 is a schematic flow chart of a method for determining a position and posture of a camera system in another embodiment;

[0097] Fig.12 is a schematic flow chart of a method for determining a position and posture of a camera system in another embodiment;

[0098] Fig.13 is a schematic flow chart of a method for determining a position and posture of a camera system in another embodiment;

[0099] Fig.14 is a schematic flow chart of a method for determining a position and posture of a camera system in another embodiment;

[0100] Fig.15 is a schematic flow chart of a method for determining a position and posture of a camera system in another embodiment;

[0101] Fig.16 is a flowchart of a pose estimation process for a camera system in one embodiment;

[0102] Fig.17 is a structural block diagram of a device for determining a position and posture of a camera system in one embodiment;

[0103] Fig.18 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0104] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0105] In the field of photography, camera systems (including multiple cameras) are required in some photography scenes. In practical applications, it is necessary to estimate the pose information of the camera system. In the related art, the images collected by each ordinary camera in the camera system are mainly processed by visual positioning to determine the pose information of the camera system. However, in the related art, due to the small viewing angle range of ordinary cameras, there will be problems such as missing feature points in adjacent frame images and feature points are easily lost, and there will be problems such as inaccurate pose information of the determined camera system. Based on this, an embodiment of the present application provides a method for determining the pose of a camera system, which can improve the accuracy of the pose information of the determined camera system.

[0106] The method for determining the position and posture of a camera system provided in an embodiment of the present application can be applied to a scene where each camera in the camera system moves rapidly, and can be specifically applied to the following scenarios: Figure 1 The movable platform shown in the figure may include a processor and a camera system, and the processor and the camera system may be communicatively connected. The movable platform may be an unmanned robot, an unmanned vehicle, an unmanned ship or other movable devices. In an embodiment of the present application, the movable platform may be a drone. The specific form of the movable platform is not limited in this embodiment. Optionally, the camera system may include at least one fisheye camera and at least one preset type camera. Optionally, a fisheye camera refers to a camera with a field of view angle close to or greater than 180 degrees. In contrast, a preset type camera may be understood as a non-fisheye camera, such as an ordinary camera with a field of view angle less than 90 degrees. Figure 1 The movable platform is a drone, and the camera system includes a fisheye camera and a preset type camera. The following embodiments will specifically introduce the specific process of the method for determining the position and posture of the camera system, and take the movable platform as an example to introduce the specific process of the method for determining the position and posture of the camera system.

[0107] like Figure 2 FIG. 1 is a flow chart of a method for determining the position and posture of a camera system provided in an embodiment of the present application. The method may include the following steps:

[0108] S100, obtaining a fisheye image of a current frame captured by a fisheye camera in a camera system and a normal image of a current frame captured by a preset type camera.

[0109] Among them, the camera system may include a fisheye camera and a preset type camera. The fisheye camera and the preset type camera are deployed at different locations. It should be noted here that the image captured by the fisheye camera can be called a fisheye image, and relatively, the image captured by the preset type camera can be called a normal image.

[0110] Optionally, the current frame fisheye image captured by the fisheye camera can be understood as the fisheye image captured by the fisheye camera at the current moment; the current frame ordinary image captured by the preset type camera can be understood as the ordinary image captured by the preset type camera at the current moment.

[0111] In practical applications, due to factors such as different interval distances between the fisheye camera and the preset type camera and the computer device, and different transmission speeds, the computer device can naturally use a synchronous method to receive the current frame of fisheye image sent by the fisheye camera and the current frame of ordinary image sent by the preset type camera, or use an asynchronous method to receive the current frame of fisheye image sent by the fisheye camera and the current frame of ordinary image sent by the preset type camera.

[0112] S200 , tracking feature points of the current frame fisheye image to obtain current frame fisheye feature points of the current frame fisheye image, and tracking feature points of the current frame normal image to obtain current frame normal feature points of the current frame normal image.

[0113] Specifically, the computer device can use a feature point tracking algorithm to track the feature points of the current frame fisheye image to obtain the current frame fisheye feature points of the current frame fisheye image. At the same time, use a feature point tracking algorithm to track the feature points of the current frame ordinary image to obtain the current frame ordinary feature points of the current frame ordinary image.

[0114] Optionally, the feature point tracking algorithm may be an optical flow tracking method, a target tracking method, a feature point matching method, etc., which is not limited in the embodiments of the present application.

[0115] In addition, the computer device can also pre-train a feature point tracking model, and then input the current frame fisheye image into the feature point tracking model, which tracks the feature points of the current frame fisheye image and outputs the current frame fisheye feature points of the current frame fisheye image; at the same time, the current frame ordinary image can also be input into the feature point tracking model, which tracks the feature points of the current frame ordinary image and outputs the current frame ordinary feature points of the current frame ordinary image.

[0116] Among them, the above-mentioned feature point tracking model can be a combination of at least one of a convolutional neural network model, a fully connected neural network model, a recurrent neural network model, a deep belief network model, a deep autoencoder and a generative adversarial network model, which is not limited to the embodiments of the present application.

[0117] S300, determining current position information of the camera system according to the fisheye feature points of the current frame and the common feature points of the current frame.

[0118] In practical applications, the computer device can perform fusion processing, rotation processing, and translation transformation processing on the fisheye feature points of the current frame and the ordinary feature points of the current frame to obtain the current posture information of the camera system. Optionally, the current posture information may include the rotation matrix and translation vector of the camera system. The rotation matrix at the i-th moment can be expressed as The translation vector at the i-th moment can be expressed as

[0119] It should be noted here that at time 0, the rotation matrix of the camera system can be the unit matrix, and the translation vector of the camera system can be the zero vector.

[0120] The technical solution in the embodiment of the present application is to obtain a current frame fisheye image captured by a fisheye camera in a camera system and a current frame ordinary image captured by a preset type camera, track feature points of the current frame fisheye image to obtain current frame fisheye feature points of the current frame fisheye image, and track feature points of the current frame ordinary image to obtain current frame ordinary feature points of the current frame ordinary image, and determine the current pose information of the camera system according to the current frame fisheye feature points and the current frame ordinary feature points; the above method can make up for the small viewing angle of the preset type camera by using a fisheye camera with a large viewing angle in the camera system, and make up for the defect of weak texture in the ordinary image captured by the preset type camera by using the fisheye image captured by the fisheye camera, so that more feature points from the surrounding environment can be obtained in the process of determining the pose of the camera system, and the pose of the camera system can be further determined by more feature points, which can improve the accuracy of the determined pose of the camera system; at the same time, because the fisheye camera has a unique imaging model, the movement of the same feature point in the inter-frame image is small, stable and not easy to lose, thereby improving the accuracy of feature point tracking, and on this basis, the accuracy of the determined pose of the camera system can be further improved.

[0121] The following is an explanation of the process of tracking the feature points of the current frame fisheye image to obtain the fisheye feature points of the current frame fisheye image. Figure 3 As shown, the step of tracking the feature points of the current frame fisheye image in S200 to obtain the fisheye feature points of the current frame fisheye image can be implemented in the following manner:

[0122] S210 , obtaining historical frame fisheye feature points of historical frame fisheye images captured by the fisheye camera.

[0123] Among them, the above-mentioned historical frame fisheye image can include any multiple frames of fisheye images before the current frame. For example, the historical frame fisheye image can include any two frames of fisheye images, any three frames of fisheye images, any four frames of fisheye images, etc. before the current frame.

[0124] In practical applications, the historical frame fisheye images collected by the fisheye camera can be stored in a local location, a cloud location, a disk or a hard disk, etc. Correspondingly, the computer device can obtain the historical frame fisheye images collected by the fisheye camera from a local location, a cloud location, a disk or a hard disk, etc., and then use a feature point extraction algorithm to extract feature points of the historical frame fisheye images to obtain feature points of the historical frame fisheye images, that is, the historical frame fisheye feature points of the historical frame fisheye images.

[0125] Optionally, the feature point extraction algorithm may be a grayscale-based corner detection algorithm, a fast corner detection algorithm, or a deep learning-based corner detection algorithm.

[0126] S220 , tracking the inter-frame feature points of the current frame fisheye image according to the historical frame fisheye feature points to obtain the current frame fisheye feature points of the current frame fisheye image.

[0127] In practical applications, the computer device may use a feature point tracking algorithm to track the feature points of the current frame fisheye image between frames based on the fisheye feature points of the historical frames, and obtain the current frame fisheye feature points of the current frame fisheye image.

[0128] In one embodiment, the camera system includes at least one fisheye camera; if the historical frame fisheye image includes a previous frame fisheye image of the current frame fisheye image, and the previous frame fisheye image is an initial frame fisheye image, then Figure 4 As shown, the step of obtaining the historical frame fisheye feature points of the historical frame fisheye image captured by the fisheye camera in the above S210 can be implemented in the following way:

[0129] S211, obtaining the previous frame of fisheye image captured by each fisheye camera.

[0130] In the embodiment of the present application, the historical frame fisheye image includes a previous frame fisheye image of the current frame fisheye image. Wherein, when the previous frame fisheye image is an initial frame fisheye image, the initial frame fisheye image can be understood as a fisheye image captured by the fisheye camera at the initial moment during the camera system shooting process.

[0131] In an embodiment of the present application, the camera system may include at least one fisheye camera. Specifically, the computer device may obtain the previous frame of fisheye image captured by each fisheye camera from a local location, a cloud, a disk, or a hard disk.

[0132] In practical applications, the sizes of fisheye images captured by different fisheye cameras may be equal or unequal, which is not limited in the embodiments of the present application.

[0133] S212, extracting a preset number of fisheye feature points from a target previous frame fisheye image in each previous frame fisheye image, wherein the target previous frame fisheye image refers to any one of each previous frame fisheye image.

[0134] The computer device may select any previous fisheye image from all previous fisheye images as the target previous fisheye image, and then use a feature point extraction algorithm to extract a preset number of feature points of fisheye features from the target previous fisheye image.

[0135] Optionally, the preset number of fisheye features may be user-defined or determined by historical experience values. However, in the embodiment of the present application, the preset number of fisheye features may be equal to the maximum number of feature points of the fisheye image.

[0136] S213, according to the feature points extracted from the previous frame fisheye image of the target, the feature points on other previous frame fisheye images in each previous frame fisheye image are extracted in sequence to obtain the previous frame fisheye feature points of the previous frame fisheye image captured by each fisheye camera.

[0137] Specifically, the computer device can pre-train an algorithm model, and then input the feature points extracted from the target previous frame fisheye image and each other previous frame fisheye image into the algorithm model. The algorithm model extracts the feature points on the other previous frame fisheye images in each previous frame fisheye image in turn, and then outputs the previous frame fisheye feature points of each other previous frame fisheye image.

[0138] In addition, the computer device can also use a feature point extraction algorithm to extract feature points from each other previous frame of fisheye image based on the feature points extracted from the previous frame of target fisheye image to obtain the previous frame of fisheye feature points of each other previous frame of fisheye image.

[0139] It should be noted here that the previous frame fisheye feature points of the previous frame fisheye image captured by each fisheye camera may include feature points extracted from the previous frame fisheye image of the target and feature points of each other previous frame fisheye image.

[0140] In an embodiment of the present application, if the previous frame of fisheye image is not the initial frame of fisheye image, the method for obtaining the previous frame of fisheye feature points of the previous frame of fisheye image can be to obtain the fisheye feature points of the previous frame of fisheye image corresponding to the fisheye image at the previous moment, and then track the inter-frame feature points of the previous frame of fisheye image based on the fisheye feature points of the previous frame of fisheye image corresponding to the fisheye image at the previous moment to obtain the previous frame of fisheye feature points of the previous frame of fisheye image.

[0141] It can be understood that, except for the initial frame fisheye image, the method for obtaining the fisheye feature points of each other frame fisheye image is to track the inter-frame feature points of the other frame fisheye image according to the fisheye feature points of the other frame fisheye image corresponding to the previous moment, so as to obtain the fisheye feature points of the other frame fisheye image.

[0142] In one embodiment, if Figure 5 As shown, the step of extracting the feature points of other previous fisheye images in each previous fisheye image in sequence according to the feature points extracted from the previous fisheye image of the target in S213 can be implemented in the following manner:

[0143] S2131, generating an image sequence from each previous frame of fisheye images; the previous frame of fisheye image in the image sequence is the starting fisheye image.

[0144] Specifically, the computer device can use the previous frame of fisheye image of the target as the starting fisheye image, and sort the previous frames of fisheye images according to information such as the deployment position of each fisheye camera, the resolution of the captured image or the production time, so as to generate an image sequence.

[0145] S2132 . For any previous fisheye image in the image sequence, track feature points from the previous fisheye image based on feature points extracted from all previous fisheye images before the previous fisheye image.

[0146] For example, the image sequence includes a previous fisheye image 1, a previous fisheye image 2, a previous fisheye image 3 and a previous fisheye image 4, wherein the previous fisheye image 1 is the target previous fisheye image. For the previous fisheye image 2, the feature points of the previous fisheye image 2 can be tracked according to the feature points extracted from the previous fisheye image 1, so as to track the feature points from the previous fisheye image 2; for the previous fisheye image 4, the feature points of the previous fisheye image 4 can be tracked according to the feature points extracted from the previous fisheye image 1, the previous fisheye image 2 and the previous fisheye image 3, so as to track the feature points from the previous fisheye image 4.

[0147] S2133: If the number of tracked feature points on the previous frame of fisheye image is equal to the preset number of fisheye features, the tracked feature points are used as the previous frame of fisheye feature points of the previous frame of fisheye image.

[0148] Meanwhile, after the step in S2132 is executed, Figure 6 As shown, the above method may also include:

[0149] S2134: If the number of feature points tracked on the previous frame of fisheye image is less than the preset number of fisheye features, determine the number of compensating feature points of the previous frame of fisheye image according to the number of feature points tracked on the previous frame of fisheye image and the preset number of fisheye features.

[0150] In an embodiment of the present application, for any previous frame of fisheye image in an image sequence, it can be determined whether the number of feature points tracked on the previous frame of fisheye image is less than a preset number of fisheye features. If so, feature point compensation extraction can be performed on the previous frame of fisheye image so that the number of fisheye feature points of the previous frame of the final previous frame of fisheye image is equal to the preset number of fisheye features.

[0151] Specifically, the computer device may obtain the number of compensated feature points of the previous frame of fisheye image by subtracting the number of feature points tracked on the previous frame of fisheye image from the preset number of fisheye features.

[0152] For example, if the number of feature points tracked on the previous fisheye image is a, the preset number of fisheye features is Th f, then the number of compensated feature points of the previous fisheye image can be equal to Th f -a.

[0153] S2135 . Perform feature point compensation extraction on the previous frame of fisheye image according to the number of compensated feature points of the previous frame of fisheye image to obtain the previous frame of fisheye feature points of the previous frame of fisheye image.

[0154] Furthermore, a feature point extraction algorithm may be used to extract feature points of the number of compensation feature points from the previous frame of fisheye image, so as to complete feature point compensation extraction of the previous frame of fisheye image and obtain the previous frame of fisheye feature points of the previous frame of fisheye image.

[0155] The technical solution in the embodiment of the present application can obtain the fisheye feature points of the previous frame of the fisheye image captured by the fisheye camera, and track the inter-frame feature points of the current frame of the fisheye image based on the fisheye feature points of the previous frame, so as to improve the accuracy of the current frame of the fisheye feature points of the current frame of the fisheye image finally obtained; at the same time, the method does not require complex algorithms to be implemented, and the processing process is relatively simple, thereby improving the speed of inter-frame feature point tracking.

[0156] The following is an explanation of the process of tracking the feature points of the current frame normal image to obtain the current frame normal feature points of the current frame normal image. Figure 7 As shown, the step of tracking the feature points of the current frame ordinary image in the above S200 to obtain the current frame ordinary feature points of the current frame ordinary image can be implemented in the following manner:

[0157] S230 , obtaining common feature points of common historical frames of common images captured by a preset type of camera.

[0158] The above-mentioned historical frame ordinary image may include any multiple frames of ordinary images before the current frame. For example, the historical frame ordinary image may include any two frames of ordinary images, any three frames of ordinary images, any four frames of ordinary images, etc. before the current frame.

[0159] In practical applications, the historical frame ordinary images collected by the preset type camera can be stored in a local, cloud, disk or hard disk location. Correspondingly, the computer device can obtain the historical frame ordinary images collected by the preset type camera from a local, cloud, disk or hard disk location, and then use the feature point extraction algorithm to extract feature points of the historical frame ordinary images to obtain feature points of the historical frame ordinary images, that is, the historical frame ordinary feature points of the historical frame ordinary images.

[0160] S240 , tracking the inter-frame feature points of the current frame ordinary image according to the historical frame ordinary feature points of the historical frame ordinary image to obtain candidate ordinary feature points of the current frame ordinary image.

[0161] In practical applications, the computer device may use a feature point tracking algorithm to track the inter-frame feature points of the current frame ordinary image based on the common feature points of the historical frames, and obtain the current frame ordinary feature points of the current frame ordinary image.

[0162] S250: If the number of candidate common feature points of the current frame common image is equal to the preset number of common features, the candidate common feature points of the current frame common image are determined as the current frame common feature points of the current frame common image.

[0163] Among them, the computer device can determine whether the number of candidate ordinary feature points of the current frame ordinary image is equal to the preset number of ordinary features. If so, the candidate ordinary feature points of the current frame ordinary image are determined as the current frame ordinary feature points of the current frame ordinary image.

[0164] Optionally, the above-mentioned preset number of common features can be user-defined or determined by historical experience values, but in the embodiment of the present application, the preset number of common features can be equal to the maximum number of feature points of a common image.

[0165] At the same time, after the step in S240 is executed, the method may further include: if the number of candidate ordinary feature points of the current frame ordinary image is less than the preset number of ordinary features, tracking the feature points of the current frame ordinary image according to the historical frame ordinary feature points of the historical frame ordinary images captured by the preset type of camera, and determining the current frame ordinary feature points of the current frame ordinary image.

[0166] In an embodiment of the present application, for a current frame ordinary image, if it is determined that the number of candidate ordinary feature points of the current frame ordinary image is less than a preset number of ordinary features, the historical frame ordinary feature points of the historical frame ordinary image captured by a preset type of camera can be obtained, and then a feature point tracking algorithm is used to perform feature point tracking on the current frame ordinary image based on the historical frame ordinary feature points of the historical frame ordinary image to obtain the current frame ordinary feature points of the current frame ordinary image.

[0167] The historical frame common feature points of the historical frame common image may be obtained by extracting feature points from a previous frame common image using a feature point extraction algorithm.

[0168] In one embodiment, the camera system includes at least one fisheye camera; if the historical frame fisheye image includes a previous frame fisheye image of the current frame fisheye image, the historical frame normal image includes a previous frame normal image of the current frame normal image, and the previous frame fisheye image is the initial frame fisheye image, and the previous frame normal image is the initial frame normal image, then Figure 8 As shown, the step of obtaining the common feature points of the common historical frame images captured by the preset type of camera in the above S230 can be implemented in the following way:

[0169] S231, obtaining the previous frame of fisheye feature points of the previous frame of fisheye image captured by each fisheye camera.

[0170] In an embodiment of the present application, the camera system may include at least one fisheye camera. Specifically, the computer device may obtain the previous frame of fisheye image captured by each fisheye camera from a local location, a cloud, a disk, or a hard disk.

[0171] In practical applications, the sizes of fisheye images captured by different fisheye cameras may be equal or unequal, which is not limited in the embodiments of the present application.

[0172] The computer device may use a feature point extraction algorithm to extract feature points from each previous frame of fisheye image to obtain the previous frame of fisheye feature points of each previous frame of fisheye image.

[0173] S232 . Track feature points of the previous frame of ordinary image according to the fisheye feature points of the previous frame to obtain candidate ordinary feature points of the previous frame of ordinary image.

[0174] When the previous frame of ordinary image is an initial frame of ordinary image, the initial frame of ordinary image can be understood as an ordinary image captured by a preset type of camera at an initial moment during the camera system shooting process.

[0175] Specifically, based on the previous frame fisheye feature points of each previous frame fisheye image obtained in the previous steps, a feature point tracking algorithm can be used to track the feature points of the previous frame ordinary image according to the previous frame fisheye feature points to obtain candidate ordinary feature points of the previous frame ordinary image.

[0176] S233: If the number of candidate common feature points of the previous frame of common image is equal to the preset number of common features, the candidate common feature points of the previous frame of common image are determined as the previous frame of common feature points of the previous frame of common image.

[0177] In an embodiment of the present application, if the previous frame of ordinary image is not the initial frame of ordinary image, the method for obtaining the previous frame of ordinary feature points of the previous frame of ordinary image may be to obtain the ordinary feature points of the ordinary image at the previous moment corresponding to the previous frame of ordinary image, and based on the ordinary feature points of the ordinary image at the previous moment corresponding to the previous frame of ordinary image, perform inter-frame feature point tracking on the previous frame of ordinary image to obtain candidate ordinary feature points of the previous frame of ordinary image, and if the number of candidate ordinary feature points of the previous frame of ordinary image is equal to the preset number of ordinary features, the candidate ordinary feature points of the previous frame of ordinary image are determined as the previous frame of ordinary feature points of the previous frame of ordinary image.

[0178] It can be understood that, except for the initial frame ordinary image, the method for obtaining the ordinary feature points of each other frame ordinary image is to track the inter-frame feature points of the other frame ordinary images according to the ordinary feature points of the other frame ordinary images corresponding to the previous moment, and obtain the candidate ordinary feature points of the other frame ordinary images. If the number of candidate ordinary feature points of the other frame ordinary images is equal to the preset number of ordinary features, the candidate ordinary feature points of the other frame ordinary images are determined as the ordinary feature points of the other frame ordinary images.

[0179] Meanwhile, after the step in S232 is executed, Fig. 9 As shown, the above method may also include:

[0180] S234: If the number of candidate common feature points of the previous frame of common image is less than the preset number of common features, determine the number of compensating feature points of the previous frame of common image according to the total number of candidate common feature points of the previous frame of common image and the preset number of common features.

[0181] In an embodiment of the present application, when it is determined that the number of candidate ordinary feature points of the previous frame of ordinary image is less than the preset number of ordinary features, the feature points of the previous frame of ordinary image can be compensated so that the number of ordinary feature points of the previous frame of ordinary image is equal to the preset number of ordinary features.

[0182] Specifically, the computer device may obtain the number of compensated feature points of the previous frame of ordinary image by subtracting the total number of candidate ordinary feature points of the previous frame of ordinary image from the preset number of ordinary features.

[0183] For example, if the number of feature points extracted from the previous frame of the normal image is b, the preset number of normal features is Th p , then the number of compensated feature points of the previous frame of ordinary image can be equal to Th p -b.

[0184] S235 . Perform feature point compensation extraction on the previous frame of ordinary image according to the number of compensated feature points of the previous frame of ordinary image.

[0185] Furthermore, a feature point extraction algorithm may be used to perform feature point compensation extraction on the previous frame of ordinary image, and feature points equal to the number of compensated feature points may be extracted from the previous frame of ordinary image.

[0186] S236 , determining the previous frame common feature points of the previous frame common image according to the feature point compensation extraction result of the previous frame common image and the candidate common feature points.

[0187] In practical applications, the feature points of the compensated feature points extracted from the previous frame of ordinary image (ie, feature point compensation extraction results) and the candidate ordinary feature points can be determined as the previous frame of ordinary feature points of the previous frame of ordinary image.

[0188] The technical solution in the embodiment of the present application can obtain the previous frame ordinary feature points of the previous frame ordinary image captured by a preset type of camera, and track the inter-frame feature points of the current frame ordinary image based on the previous frame ordinary feature points of the previous frame ordinary image to obtain candidate ordinary feature points of the current frame ordinary image. When the number of candidate ordinary feature points of the current frame ordinary image is equal to the preset number of ordinary features, the candidate ordinary feature points of the current frame ordinary image are determined as the current frame ordinary feature points of the current frame ordinary image, so that the current frame fisheye feature points of the current frame fisheye image finally obtained are more accurate. At the same time, the method does not require complex algorithms to be implemented, and the processing process is relatively simple, thereby improving the speed of inter-frame feature point tracking.

[0189] The following is an explanation of the process of determining the current pose information of the camera system based on the fisheye feature points of the current frame and the common feature points of the current frame. Fig.10 As shown, the steps in the above S300 can be implemented in the following manner:

[0190] S310, obtaining fisheye feature points of a previous frame of fisheye image captured by a fisheye camera, and common feature points of a previous frame of common image captured by a preset type camera.

[0191] The computer device may obtain fisheye feature points of a previous frame of a fisheye image captured by a predetermined fisheye camera, and common feature points of a previous frame of a common image captured by a preset type of camera.

[0192] In addition, the computer device can also obtain the previous frame of fisheye image captured by the fisheye camera and the previous frame of ordinary image captured by a preset type camera from local locations, cloud locations, disks or hard disks, and then use a feature point extraction algorithm to extract feature points from the previous frame of fisheye image to obtain the previous frame of fisheye feature points of the previous frame of fisheye image; at the same time, a feature point extraction algorithm can also be used to extract feature points from the previous frame of ordinary image to obtain the previous frame of ordinary feature points of the previous frame of ordinary image.

[0193] S320, constructing a current posture observation model of the camera system according to the fisheye feature points of the current frame, the fisheye feature points of the previous frame, the common feature points of the current frame and the common feature points of the previous frame.

[0194] Specifically, the computer device can use a pose estimation algorithm to perform pose estimation processing on the fisheye feature points of the current frame, the fisheye feature points of the previous frame, the ordinary feature points of the current frame and the ordinary feature points of the previous frame to obtain the current pose observation model of the camera system.

[0195] Optionally, the above-mentioned pose estimation algorithm can be a model-based pose estimation algorithm, a feature-based pose estimation algorithm, or a deep learning-based camera pose estimation algorithm, etc., which is not limited to this embodiment of the present application.

[0196] In one embodiment, the current pose observation model includes a depth information observation model and a non-depth information observation model; the step of constructing the current pose observation model of the camera system according to the fisheye feature points of the current frame, the fisheye feature points of the previous frame, the ordinary feature points of the current frame and the ordinary feature points of the previous frame in the above S320 can be implemented in the following way: determining the depth information observation model according to the fisheye feature points of the current frame and the ordinary feature points of the current frame; and constructing the non-depth information observation model according to the fisheye feature points of the current frame, the fisheye feature points of the previous frame, the ordinary feature points of the current frame and the ordinary feature points of the previous frame.

[0197] In the embodiment of the present application, the computer device may obtain some or all feature points from the fisheye feature points of the current frame and the common feature points of the current frame, and then process the obtained feature points to obtain the depth information observation model. Alternatively, the computer device may directly process the fisheye feature points of the current frame and the common feature points of the current frame according to the construction strategy of the depth information observation model to obtain the depth information observation model.

[0198] At the same time, the computer device can obtain some or all feature points from the fisheye feature points of the current frame, the fisheye feature points of the previous frame, the common feature points of the current frame, and the common feature points of the previous frame, and then process the obtained feature points to obtain a non-depth information observation model. Alternatively, the computer device can also directly process the fisheye feature points of the current frame, the fisheye feature points of the previous frame, the common feature points of the current frame, and the common feature points of the previous frame according to the construction strategy of the non-depth information observation model to obtain a non-depth information observation model.

[0199] S330. Construct a current posture optimization function of the camera system according to the current posture observation model.

[0200] In the embodiment of the present application, there may be multiple current posture observation models. Based on the current posture observation model obtained in the previous steps, an arithmetic operation can be performed on the current posture observation model to obtain a current posture optimization function of the camera system.

[0201] Optionally, the arithmetic operation may be at least one of addition, subtraction, multiplication, division, logarithmic operation and exponential operation.

[0202] S340, determining current posture information of the camera system based on the current posture optimization function.

[0203] In practical applications, the computer device can use an optimization algorithm to solve the current posture optimization function to obtain the current posture information of the camera system. Optionally, the above-mentioned optimization algorithm can be a gradient descent method, a Newton method, a conjugate gradient method and / or a genetic algorithm, etc. In the embodiment of the present application, the above-mentioned optimization algorithm can be a least squares method, and the least squares method can be a Levenberg-Marquardt method or a Gauss-Newton iteration method, etc.

[0204] The technical solution in the embodiment of the present application obtains the fisheye feature points of the previous frame of the fisheye image captured by the fisheye camera, and the ordinary feature points of the previous frame of the ordinary image captured by the preset type camera, and constructs a current pose observation model of the camera system according to the fisheye feature points of the current frame, the fisheye feature points of the previous frame, the ordinary feature points of the current frame and the ordinary feature points of the previous frame, and constructs a current pose optimization function of the camera system according to the current pose observation model, and determines the current pose information of the camera system based on the current pose optimization function; the method can construct the current pose observation model of the camera system through the feature points of adjacent frame images, so that the constructed current pose observation model of the camera system is more accurate, and on this basis, the current pose information of the camera system can also be accurately determined.

[0205] In one embodiment, if Fig.11 As shown, the above step of determining the depth information observation model according to the fisheye feature points of the current frame and the common feature points of the current frame can be implemented in the following way:

[0206] S321, obtaining depth feature points having depth information from among the fisheye feature points of the current frame and the common feature points of the current frame; the depth feature points include the three-dimensional position coordinates of the feature points.

[0207] In practical applications, the computer device may filter feature points having depth information, ie, depth feature points, from the fisheye feature points of the current frame and the common feature points of the current frame.

[0208] It should be noted here that the fisheye feature points of the current frame may be feature points with depth information, and the ordinary feature points of the current frame may also be feature points with depth information.

[0209] S322 . Determine the unit sphere coordinates of each depth feature point according to the current position coordinates of each depth feature point in the current frame image.

[0210] Optionally, the current frame image to which the depth feature point belongs may be a current frame fisheye image or a current frame normal image. Optionally, the current position coordinates of the depth feature point may be understood as the position coordinates corresponding to the depth feature point in the current frame image.

[0211] Among them, for any depth feature point, the current position coordinates of the depth feature point in the current frame image can be obtained based on the result of feature point extraction on the current frame image, and then the current position coordinates of the depth feature point in the current frame image can be converted to obtain the unit sphere coordinates of the depth feature point.

[0212] S323, determining a depth residual function of each depth feature point according to the current position coordinates of each depth feature point in the world coordinate system and the unit sphere coordinates of each depth feature point, as well as an external parameter between the current camera system coordinate system and the camera coordinate system corresponding to the current frame image to which each depth feature point belongs.

[0213] Among them, the camera coordinate system corresponding to the current frame image to which the above-mentioned depth feature points belong can be understood as the coordinate system corresponding to the camera that collects the current frame image to which the depth feature points belong. It should be noted here that if the current frame image is a current frame fisheye image, the camera of the current frame image can be a fisheye camera that collects the current frame image; if the current frame image is a current frame ordinary image, the camera of the current frame image can be a preset type camera that collects the current frame image.

[0214] Optionally, the above-mentioned camera system coordinate system refers to a coordinate system created based on all cameras in the camera system. In the embodiment of the present application, the camera system coordinate system may be the same or different at different times, and the embodiment of the present application is not limited to this.

[0215] Specifically, the computer device can perform arithmetic operations on the current position coordinates of each depth feature point in the world coordinate system and the unit sphere coordinates of each depth feature point, as well as the external parameters between the current camera system coordinate system and the camera coordinate system corresponding to the current frame image to which each depth feature point belongs, to obtain the depth residual function of each depth feature point.

[0216] For the convenience of calculation, the camera system coordinate system corresponding to the initial frame can be used as the world coordinate system. At the same time, the above external parameters can include a rotation matrix and a translation vector.

[0217] In the embodiment of the present application, if the camera system includes m (m is greater than or equal to 1) fisheye cameras, the m fisheye cameras C f Can be expressed as C f1 , C f2 ,…,C fk ,…,C fm , k is greater than or equal to 1 and less than or equal to m. Correspondingly, the current frame fisheye image collected by m fisheye cameras can be expressed as The previous fisheye image captured by m fisheye cameras can be expressed as Meanwhile, the camera system includes n (n is greater than or equal to 1) preset type cameras, n preset type cameras Cp Can be expressed as C p1 , C p2 ,…,C po ,…,C pm , o is greater than or equal to 1 and less than or equal to n. Correspondingly, the current frame normal image collected by n preset type cameras can be expressed as The previous frame of ordinary image collected by n preset type cameras can be expressed as And each depth feature point is represented as P 11 , P 12 , ..., P 1a In the embodiment of the present application, for the depth feature point P 1a , the depth feature point P can be constructed by the following formula (1) 1a The depth residual function

[0218]

[0219] In formula (1), Represents the depth feature point P 1a The current position coordinates in the world coordinate system, Represents the depth feature point P 1a The unit spherical coordinates of and Represent the current camera system coordinate system and depth feature point P respectively 1a The external parameters between the camera coordinate systems corresponding to the current frame image (i.e., rotation matrix and translation vector).

[0220] S324: Determine a depth information observation model according to the depth residual function of each depth feature point.

[0221] Furthermore, the depth residual function of each depth feature point can be subjected to arithmetic operations to obtain a depth information observation model. In an embodiment of the present application, the second norm of the depth residual function of each depth feature point can be calculated, and then the second norm of the depth residual function of each depth feature point can be squared, and then the squares of the second norms of the depth residual function of each depth feature point can be summed to obtain the depth information observation model.

[0222] In the embodiment of the present application, if the depth information observation model represents F1, then F1 can be expressed by the following formula (2).

[0223]

[0224] At the same time, in one embodiment, Fig.12As shown, the above step of constructing a non-depth information observation model based on the fisheye feature points of the current frame, the fisheye feature points of the previous frame, the common feature points of the current frame and the common feature points of the previous frame can be implemented in the following way:

[0225] S325 . Obtain a plurality of first non-depth feature points, a plurality of second non-depth feature points, and a plurality of third non-depth feature points from the fisheye feature points of the current frame and the common feature points of the current frame.

[0226] In the embodiment of the present application, the non-depth feature point screening conditions for screening the first non-depth feature point, the second non-depth feature point, and the third non-depth feature point are different.

[0227] In practical applications, the computer device may use different non-depth feature point screening conditions to screen multiple first non-depth feature points, multiple second non-depth feature points, and multiple third non-depth feature points from the current frame fisheye feature points and the current frame ordinary feature points.

[0228] S326 , constructing a non-depth residual function of each first non-depth feature point, a non-depth residual function of each second non-depth feature point, and a non-depth residual function of each third non-depth feature point.

[0229] Specifically, for any first non-depth feature point, the computer device may perform arithmetic operation and / or coordinate conversion processing on the current position coordinates of the first non-depth feature point to obtain the non-depth residual function of the first non-depth feature point.

[0230] Meanwhile, for any second non-depth feature point, the computer device may perform arithmetic operation and / or coordinate conversion processing on the current position coordinates of the second non-depth feature point to obtain the non-depth residual function of the second non-depth feature point.

[0231] In addition, for any third non-depth feature point, the computer device may perform arithmetic operation and / or coordinate conversion processing on the current position coordinates of the third non-depth feature point to obtain the non-depth residual function of the third non-depth feature point.

[0232] In the embodiment of the present application, the construction methods of the non-depth residual function of each first non-depth feature point, the non-depth residual function of each second non-depth feature point, and the non-depth residual function of each third non-depth feature point may be the same or different.

[0233] S327 . Determine a non-depth information observation model according to the non-depth residual function of each first non-depth feature point, the non-depth residual function of each second non-depth feature point, and the non-depth residual function of each third non-depth feature point.

[0234] The computer device may perform arithmetic operations on the non-depth residual function of each first non-depth feature point to obtain a corresponding non-depth information observation model. In an embodiment of the present application, the second norm of the non-depth residual function of each first non-depth feature point may be calculated, and then the second norm of the non-depth residual function of each first non-depth feature point may be squared, and then the squares of the second norms of the non-depth residual function of each first non-depth feature point may be summed to obtain the corresponding non-depth information observation model.

[0235] At the same time, the computer device can perform arithmetic operations on the non-depth residual function of each second non-depth feature point to obtain a corresponding non-depth information observation model. In an embodiment of the present application, the second norm of the non-depth residual function of each second non-depth feature point can be calculated, and then the second norm of the non-depth residual function of each second non-depth feature point can be squared, and then the squares of the second norms of the non-depth residual function of each second non-depth feature point can be summed to obtain the corresponding non-depth information observation model.

[0236] In addition, the computer device can perform arithmetic operations on the non-depth residual function of each third non-depth feature point to obtain a corresponding non-depth information observation model. In an embodiment of the present application, the second norm of the non-depth residual function of each third non-depth feature point can be calculated, and then the second norm of the non-depth residual function of each third non-depth feature point can be squared, and then the squares of the second norms of the non-depth residual function of each third non-depth feature point can be summed to obtain the corresponding non-depth information observation model.

[0237] In an embodiment of the present application, each first non-depth feature point represents a current frame feature point without depth information that is in at least two current frame images and in at least two previous frame images; each second non-depth feature point represents a current frame feature point without depth information that is only in one current frame image and in at least two previous frame images; each third non-depth feature point represents a current frame feature point without depth information that is in at least two current frame images and in only one previous frame image; wherein the current frame image is either a current frame fisheye image or a current frame ordinary image; and the previous frame image is either a previous frame fisheye image or a previous frame ordinary image.

[0238] It should be noted that the camera system may include multiple fisheye cameras and multiple preset type cameras. Correspondingly, the computer device may obtain multiple current frame fisheye images and multiple current frame normal images, and then obtain the current frame fisheye feature points of each current frame fisheye image and the current frame normal feature points of each current frame normal image, and then first filter out all current frame feature points without depth information from all current frame fisheye feature points and all current frame normal feature points.

[0239] Furthermore, based on all the acquired current frame feature points without depth information, for each current frame feature point without depth information, it can be determined whether the current frame feature point without depth information is in at least two current frame images and in at least two previous frame images. If so, the current frame feature point without depth information is determined as the first non-depth feature point.

[0240] For each current frame feature point without depth information, it is also possible to determine whether the current frame feature point without depth information is only in one current frame image and in at least two previous frame images. If so, the current frame feature point without depth information is determined as the second non-depth feature point.

[0241] For each current frame feature point without depth information, it is also possible to determine whether the current frame feature point without depth information is in at least two current frame images and only in one previous frame image. If so, the current frame feature point without depth information is determined as the third non-depth feature point.

[0242] The embodiment of the present application can obtain non-depth feature points with non-depth information, and then determine the corresponding non-depth information observation model according to each non-depth feature point, and use the non-depth information observation model as basic information to provide a reference basis for accurately determining the current posture information of the camera system.

[0243] The technical solution in the embodiments of the present application obtains depth feature points with depth information from the fisheye feature points of the current frame and the ordinary feature points of the current frame, determines the unit spherical coordinates of each depth feature point according to the current position coordinates of each depth feature point in the current frame image to which it belongs, determines the depth residual function of each depth feature point according to the current position coordinates of each depth feature point in the world coordinate system and the unit spherical coordinates of each depth feature point, and the external parameters between the current camera system coordinate system and the camera coordinate system corresponding to the current frame image to which each depth feature point belongs, and determines the depth information observation model according to the depth residual function of each depth feature point; the method can obtain depth feature points with depth information, and then determine the corresponding depth information observation model according to each depth feature point, and use the depth information observation model as basic information to provide a reference basis for accurately determining the current posture information of the camera system.

[0244] The process of constructing the non-depth residual function of each first non-depth feature point is described below. Fig.13 As shown, the step of constructing the non-depth residual function of each first non-depth feature point in the above S326 can be implemented in the following way:

[0245] S326a, obtaining the front position coordinates of each first non-depth feature point on the corresponding previous frame image.

[0246] Specifically, the front position coordinates of each first non-depth feature point on the corresponding previous frame image may be obtained according to the result of feature point extraction of each first non-depth feature point on the corresponding previous frame image.

[0247] It should be noted here that the previous position coordinates of the first non-depth feature point can be understood as the position coordinates of the first non-depth feature point on the previous frame image.

[0248] S326b. Triangulate the front position coordinates of each first non-depth feature point to obtain the three-dimensional front position coordinates of each first non-depth feature point in the world coordinate system; and triangulate the current position coordinates of each first non-depth feature point to obtain the three-dimensional current position coordinates of each first non-depth feature point in the current camera system coordinate system.

[0249] In practical applications, the depth information of feature points can be obtained by triangulating the position coordinates of feature points.

[0250] In an embodiment of the present application, for any first non-depth feature point, the computer device can triangulate the front position coordinates of the first non-depth feature point to obtain the three-dimensional front position coordinates of the first non-depth feature point in the world coordinate system; at the same time, the current position coordinates of the first non-depth feature point can also be triangulated to obtain the three-dimensional current position coordinates of the first non-depth feature point in the current camera system coordinate system.

[0251] It should be noted here that the first non-depth feature point can be in at least one current frame image, and naturally, the number of current position coordinates of the first non-depth feature point can be at least one; the first non-depth feature point can also be in at least one previous frame image, and naturally, the number of previous position coordinates of the first non-depth feature point can be at least one.

[0252] S326c, constructing a non-depth residual function of each first non-depth feature point according to the three-dimensional front position coordinates of each first non-depth feature point in the world coordinate system and the three-dimensional current position coordinates of each first non-depth feature point in the current camera system coordinate system.

[0253] Among them, for any first non-depth feature point, arithmetic operation can be performed on the three-dimensional front position coordinates of the first non-depth feature point in the world coordinate system and the three-dimensional current position coordinates of the first non-depth feature point in the current camera system coordinate system to obtain the non-depth residual function of the first non-depth feature point.

[0254] In the embodiment of the present application, referring to the previous example, if each first non-depth feature point is represented by P 21 , P 22 , ..., P 2b, for the first non-depth feature point P 2b The first non-depth feature point P can be constructed by the following formula (3): 2b The depth residual function

[0255]

[0256] Among them, in formula (3) Represents the first non-depth feature point P 2b The three-dimensional front position coordinates in the world coordinate system, Represents the first non-depth feature point P 2b The 3D coordinates of the current position in the current camera system coordinate system.

[0257] Further, according to the first non-depth feature point P 2b The depth residual function The determined non-depth information observation model can be expressed by the following formula (4).

[0258]

[0259] Meanwhile, in another embodiment, if Fig.14 As shown, the step of constructing the non-depth residual function of each second non-depth feature point in S326 can be implemented in the following way:

[0260] S326d, obtaining the front position coordinates of each second non-depth feature point on the corresponding previous frame image.

[0261] Specifically, the front position coordinates of each second non-depth feature point on the corresponding previous frame image may be obtained according to the result of feature point extraction on each previous frame image.

[0262] It should be noted here that the previous position coordinates of the second non-depth feature point can be understood as the position coordinates of the second non-depth feature point on the previous frame image.

[0263] S326e: Triangulate the front position coordinates of each second non-depth feature point to obtain the three-dimensional front position coordinates of each second non-depth feature point in the world coordinate system.

[0264] In the embodiment of the present application, for any second non-depth feature point, the computer device can triangulate the front position coordinates of the second non-depth feature point to obtain the three-dimensional front position coordinates of the second non-depth feature point in the world coordinate system.

[0265] It should be noted here that the second non-depth feature point may be in at least one previous frame image, and naturally, the number of previous position coordinates of the second non-depth feature point may be at least one.

[0266] S326f, determining the unit sphere coordinates of each second non-depth feature point according to the current position coordinates of each second non-depth feature point in the current frame image to which it belongs.

[0267] Optionally, the current frame image to which the second non-depth feature point belongs may be a current frame fisheye image or a current frame normal image. Optionally, the current position coordinates of the second non-depth feature point may be understood as the position coordinates corresponding to the second non-depth feature point in the current frame image.

[0268] Among them, for any second non-depth feature point, the current position coordinates of the second non-depth feature point in the current frame image can be obtained based on the result of feature point extraction on the current frame image, and then the current position coordinates of the second non-depth feature point in the current frame image can be converted to obtain the unit sphere coordinates of the second non-depth feature point.

[0269] S326g. Construct a non-depth residual function of each second non-depth feature point based on the three-dimensional front position coordinates of each second non-depth feature point in the world coordinate system, the unit sphere coordinates of each second non-depth feature point, and the external parameters between the current camera system coordinate system and the camera coordinate system corresponding to the current frame image to which each second non-depth feature point belongs.

[0270] Specifically, for any second non-depth feature point, arithmetic operations can be performed on the three-dimensional front position coordinates of the second non-depth feature point in the world coordinate system, the unit sphere coordinates of the second non-depth feature point, and the external parameters between the current camera system coordinate system and the camera coordinate system corresponding to the current frame image to which the second non-depth feature point belongs to obtain the non-depth residual function of the second non-depth feature point.

[0271] In the embodiment of the present application, referring to the previous example, if each second non-depth feature point is represented by P 31 , P 32 , ..., P 3c , for the second non-depth feature point P 3c The second non-depth feature point P can be constructed by the following formula (5): 3c The depth residual function

[0272]

[0273] Among them, in formula (5) Represents the second non-depth feature point P 3c The three-dimensional front position coordinates in the world coordinate system, Represents the second non-depth feature point P 3c The unit spherical coordinates of and Respectively represent the current camera system coordinate system and the second non-depth feature point P 3c The external parameters between the camera coordinate systems corresponding to the current frame image.

[0274] Further, according to the second non-depth feature point P 3c The depth residual function The determined non-depth information observation model can be expressed by the following formula (6).

[0275]

[0276] In addition, in one embodiment, Fig.15 As shown, the step of constructing the non-depth residual function of each third non-depth feature point in the above S326 can be implemented in the following way:

[0277] S326h: triangulate the current position coordinates of each third non-depth feature point on the corresponding current frame image to obtain the three-dimensional current position coordinates of each third non-depth feature point in the current camera coordinate system.

[0278] Optionally, the current position coordinates of the third non-depth feature point may be understood as the position coordinates of the third non-depth feature point on the current frame image.

[0279] In actual applications, the computer device can obtain the current position coordinates of each third non-depth feature point on the current frame image to which it belongs, and then for any third non-depth feature point, the current position coordinates of the third non-depth feature point on the current frame image to which it belongs can be triangulated to obtain the three-dimensional current position coordinates of the third non-depth feature point in the current camera coordinate system.

[0280] S326i. Determine the unit sphere coordinates of each third non-depth feature point according to the front position coordinates of each third non-depth feature point in the corresponding previous frame image.

[0281] Optionally, the previous frame image to which the third non-depth feature point belongs can be a previous frame fisheye image or a previous frame normal image. Optionally, the previous position coordinates of the third non-depth feature point can be understood as the position coordinates corresponding to the third non-depth feature point in the previous frame image.

[0282] Among them, for any third non-depth feature point, the front position coordinates of the third non-depth feature point in the previous frame image can be obtained based on the result of feature point extraction on the previous frame image, and then the front position coordinates of the third non-depth feature point in the previous frame image can be converted to obtain the unit sphere coordinates of the third non-depth feature point.

[0283] S326j. Construct a non-depth residual function of each third non-depth feature point based on the three-dimensional current position coordinates of each third non-depth feature point in the current camera coordinate system, the unit sphere coordinates of each third non-depth feature point, the transformation relationship between the world coordinate system and the front camera system coordinate system, and the external parameters between the current camera system coordinate system and the camera coordinate system corresponding to the previous frame image to which each third non-depth feature point belongs.

[0284] Specifically, for any third non-depth feature point, arithmetic operations can be performed on the three-dimensional current position coordinates of the third non-depth feature point in the current camera coordinate system, the unit sphere coordinates of the third non-depth feature point, the transformation relationship between the world coordinate system and the front camera system coordinate system, and the external parameters between the current camera system coordinate system and the camera coordinate system corresponding to the previous frame image to which the third non-depth feature point belongs, to obtain the non-depth residual function of the third non-depth feature point.

[0285] In the embodiment of the present application, referring to the previous example, if each third non-depth feature point is represented by P 41 , P 42 , ..., P 4d , for the third non-depth feature point P 4d The third non-depth feature point P can be constructed by the following formula (7): 4d The depth residual function

[0286]

[0287] Among them, in formula (7) Represents the third non-depth feature point P 4d The three-dimensional current position coordinates in the current camera coordinate system, Represents the third non-depth feature point P 4d The unit spherical coordinates of and Represents the transformation relationship between the world coordinate system and the front camera system coordinate system, and Respectively represent the current camera system coordinate system and the third non-depth feature point P 4d The external parameters between the camera coordinate system corresponding to the previous frame image.

[0288] Further, according to the third non-depth feature point P 4d The depth residual function The determined non-depth information observation model can be expressed by the following formula (8).

[0289]

[0290] It should be noted here that, in the embodiment of the present application, based on the depth information observation model F1 corresponding to the constructed depth feature points, the non-depth information observation model F2 corresponding to the first non-depth feature points, the non-depth information observation model F3 corresponding to the second non-depth feature points and the non-depth information observation model F4 corresponding to the third non-depth feature points, the current posture optimization function of the constructed camera system can be expressed by formula (9).

[0291]

[0292] In addition, it should be noted that the computer device can obtain the front pose information of the camera system, and triangulate all the current frame feature points (the current frame fisheye feature points and the current frame ordinary feature points) according to the current pose information of the camera system and the front pose information of the camera system, to obtain the three-dimensional position coordinates of each current frame feature point in the world coordinate system, in preparation for determining the rear pose information of the camera system at the next moment.

[0293] At the same time, feature point post-processing (i.e., feature point supplementation) can be performed on the current frame normal image and the current frame fisheye image. Specifically, the normal feature points of the current frame normal image are supplemented, and the fisheye feature points of the current frame fisheye image are supplemented to prepare for determining the post-pose information of the camera system at the next moment.

[0294] Among them, for the current frame fisheye image The number of fisheye feature points that need to be supplemented is A fk , the number of fisheye feature points currently tracked in the current frame is T fk Among them, if T fk Less than Th f When A fk Equal to Th f -T fk If T fk Equal to Th f When A fk is equal to 0.

[0295] Specifically, the fisheye images of each current frame can be sorted. For the fisheye image of the current frame with sorting 1, The feature point extraction algorithm can be used to extract the fisheye image from the current frame Extract A f1 feature points as the fisheye image of the current frame Supplementary fisheye feature points. For the current frame fisheye image of sort 2 Based on the current frame fisheye image Supplementary fisheye feature points for the current frame fisheye image Perform feature point tracking. If the current frame fisheye image The number of newly tracked fisheye feature points (i.e., supplementary fisheye feature points) is equal to A fk , then the current frame fisheye image is finished Tracking, if the current frame fisheye image The number of newly tracked fisheye feature points is less than A fk , then from the current frame fisheye image Extract A fk -r f2 feature points, among which r f2 Indicates the fisheye image of the current frame The number of newly tracked fisheye feature points.

[0296] If the current fisheye image of sort 4 Based on the current frame fisheye image Current frame fisheye image and the current frame fisheye image Supplementary fisheye feature points for the current frame fisheye image Perform feature point tracking. If the current frame fisheye image The number of newly tracked fisheye feature points (i.e., supplementary fisheye feature points) is equal to A fk , then the current frame fisheye image is finished Tracking, if the current frame fisheye image The number of newly tracked fisheye feature points is less than A fk , then from the current frame fisheye image Extract A fk -r f4 feature points, among which r f4 Indicates the fisheye image of the current frame The number of newly tracked fisheye feature points. By analogy, when supplementing feature points for any current frame fisheye image that is ranked later, feature points of the current frame fisheye image can be tracked based on the supplemented fisheye feature points of all current frame fisheye images before the current frame fisheye image, until the number of supplemented fisheye feature points is equal to A. fk until.

[0297] In addition, for the current frame normal image The number of common feature points that need to be supplemented is A po , the number of common feature points currently tracked is T po Among them, if T po Less than Th p When E po Equal to Th p -T po If T po Equal to Th p When E po is equal to 0.

[0298] For any current frame normal image, feature point tracking can be performed on the current frame normal image according to the supplementary fisheye feature points of each current frame fisheye image to obtain the newly supplemented normal feature points of the current frame normal image. If the number of newly supplemented normal feature points of the current frame normal image is equal to the number of normal feature points A that need to be supplemented in the current frame normal image, po , then the tracking of the current frame ordinary image ends. If the number of newly tracked ordinary feature points of the current frame ordinary image is less than A po , then extract A from the current frame normal image po -r p feature points, among which r p Indicates the number of newly tracked common feature points in the current frame of the common image.

[0299] Exemplarily, a camera system includes a fisheye camera and a preset type camera, the fisheye camera captures fisheye video (including a first frame of fisheye image 1, a second frame of fisheye image 2, a third frame of fisheye image 3, a fourth frame of fisheye image 4 and a fifth frame of fisheye image 5), the preset type camera captures ordinary video (including a first frame of ordinary image 1, a second frame of ordinary image 2, a third frame of ordinary image 3, a fourth frame of ordinary image 4 and a fifth frame of ordinary image 5), and the fisheye video and the ordinary video have the same time period, such as Fig.16 The flowchart of the pose estimation process of the camera system is shown in FIG. When the pose estimation process is performed on the camera system, it can be initialized by the first frame fisheye image 1 in the fisheye video and the first frame ordinary image 1 in the ordinary video.

[0300] In practical applications, the above initialization process includes: obtaining the initial posture information of the camera system in the first frame, and extracting the feature points of fisheye image 1 and ordinary image 1 to obtain fisheye feature points 1 and ordinary feature points 2, respectively. At this time, it is determined that the initialization is completed; in addition, in the case that the feature point extraction of fisheye image 1 and ordinary image 1 fails, the initialization can be continued, and the feature point extraction of fisheye image 2 and ordinary image 2 continues. If the feature point extraction fails, the feature point extraction of the next frame of fisheye image and the next frame of ordinary image continues, until the feature point extraction is successful, and the initialization is determined to be completed. It should be noted here that if the feature point extraction of fisheye image 3 and ordinary image 3 is successful, it indicates that the initialization of the third frame is successful.

[0301] Furthermore, based on the successful initialization mentioned above, the fourth frame fisheye image 4 can be tracked between frames through the third frame fisheye image 3 to obtain the fisheye feature points 4 of the fisheye image 4, and the fourth frame ordinary image 4 can be tracked between frames through the third frame ordinary image 3 to obtain the ordinary feature points 4 of the ordinary image 4, and then the fisheye feature points 4, fisheye feature points 3, ordinary feature points 4 and ordinary feature points 3 are processed for pose estimation to obtain the pose information of the camera system in the fourth frame.

[0302] Next, feature point post-processing is performed on the normal image 4 and the fisheye image 4 to obtain normal feature points 41 and fisheye feature points 41 respectively, and then inter-frame feature point tracking is performed after successful initialization, that is, inter-frame feature point tracking is performed on the fifth frame normal image 5 through the normal feature points 41, and inter-frame feature point tracking is performed on the fifth frame fisheye image 5 through the fisheye feature points 41, and then the pose estimation processing is performed on the camera system of the fifth frame to obtain the pose information of the camera system in the fifth frame. If both the normal image and the fisheye image include multiple frames, other frames can be processed according to the above steps to obtain the pose information of the camera system in each frame.

[0303] It should be noted here that if both the fisheye video and the ordinary video include only one frame of image, it is sufficient to obtain only the initial posture information of the camera system, and there is no need to extract feature points from the first frame of image.

[0304] The technical solution in the embodiment of the present application can respectively construct non-depth residual functions of different non-depth feature points to provide basic information for constructing different non-depth information observation models; at the same time, the method does not require complex algorithms to be implemented, and the processing process is relatively simple, thereby improving the construction speed of the non-depth residual function; at the same time, the method can obtain the position coordinates of the non-depth feature points corresponding to the depth information, and on this basis, the method can finally determine the position and posture of the camera system when facing scenes such as weak textures and rapid camera motion, and still show high accuracy and good robustness.

[0305] In one embodiment, the present application also provides a method for determining the position and posture of a camera system, the method comprising the following process:

[0306] (1) obtaining a current frame of fisheye images captured by each fisheye camera in the camera system and a current frame of ordinary images captured by each preset type of camera;

[0307] (2) obtaining the historical frame fisheye feature points of the historical frame fisheye images collected by each fisheye camera;

[0308] If the historical frame fisheye image includes a previous frame fisheye image of the current frame fisheye image, and the previous frame fisheye image is the initial frame fisheye image, then the above step (2) may include:

[0309] (21) Obtain the previous frame of fisheye image captured by each fisheye camera.

[0310] (22) extracting a preset number of fisheye feature feature points from a target previous frame fisheye image in each previous frame fisheye image; the target previous frame fisheye image represents any one of the previous frame fisheye images.

[0311] (23) Generate an image sequence using each previous frame of fisheye images; the previous frame of fisheye image of the target in the image sequence is the starting fisheye image.

[0312] (24) For any previous fisheye image in the image sequence, the feature points are tracked from the previous fisheye image based on the feature points extracted from all previous fisheye images before the previous fisheye image.

[0313] (25) If the number of tracked feature points on the previous frame of fisheye image is equal to the preset number of fisheye features, the tracked feature points are used as the previous frame of fisheye feature points of the previous frame of fisheye image;

[0314] Alternatively, if the number of feature points tracked on the previous frame of fisheye image is less than the preset number of fisheye features, the number of compensating feature points of the previous frame of fisheye image is determined based on the number of feature points tracked on the previous frame of fisheye image and the preset number of fisheye features; based on the number of compensating feature points of the previous frame of fisheye image, feature point compensation extraction is performed on the previous frame of fisheye image to obtain the previous frame of fisheye feature points of the previous frame of fisheye image.

[0315] (3) obtaining common feature points of historical frames of common images collected by cameras of each preset type;

[0316] If the historical frame fisheye image includes the previous frame fisheye image of the current frame fisheye image, the historical frame normal image includes the previous frame normal image of the current frame normal image, and the previous frame fisheye image is the initial frame fisheye image, and the previous frame normal image is the initial frame normal image, then the above step (3) may include:

[0317] (31) obtaining the previous frame fisheye feature points of the previous frame fisheye image captured by each fisheye camera;

[0318] (32) For any previous frame of ordinary image, tracking the feature points of the previous frame of ordinary image is performed according to the fisheye feature points of each previous frame to obtain candidate ordinary feature points of the previous frame of ordinary image;

[0319] (33) If the number of candidate common feature points of the previous frame of common image is equal to the preset number of common features, the candidate common feature points of the previous frame of common image are determined as the previous frame of common feature points of the previous frame of common image;

[0320] Alternatively, if the number of candidate ordinary feature points of the previous frame ordinary image is less than the preset number of ordinary features, the number of compensating feature points of the previous frame ordinary image is determined based on the total number of candidate ordinary feature points of the previous frame ordinary image and the preset number of ordinary features; based on the number of compensating feature points of the previous frame ordinary image, feature point compensation extraction is performed on the previous frame ordinary image; based on the feature point compensation extraction result of the previous frame ordinary image and the candidate ordinary feature points, the previous frame ordinary feature points of the previous frame ordinary image are determined.

[0321] (4) tracking the inter-frame feature points of the current frame of the ordinary image based on the previous frame of the ordinary image, and obtaining candidate ordinary feature points of the current frame of the ordinary image;

[0322] (5) If the number of candidate ordinary feature points of the current frame ordinary image is equal to the preset number of ordinary features, the candidate ordinary feature points of the current frame ordinary image are determined as the current frame ordinary feature points; or, if the number of candidate ordinary feature points of the current frame ordinary image is less than the preset number of ordinary features, the feature points of the current frame ordinary image are tracked according to the previous frame ordinary feature points of the previous frame ordinary image captured by the preset type camera, so as to determine the current frame ordinary feature points of the current frame ordinary image.

[0323] (6) Obtaining the previous frame of fisheye feature points of the previous frame of fisheye image captured by each fisheye camera, and the previous frame of common feature points of the previous frame of common image captured by each preset type of camera.

[0324] (7) Obtaining depth feature points having depth information from the fisheye feature points of each current frame and the ordinary feature points of each current frame; the depth feature points include the three-dimensional position coordinates of the feature points.

[0325] (8) Determine the unit sphere coordinates of each depth feature point according to the current position coordinates of each depth feature point in the current frame image to which it belongs.

[0326] (9) Determine the depth residual function of each depth feature point based on the current position coordinates of each depth feature point in the world coordinate system and the unit sphere coordinates of each depth feature point, as well as the external parameters between the current camera system coordinate system and the camera coordinate system corresponding to the current frame image to which each depth feature point belongs.

[0327] (10) Determine the depth information observation model based on the depth residual function of each depth feature point.

[0328] (11) Obtain multiple first non-depth feature points, multiple second non-depth feature points and multiple third non-depth feature points from each current frame fisheye feature point and each current frame ordinary feature point; wherein each first non-depth feature point represents a current frame feature point without depth information that is in at least two current frame images and in at least two previous frame images; each second non-depth feature point represents a current frame feature point without depth information that is only in one current frame image and in at least two previous frame images; each third non-depth feature point represents a current frame feature point without depth information that is in at least two current frame images and in only one previous frame image; the current frame image is either the current frame fisheye image or the current frame ordinary image; the previous frame image is either the previous frame fisheye image or the previous frame ordinary image.

[0329] (12) Obtain the previous position coordinates of each first non-depth feature point on the previous frame image.

[0330] (13) Triangulating the front position coordinates of each first non-depth feature point to obtain the three-dimensional front position coordinates of each first non-depth feature point in the world coordinate system; and triangulating the current position coordinates of each first non-depth feature point to obtain the three-dimensional current position coordinates of each first non-depth feature point in the current camera system coordinate system.

[0331] (14) Constructing a non-depth residual function of each first non-depth feature point according to the three-dimensional front position coordinates of each first non-depth feature point in the world coordinate system and the three-dimensional current position coordinates of each first non-depth feature point in the current camera system coordinate system.

[0332] (15) Obtain the previous position coordinates of each second non-depth feature point on the previous frame image.

[0333] (16) Triangulate the front position coordinates of each second non-depth feature point to obtain the three-dimensional front position coordinates of each second non-depth feature point in the world coordinate system.

[0334] (17) Determine the unit sphere coordinates of each second non-depth feature point according to the current position coordinates of each second non-depth feature point in the current frame image to which it belongs.

[0335] (18) Constructing a non-depth residual function of each second non-depth feature point based on the three-dimensional front position coordinates of each second non-depth feature point in the world coordinate system, the unit sphere coordinates of each second non-depth feature point, and the external parameters between the current camera system coordinate system and the camera coordinate system corresponding to the current frame image to which each second non-depth feature point belongs.

[0336] (19) Triangulate the current position coordinates of each third non-depth feature point on the current frame image to obtain the three-dimensional current position coordinates of each third non-depth feature point in the current camera coordinate system.

[0337] (20) Determine the unit sphere coordinates of each third non-depth feature point according to the previous position coordinates of each third non-depth feature point in the corresponding previous frame image.

[0338] (21) Construct a non-depth residual function of each third non-depth feature point based on the three-dimensional current position coordinates of each third non-depth feature point in the current camera coordinate system, the unit sphere coordinates of each third non-depth feature point, the transformation relationship between the world coordinate system and the front camera system coordinate system, and the external parameters between the current camera system coordinate system and the camera coordinate system corresponding to the previous frame image to which each third non-depth feature point belongs.

[0339] (22) Determine a non-depth information observation model according to the non-depth residual function of each first non-depth feature point, the non-depth residual function of each second non-depth feature point, and the non-depth residual function of each third non-depth feature point.

[0340] (23) Based on the current posture observation model, construct the current posture optimization function of the camera system.

[0341] (24) Based on the current posture optimization function, determine the current posture information of the camera system.

[0342] The execution process of the above (1) to (24) can be specifically referred to the description of the above embodiment. The implementation principles and technical effects are similar and will not be repeated here.

[0343] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0344] Based on the same inventive concept, the embodiment of the present application also provides a camera system posture determination device for implementing the above-mentioned camera system posture determination method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in the embodiments of the posture determination device for one or more camera systems provided below can refer to the limitations of the camera system posture determination method above, and will not be repeated here.

[0345] In one embodiment, Fig.17 Schematic diagram of the structure of a camera system posture determination device in one embodiment of the present application. The camera system posture determination device provided in the embodiment of the present application can be applied to a computer device. Fig.17 As shown, the apparatus for determining the position and posture of a camera system in an embodiment of the present application may include: an image acquisition module 11, a feature point tracking module 12 and a determination module 13, wherein:

[0346] An image acquisition module 11 is used to acquire a fisheye image of a current frame acquired by a fisheye camera in a camera system and a normal image of a current frame acquired by a preset type camera;

[0347] A feature point tracking module 12 is used to track the feature points of the current frame fisheye image to obtain the current frame fisheye feature points of the current frame fisheye image, and to track the feature points of the current frame ordinary image to obtain the current frame ordinary feature points of the current frame ordinary image;

[0348] The determination module 13 is used to determine the current posture information of the camera system according to the fisheye feature points of the current frame and the common feature points of the current frame.

[0349] The device for determining the posture of a camera system provided in the embodiment of the present application can be used to execute the technical solution in the embodiment of the method for determining the posture of a camera system described above in the present application. The implementation principle and technical effects are similar and will not be repeated here.

[0350] In one embodiment, the feature point tracking module 12 includes: a first acquisition unit and a first feature point tracking unit, wherein:

[0351] A first acquisition unit is used to acquire historical frame fisheye feature points of historical frame fisheye images captured by the fisheye camera;

[0352] The first feature point tracking unit is used to track the inter-frame feature points of the current frame fisheye image according to the historical frame fisheye feature points to obtain the current frame fisheye feature points of the current frame fisheye image.

[0353] The device for determining the posture of a camera system provided in the embodiment of the present application can be used to execute the technical solution in the embodiment of the method for determining the posture of a camera system described above in the present application. The implementation principle and technical effects are similar and will not be repeated here.

[0354] In one embodiment, the camera system includes at least one fisheye camera. If the historical frame fisheye image includes a previous frame fisheye image of the current frame fisheye image, and the previous frame fisheye image is the initial frame fisheye image, the first acquisition unit includes: a first acquisition subunit, a first extraction subunit and a second extraction subunit, wherein:

[0355] The first acquisition subunit is used to acquire the previous frame of fisheye image captured by each fisheye camera;

[0356] The first extraction subunit is used to extract a preset number of fisheye feature points from a target previous frame fisheye image in each previous frame fisheye image; the target previous frame fisheye image represents any one of the previous frame fisheye images;

[0357] The second extraction subunit is used to extract feature points on other previous fisheye images in each previous fisheye image in turn according to the feature points extracted on the previous fisheye image of the target, so as to obtain the previous fisheye feature points of the previous fisheye image captured by each fisheye camera.

[0358] The device for determining the posture of a camera system provided in the embodiment of the present application can be used to execute the technical solution in the embodiment of the method for determining the posture of a camera system described above in the present application. The implementation principle and technical effects are similar and will not be repeated here.

[0359] In one embodiment, the second extraction subunit is specifically used for:

[0360] Generate an image sequence from each previous frame of fisheye images; the previous frame of fisheye image of the target in the image sequence is the starting fisheye image;

[0361] For any previous fisheye image in the image sequence, the feature points are tracked from the previous fisheye image according to the feature points extracted from all previous fisheye images before the previous fisheye image;

[0362] If the number of tracked feature points on the previous frame of fisheye image is equal to the preset number of fisheye features, the tracked feature points are used as the previous frame of fisheye feature points of the previous frame of fisheye image.

[0363] The device for determining the posture of a camera system provided in the embodiment of the present application can be used to execute the technical solution in the embodiment of the method for determining the posture of a camera system described above in the present application. The implementation principle and technical effects are similar and will not be repeated here.

[0364] In one embodiment, the second extraction subunit is further configured to:

[0365] If the number of feature points tracked on the previous frame of fisheye image is less than the preset number of fisheye features, the number of compensating feature points of the previous frame of fisheye image is determined according to the number of feature points tracked on the previous frame of fisheye image and the preset number of fisheye features;

[0366] According to the number of compensated feature points of the previous frame of fisheye image, feature point compensation extraction is performed on the previous frame of fisheye image to obtain the previous frame of fisheye feature points of the previous frame of fisheye image.

[0367] The device for determining the posture of a camera system provided in the embodiment of the present application can be used to execute the technical solution in the embodiment of the method for determining the posture of a camera system described above in the present application. The implementation principle and technical effects are similar and will not be repeated here.

[0368] In one embodiment, the feature point tracking module 12 further includes: a second acquisition unit, a second feature point tracking unit and a first determination unit, wherein:

[0369] A second acquisition unit is used to acquire common feature points of historical frames of common images captured by a preset type of camera;

[0370] The second feature point tracking unit is used to track the inter-frame feature points of the current frame ordinary image according to the historical frame ordinary feature points of the historical frame ordinary image, so as to obtain candidate ordinary feature points of the current frame ordinary image;

[0371] The first determining unit is used to determine the candidate ordinary feature points of the current frame ordinary image as the current frame ordinary feature points of the current frame ordinary image when the number of candidate ordinary feature points of the current frame ordinary image is equal to the preset number of ordinary features.

[0372] The device for determining the posture of a camera system provided in the embodiment of the present application can be used to execute the technical solution in the embodiment of the method for determining the posture of a camera system described above in the present application. The implementation principle and technical effects are similar and will not be repeated here.

[0373] In one embodiment, the camera system includes at least one fisheye camera, if the historical frame fisheye image includes a previous frame fisheye image of the current frame fisheye image, the historical frame normal image includes a previous frame normal image of the current frame normal image, and the previous frame fisheye image is an initial frame fisheye image, and the previous frame normal image is an initial frame normal image, then the second acquisition unit includes: a second acquisition subunit, a first tracking subunit and a first determination subunit, wherein:

[0374] The second acquisition subunit is used to acquire the previous frame of fisheye feature points of the previous frame of fisheye image captured by each fisheye camera;

[0375] The first tracking subunit is used to track the feature points of the previous frame of ordinary image according to the fisheye feature points of each previous frame, so as to obtain candidate ordinary feature points of the previous frame of ordinary image;

[0376] The first determination subunit is used to determine the candidate ordinary feature points of the previous frame of ordinary image as the previous frame ordinary feature points of the previous frame of ordinary image when the number of candidate ordinary feature points of the previous frame of ordinary image is equal to the preset number of ordinary features.

[0377] The device for determining the posture of a camera system provided in the embodiment of the present application can be used to execute the technical solution in the embodiment of the method for determining the posture of a camera system described above in the present application. The implementation principle and technical effects are similar and will not be repeated here.

[0378] In one embodiment, the second acquisition unit further includes: a second determination subunit, a feature point extraction subunit and a third determination subunit, wherein:

[0379] The second determining subunit is used to determine the number of compensating feature points of the previous frame of ordinary image according to the total number of candidate ordinary feature points of the previous frame of ordinary image and the preset number of ordinary features when the number of candidate ordinary feature points of the previous frame of ordinary image is less than the preset number of ordinary features;

[0380] The feature point extraction subunit is used to perform feature point compensation extraction on the previous frame of ordinary image according to the number of compensated feature points of the previous frame of ordinary image;

[0381] The third determination subunit is used to determine the common feature points of the previous frame according to the feature point compensation extraction result of the previous frame of the common image and the candidate common feature points.

[0382] The device for determining the posture of a camera system provided in the embodiment of the present application can be used to execute the technical solution in the embodiment of the method for determining the posture of a camera system described above in the present application. The implementation principle and technical effects are similar and will not be repeated here.

[0383] In one embodiment, the feature point tracking module 12 further includes: a third feature point tracking unit, wherein:

[0384] The third feature point tracking unit is used to track the feature points of the current frame ordinary image according to the historical frame ordinary feature points of the historical frame ordinary images collected by a preset type of camera when the number of candidate ordinary feature points of the current frame ordinary image is less than the preset number of ordinary features, so as to determine the current frame ordinary feature points of the current frame ordinary image.

[0385] The device for determining the posture of a camera system provided in the embodiment of the present application can be used to execute the technical solution in the embodiment of the method for determining the posture of a camera system described above in the present application. The implementation principle and technical effects are similar and will not be repeated here.

[0386] In one embodiment, the determination module 13 includes: a feature point acquisition unit, an observation model construction unit, an optimization function construction unit and a posture information determination unit, wherein:

[0387] A feature point acquisition unit, used to acquire the previous frame of fisheye feature points of the previous frame of fisheye image acquired by the fisheye camera, and the previous frame of common feature points of the previous frame of common image acquired by the preset type camera;

[0388] An observation model building unit, used to build a current posture observation model of the camera system according to the fisheye feature points of the current frame, the fisheye feature points of the previous frame, the common feature points of the current frame and the common feature points of the previous frame;

[0389] An optimization function construction unit, used to construct a current posture optimization function of the camera system according to the current posture observation model;

[0390] The posture information determination unit is used to determine the current posture information of the camera system according to the current posture optimization function.

[0391] The device for determining the posture of a camera system provided in the embodiment of the present application can be used to execute the technical solution in the embodiment of the method for determining the posture of a camera system described above in the present application. The implementation principle and technical effects are similar and will not be repeated here.

[0392] In one embodiment, the current posture observation model includes a depth information observation model and a non-depth information observation model; the observation model construction unit includes: a first construction subunit and a second construction subunit, wherein:

[0393] A first construction subunit is used to determine a depth information observation model according to the fisheye feature points of the current frame and the common feature points of the current frame;

[0394] The second construction subunit is used to construct a non-depth information observation model according to the fisheye feature points of the current frame, the fisheye feature points of the previous frame, the common feature points of the current frame and the common feature points of the previous frame.

[0395] The device for determining the posture of a camera system provided in the embodiment of the present application can be used to execute the technical solution in the embodiment of the method for determining the posture of a camera system described above in the present application. The implementation principle and technical effects are similar and will not be repeated here.

[0396] In one embodiment, the first construction subunit is specifically used for:

[0397] Obtaining depth feature points with depth information from the fisheye feature points of the current frame and the common feature points of the current frame; the depth feature points include the three-dimensional position coordinates of the feature points;

[0398] Determine the unit sphere coordinates of each depth feature point according to the current position coordinates of each depth feature point in the current frame image;

[0399] Determine the depth residual function of each depth feature point according to the current position coordinates of each depth feature point in the world coordinate system and the unit sphere coordinates of each depth feature point, as well as the external parameters between the current camera system coordinate system and the camera coordinate system corresponding to the current frame image to which each depth feature point belongs;

[0400] The depth information observation model is determined according to the depth residual function of each depth feature point.

[0401] The device for determining the posture of a camera system provided in the embodiment of the present application can be used to execute the technical solution in the embodiment of the method for determining the posture of a camera system described above in the present application. The implementation principle and technical effects are similar and will not be repeated here.

[0402] In one embodiment, the second construction subunit includes: a non-depth feature point acquisition subunit, a residual function construction subunit and an observation model determination subunit, wherein:

[0403] The non-depth feature point acquisition subunit is used to acquire a plurality of first non-depth feature points, a plurality of second non-depth feature points and a plurality of third non-depth feature points from the fisheye feature points of the current frame and the common feature points of the current frame;

[0404] A residual function construction subunit, used to construct a non-depth residual function of each first non-depth feature point, a non-depth residual function of each second non-depth feature point, and a non-depth residual function of each third non-depth feature point;

[0405] An observation model determination subunit, configured to determine a non-depth information observation model according to a non-depth residual function of each first non-depth feature point, a non-depth residual function of each second non-depth feature point, and a non-depth residual function of each third non-depth feature point;

[0406] Among them, each first non-depth feature point represents a current frame feature point without depth information that is in at least two current frame images and in at least two previous frame images; each second non-depth feature point represents a current frame feature point without depth information that is only in one current frame image and in at least two previous frame images; each third non-depth feature point represents a current frame feature point without depth information that is in at least two current frame images and only in one previous frame image; the current frame image is either the current frame fisheye image or the current frame ordinary image; the previous frame image is either the previous frame fisheye image or the previous frame ordinary image.

[0407] The device for determining the posture of a camera system provided in the embodiment of the present application can be used to execute the technical solution in the embodiment of the method for determining the posture of a camera system described above in the present application. The implementation principle and technical effects are similar and will not be repeated here.

[0408] In one embodiment, the residual function construction subunit is specifically used for:

[0409] Obtaining the previous position coordinates of each first non-depth feature point on the corresponding previous frame image;

[0410] Triangulate the front position coordinates of each first non-depth feature point to obtain the three-dimensional front position coordinates of each first non-depth feature point in the world coordinate system; and triangulate the current position coordinates of each first non-depth feature point to obtain the three-dimensional current position coordinates of each first non-depth feature point in the current camera system coordinate system;

[0411] A non-depth residual function of each first non-depth feature point is constructed according to the three-dimensional front position coordinates of each first non-depth feature point in the world coordinate system and the three-dimensional current position coordinates of each first non-depth feature point in the current camera system coordinate system.

[0412] The device for determining the posture of a camera system provided in the embodiment of the present application can be used to execute the technical solution in the embodiment of the method for determining the posture of a camera system described above in the present application. The implementation principle and technical effects are similar and will not be repeated here.

[0413] In one embodiment, the residual function construction subunit is further used for:

[0414] Obtaining the previous position coordinates of each second non-depth feature point on the corresponding previous frame image;

[0415] Triangulating the front position coordinates of each second non-depth feature point to obtain the three-dimensional front position coordinates of each second non-depth feature point in the world coordinate system;

[0416] Determine the unit sphere coordinates of each second non-depth feature point according to the current position coordinates of each second non-depth feature point in the current frame image to which it belongs;

[0417] A non-depth residual function of each second non-depth feature point is constructed according to the three-dimensional front position coordinates of each second non-depth feature point in the world coordinate system, the unit sphere coordinates of each second non-depth feature point, and the external parameters between the current camera system coordinate system and the camera coordinate system corresponding to the current frame image to which each second non-depth feature point belongs.

[0418] The device for determining the posture of a camera system provided in the embodiment of the present application can be used to execute the technical solution in the embodiment of the method for determining the posture of a camera system described above in the present application. The implementation principle and technical effects are similar and will not be repeated here.

[0419] In one embodiment, the residual function construction subunit is further used for:

[0420] Triangulate the current position coordinates of each third non-depth feature point on the current frame image to obtain the three-dimensional current position coordinates of each third non-depth feature point in the current camera coordinate system;

[0421] Determine the unit sphere coordinates of each third non-depth feature point according to the previous position coordinates of each third non-depth feature point in the corresponding previous frame image;

[0422] According to the three-dimensional current position coordinates of each third non-depth feature point in the current camera coordinate system, the unit sphere coordinates of each third non-depth feature point, the transformation relationship between the world coordinate system and the front camera system coordinate system, and the external parameters between the current camera system coordinate system and the camera coordinate system corresponding to the previous frame image to which each third non-depth feature point belongs, a non-depth residual function of each third non-depth feature point is constructed.

[0423] The device for determining the posture of a camera system provided in the embodiment of the present application can be used to execute the technical solution in the embodiment of the method for determining the posture of a camera system described above in the present application. The implementation principle and technical effects are similar and will not be repeated here.

[0424] For the specific definition of the posture determination device of the camera system, please refer to the definition of the posture determination method of the camera system above, which will not be repeated here. Each module in the posture determination device of the above-mentioned camera system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0425] In one embodiment, a movable platform is also provided. Figure 1 As shown, the movable platform includes:

[0426] Body;

[0427] A power system is arranged in the body, and the power system is used to provide power for the movable platform;

[0428] Camera system;

[0429] processor;

[0430] a memory for storing a computer program for execution by a processor;

[0431] When the processor executes the computer program, the steps of the method in any of the above embodiments are implemented.

[0432] The movable platform provided in the embodiment of the present application can be used to execute the technical solution in the embodiment of the method for determining the posture of the camera system mentioned above in the present application. Its implementation principle and technical effect are similar and will not be repeated here.

[0433] In one embodiment, a computer device is also provided. The internal structure diagram of the computer device can be as follows: Fig.18 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide processing capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store each current frame fisheye image, each current frame normal image, each previous frame fisheye image and each previous frame normal image. The network interface of the computer device is used to communicate with an external endpoint through a network connection. When the computer program is executed by the processor, a method for determining the posture of a camera system is implemented.

[0434] Those skilled in the art will understand that Fig.18 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0435] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of the method in any of the above embodiments when executing the computer program.

[0436] In one embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in any of the above embodiments are implemented.

[0437] In one embodiment, a computer program product is also provided, including a computer program, which implements the steps of the method in any of the above embodiments when executed by a processor.

[0438] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0439] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0440] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for determining the position and posture of a camera system, characterized in that: The method comprises: Obtain a fisheye image of a current frame captured by a fisheye camera and a normal image of a current frame captured by a preset type camera in the camera system; Tracking feature points of the current frame fisheye image to obtain current frame fisheye feature points of the current frame fisheye image, and tracking feature points of the current frame normal image to obtain current frame normal feature points of the current frame normal image; The current position information of the camera system is determined according to the fisheye feature points of the current frame and the common feature points of the current frame.

2. The method according to claim 1, characterized in that The step of tracking the feature points of the current frame fisheye image to obtain the current frame fisheye feature points of the current frame fisheye image includes: Acquire historical frame fisheye feature points of historical frame fisheye images captured by the fisheye camera; According to the historical frame fisheye feature points, inter-frame feature point tracking is performed on the current frame fisheye image to obtain the current frame fisheye feature points of the current frame fisheye image.

3. The method according to claim 2, characterized in that The camera system includes at least one fisheye camera; if the historical frame fisheye image includes a previous frame fisheye image of the current frame fisheye image, and the previous frame fisheye image is an initial frame fisheye image, then obtaining the historical frame fisheye feature points of the historical frame fisheye image acquired by the fisheye camera includes: Acquire a previous frame of fisheye image captured by each of the fisheye cameras; Extracting a preset number of feature points of fisheye features from a target previous frame fisheye image in each of the previous frame fisheye images; the target previous frame fisheye image represents any one of the previous frame fisheye images; According to the feature points extracted from the previous frame fisheye image of the target, the feature points on other previous frame fisheye images in each previous frame fisheye image are extracted in sequence to obtain the previous frame fisheye feature points of the previous frame fisheye image captured by each fisheye camera.

4. The method according to claim 3, characterized in that The extracting, in sequence, feature points on other previous fisheye images in each of the previous fisheye images according to the feature points extracted on the previous fisheye image of the target, comprises: Generate an image sequence from each of the previous fisheye images; the previous target fisheye image in the image sequence is the starting fisheye image; For any previous fisheye image in the image sequence, tracking feature points from the previous fisheye image according to feature points extracted from all previous fisheye images before the previous fisheye image; If the number of tracked feature points on the previous frame of fisheye image is equal to the preset number of fisheye features, the tracked feature points are used as the previous frame of fisheye feature points of the previous frame of fisheye image.

5. The method according to claim 4, characterized in that The method further comprises: If the number of feature points tracked on the previous frame of fisheye image is less than the preset number of fisheye features, determining the number of compensating feature points of the previous frame of fisheye image according to the number of feature points tracked on the previous frame of fisheye image and the preset number of fisheye features; According to the number of compensated feature points of the previous frame of fisheye image, feature point compensation extraction is performed on the previous frame of fisheye image to obtain the previous frame of fisheye feature points of the previous frame of fisheye image.

6. The method according to any one of claims 1 to 5, characterized in that The step of tracking feature points of the current frame normal image to obtain the current frame normal feature points of the current frame normal image includes: Acquire common feature points of historical frames of common images captured by the preset type of camera; According to the historical frame common feature points of the historical frame common image, inter-frame feature point tracking is performed on the current frame common image to obtain candidate common feature points of the current frame common image; If the number of candidate common feature points of the current frame common image is equal to the preset number of common features, the candidate common feature points of the current frame common image are determined as the current frame common feature points of the current frame common image.

7. The method according to claim 6, characterized in that The camera system includes at least one fisheye camera; if the historical frame fisheye image includes a previous frame fisheye image of the current frame fisheye image, the historical frame normal image includes a previous frame normal image of the current frame normal image, and the previous frame fisheye image is an initial frame fisheye image, and the previous frame normal image is an initial frame normal image, then obtaining the historical frame normal feature points of the historical frame normal image acquired by the preset type camera includes: Acquire the previous frame of fisheye feature points of the previous frame of fisheye image captured by each of the fisheye cameras; According to each of the fisheye feature points of the previous frame, tracking the feature points of the previous frame of ordinary image, to obtain candidate ordinary feature points of the previous frame of ordinary image; If the number of candidate common feature points of the previous frame of common image is equal to the preset number of common features, the candidate common feature points of the previous frame of common image are determined as the previous frame of common feature points of the previous frame of common image.

8. The method according to claim 7, characterized in that The method further comprises: If the number of candidate common feature points of the previous frame of common image is less than the preset number of common features, determining the number of compensating feature points of the previous frame of common image according to the total number of candidate common feature points of the previous frame of common image and the preset number of common features; According to the number of compensated feature points of the previous frame of ordinary image, extracting feature points from the previous frame of ordinary image; The common feature points of the previous frame are determined according to the feature point compensation extraction result of the previous frame common image and the candidate common feature points.

9. The method according to claim 6, characterized in that The method further comprises: If the number of candidate ordinary feature points of the current frame ordinary image is less than the preset number of ordinary features, feature point tracking is performed on the current frame ordinary image according to the historical frame ordinary feature points of the historical frame ordinary images captured by the preset type of camera to determine the current frame ordinary feature points of the current frame ordinary image.

10. The method according to any one of claims 1 to 5, characterized in that The determining the current posture information of the camera system according to the fisheye feature points of the current frame and the common feature points of the current frame includes: Acquire the previous frame of fisheye feature points of the previous frame of fisheye image captured by the fisheye camera, and the previous frame of common feature points of the previous frame of common image captured by the preset type camera; Constructing a current posture observation model of the camera system according to the fisheye feature points of the current frame, the fisheye feature points of the previous frame, the common feature points of the current frame and the common feature points of the previous frame; Constructing a current posture optimization function of the camera system according to the current posture observation model; Based on the current pose optimization function, current pose information of the camera system is determined.

11. The method according to claim 10, characterized in that The current posture observation model includes a depth information observation model and a non-depth information observation model; the current posture observation model of the camera system is constructed according to the fisheye feature points of the current frame, the fisheye feature points of the previous frame, the common feature points of the current frame and the common feature points of the previous frame, including: Determining the depth information observation model according to the fisheye feature points of the current frame and the common feature points of the current frame; The non-depth information observation model is constructed according to the fisheye feature points of the current frame, the fisheye feature points of the previous frame, the common feature points of the current frame and the common feature points of the previous frame.

12. The method according to claim 11, characterized in that The determining the depth information observation model according to the fisheye feature points of the current frame and the common feature points of the current frame includes: Acquire a depth feature point having depth information from the fisheye feature points of the current frame and the common feature points of the current frame; the depth feature point includes the three-dimensional position coordinates of the feature point; Determining the unit sphere coordinates of each of the depth feature points according to the current position coordinates of each of the depth feature points in the current frame image to which it belongs; Determine a depth residual function of each depth feature point according to the current position coordinates of each depth feature point in the world coordinate system and the unit sphere coordinates of each depth feature point, as well as an external parameter between a current camera system coordinate system and a camera coordinate system corresponding to a current frame image to which each depth feature point belongs; The depth information observation model is determined according to the depth residual function of each of the depth feature points.

13. The method according to claim 11, characterized in that The constructing the non-depth information observation model according to the fisheye feature points of the current frame, the fisheye feature points of the previous frame, the common feature points of the current frame and the common feature points of the previous frame includes: Acquire a plurality of first non-depth feature points, a plurality of second non-depth feature points, and a plurality of third non-depth feature points from the current frame fisheye feature points and the current frame common feature points; Constructing a non-depth residual function of each of the first non-depth feature points, a non-depth residual function of each of the second non-depth feature points, and a non-depth residual function of each of the third non-depth feature points; The non-depth information observation model is determined according to the non-depth residual function of each of the first non-depth feature points, the non-depth residual function of each of the second non-depth feature points, and the non-depth residual function of each of the third non-depth feature points.

14. The method according to claim 13, characterized in that Each of the first non-depth feature points represents a current frame feature point without depth information that is in at least two current frame images and in at least two previous frame images; Each of the second non-depth feature points represents a current frame feature point without depth information that is only in one current frame image and in at least two previous frame images; Each of the third non-depth feature points represents a current frame feature point without depth information that is in at least two current frame images and in only one previous frame image; The current frame image is either a current frame fisheye image or a current frame ordinary image; the previous frame image is either a previous frame fisheye image or a previous frame ordinary image.

15. The method according to claim 14, characterized in that The constructing the non-depth residual function of each of the first non-depth feature points includes: Obtaining the previous position coordinates of each of the first non-depth feature points on the corresponding previous frame image; Triangulating the front position coordinates of each of the first non-depth feature points to obtain the three-dimensional front position coordinates of each of the first non-depth feature points in the world coordinate system; and triangulating the current position coordinates of each of the first non-depth feature points to obtain the three-dimensional current position coordinates of each of the first non-depth feature points in the current camera system coordinate system; A non-depth residual function of each of the first non-depth feature points is constructed according to the three-dimensional front position coordinates of each of the first non-depth feature points in the world coordinate system and the three-dimensional current position coordinates of each of the first non-depth feature points in the current camera system coordinate system.

16. The method according to claim 14, characterized in that The constructing the non-depth residual function of each of the second non-depth feature points includes: Obtaining the previous position coordinates of each of the second non-depth feature points on the corresponding previous frame image; Performing triangulation processing on the front position coordinates of each of the second non-depth feature points to obtain the three-dimensional front position coordinates of each of the second non-depth feature points in the world coordinate system; Determining the unit sphere coordinates of each of the second non-depth feature points according to the current position coordinates of each of the second non-depth feature points in the current frame image to which it belongs; A non-depth residual function of each second non-depth feature point is constructed based on the three-dimensional front position coordinates of each second non-depth feature point in the world coordinate system, the unit sphere coordinates of each second non-depth feature point, and the external parameters between the current camera system coordinate system and the camera coordinate system corresponding to the current frame image to which each second non-depth feature point belongs.

17. The method according to claim 14, characterized in that The constructing of the non-depth residual function of each of the third non-depth feature points includes: Performing triangulation processing on the current position coordinates of each of the third non-depth feature points on the corresponding current frame image to obtain the three-dimensional current position coordinates of each of the third non-depth feature points in the current camera coordinate system; Determining the unit sphere coordinates of each of the third non-depth feature points according to the front position coordinates of each of the third non-depth feature points in the previous frame image to which it belongs; A non-depth residual function of each of the third non-depth feature points is constructed based on the three-dimensional current position coordinates of each of the third non-depth feature points in the current camera coordinate system, the unit sphere coordinates of each of the third non-depth feature points, the transformation relationship between the world coordinate system and the front camera system coordinate system, and the external parameters between the current camera system coordinate system and the camera coordinate system corresponding to the previous frame image to which each of the third non-depth feature points belongs.

18. A device for determining the position and posture of a camera system, characterized in that: The device comprises: An image acquisition module, used to acquire a fisheye image of a current frame acquired by a fisheye camera in a camera system and a normal image of a current frame acquired by a preset type camera; A feature point tracking module is used to track the feature points of the current frame fisheye image to obtain the current frame fisheye feature points of the current frame fisheye image, and to track the feature points of the current frame ordinary image to obtain the current frame ordinary feature points of the current frame ordinary image; A determination module is used to determine the current posture information of the camera system according to the fisheye feature points of the current frame and the common feature points of the current frame.

19. A movable platform, characterized in that: The movable platform comprises: Camera system; processor; A memory for storing a computer program executed by the processor; Wherein, when the processor executes the computer program, the steps of the method described in any one of claims 1-17 are implemented.

20. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 17 are implemented.

21. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

22. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

Images