A multi-view panoramic camera-based target coordinate positioning system and method

Abstract

The application discloses a target coordinate positioning system and method based on a multi-view panoramic camera, relates to the technical field of target coordinate positioning, and comprises the following steps: evaluating the spatial geometric relationship between different camera units; performing target object detection on images to obtain to-be-determined target objects in the images, and screening the to-be-determined target objects in the images in combination with historical images; evaluating the comprehensive cost between the target objects in the images and candidate matching objects in other images, constructing a target cost matrix between the images and the other images and solving the target cost matrix; performing image feature fusion on the images according to the target object matching data between the images, obtaining a fused image, obtaining the three-dimensional coordinates of the target objects in the fused image and collecting the three-dimensional coordinates to obtain target coordinate data in the fused image, so that more target objects can be detected, and accurate positioning of the target objects can be realized.

Description

Technical Field

[0001] This invention relates to the field of target coordinate positioning technology, specifically a target coordinate positioning system and method based on a multi-view panoramic camera. Background Technology

[0002] A multi-view panoramic camera is composed of multiple camera units combined in a specific way. The images captured by the multiple camera units are stitched together using software and hardware to obtain a panoramic image with an ultra-wide field of view. For fast-moving and small targets, the field of view of a traditional single camera is limited, and the target can easily move out of the field of view of a single camera, resulting in the failure of target tracking. Using a gimbal camera to track targets cannot simultaneously grasp the position of multiple targets, and it is easy to lose track of targets when there are multiple targets. However, a multi-view panoramic camera can achieve horizontal panoramic coverage by arranging multiple cameras around the view, completely eliminating blind spots. Moreover, a multi-view panoramic camera can directly calculate the specific indicators of the target in three-dimensional space by using the parallax of multiple cameras viewing the same scene from different perspectives.

[0003] While multi-view panoramic cameras play an irreplaceable role in acquiring the position of fast-moving, small targets, there are still many problems in the actual target localization process. For example, due to the large distance between the actual target and the panoramic multi-view camera, and the complex changes in the target's movement posture, it is difficult to detect accurately and stably. Furthermore, because different types of targets have similar appearance and size, it is easy for different targets in the images captured by multiple camera units to be identified as the same target, making it impossible to achieve accurate target matching. This can not only lead to the loss of targets, but also to the generation of the coordinates of incorrect targets, resulting in false alarms and wasted resources. Summary of the Invention

[0004] The purpose of this invention is to provide a target coordinate positioning system and method based on a multi-view panoramic camera to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a target coordinate localization method based on a multi-view panoramic camera, the method comprising:

[0006] Step S1: Obtain camera parameter data of different camera units in the multi-view panoramic camera, construct the basic matrix between different camera units, evaluate the spatial geometric relationship between different camera units, and obtain spatial epipolar data;

[0007] Step S2: Acquire images captured by camera units in the multi-view panoramic camera, acquire historical images of the area where the images are located, acquire the target database in the platform, perform target object detection on the images, obtain the target objects to be determined, and filter the target objects to obtain target object data.

[0008] Step S3: Obtain target object data from other images captured at the same time as the image capture, obtain spatial epipolar data between camera units of the image and camera units of other images, and obtain candidate matching objects of the target object in other images to obtain target candidate matching data.

[0009] Step S4: Obtain target candidate matching data between the image and other images, evaluate the comprehensive cost between the target object in the image and the candidate matching objects in other images, construct the target cost matrix between the image and other images and solve it to obtain the target object matching data;

[0010] Step S5: Acquire images taken at the same time, obtain target object matching data between the images, perform image feature fusion to obtain a fused image, obtain the three-dimensional coordinates of the target objects in the fused image and aggregate them to obtain target coordinate data.

[0011] Furthermore, step S1 includes:

[0012] Step S11: Obtain camera parameter data for each camera unit in the multi-view panoramic camera, wherein the camera parameter data includes the principal point coordinates and focal length values ​​of the camera unit in the multi-view panoramic camera.

[0013] Based on the camera parameter data, construct the intrinsic parameter matrix K of the camera unit, and construct the intrinsic parameter matrix of each camera unit;

[0014] Step S12: Construct the fundamental matrix between each camera unit. The process for constructing the fundamental matrix between the a-th camera unit and the b-th camera unit is as follows:

[0015] Obtain the three-dimensional coordinates of physical point P in the coordinate systems of the a-th and b-th camera units respectively. a and P b ;

[0016] The rotation matrix R and translation vector t between the a-th camera unit and the b-th camera unit are calculated using the following formula: P a =P b ·R+t;

[0017] Obtain the antisymmetric matrix t´ of the translation vector t, and calculate the fundamental matrix F between the a-th camera unit and the b-th camera unit. (a,b) =K b-T ·R·K a -1 , where K b -T K is the transpose of the inverse of the intrinsic parameter matrix K of the b-th camera unit; a -1 K is the intrinsic parameter matrix of the a-th camera unit. a The inverse matrix;

[0018] Step S13: Evaluate the spatial geometric relationship between the a-th camera unit and the b-th camera unit. The specific evaluation process is as follows:

[0019] Obtain the coordinates (x, y) of image point α in the a-th camera unit. a α ,y a α ), obtain the homogeneous coordinates α´ of image point α. a =[x a α ,y a α ,1] T Calculate the epipolar line L of image point α in the b-th camera unit. b α =F (a,b) ·α´ a ;

[0020] Obtain the coordinates (x, y) of image point β in the b-th camera unit. b β ,y b β ), obtain the homogeneous coordinates β´ of image point β. b =[x b β ,y b β ,1] T Calculate the epipolar line L of image point β in the a-th camera unit. a β =F (a,b) T ·β´ b , of which F (a,b) T Basic matrix F (a,b) Transpose of;

[0021] Obtain the epipolar lines of each image point of the a-th camera unit in the b-th camera unit, obtain the epipolar lines of each image point of the b-th camera unit in the a-th camera unit, and aggregate them to obtain the spatial epipolar data between the a-th camera unit and the b-th camera unit.

[0022] Step S14: Evaluate the spatial geometric relationship between different camera units to obtain spatial epipolar data between different camera units.

[0023] Furthermore, step S2 includes:

[0024] Step S21: Acquire images captured by the camera unit in the multi-view panoramic camera during the current cycle, acquire historical images within the area where the images are located, construct a two-dimensional coordinate system for the images, preprocess the images, and acquire the pixel values ​​of each pixel in the images.

[0025] Step S22: Obtain the preset target database in the platform, wherein the target database includes the image set corresponding to each preset target object;

[0026] Based on the target database and using a preset target detection model, target objects in the image are detected to obtain object region data for each target object to be determined in the image. The object region data is the smallest square region in the image where the target object to be determined is located.

[0027] Get the side length of the smallest square region and record it as the side length of the target region of the target object to be determined. Get the average value of the pixels in the smallest square region where the target object to be determined is located in the image and record it as the pixel value ζ of the marked region of the target object to be determined in the image.

[0028] Step S23: Obtain the three-dimensional coordinates of the target object in the image, obtain the historical images of the region where the image is located, obtain the coordinates of the center point of the target object in the historical image based on the three-dimensional coordinates of the target object in the image, construct a square region with the side length of the target region of the target object as the side length and the coordinates of the center point as the center coordinates, obtain the average value of the pixels in the square region in the historical image, and obtain the pixel value of the feature region of the target object in the historical image.

[0029] Step S24: Filter the target objects to be determined. The specific filtering process is as follows: obtain the mean value ζ´ of the pixel values ​​of the feature regions of the target objects to be determined in each historical image. When the absolute value of the pixel value ζ minus the mean value ζ´ is greater than the preset absolute value threshold, the target object to be determined is determined to be a target object in the target database, and the target object to be determined is retained and recorded as a target object in the image. Otherwise, the target object to be determined is determined not to be a target object in the target database, and the target object to be determined is removed. Obtain the coordinates of the center points of each target object in the image and collect them to obtain the target object data in the image.

[0030] Furthermore, step S3 includes:

[0031] Step S31: Obtain other images from the multi-view panoramic camera at the same time as the image capture, obtain target object data from the other images, obtain target object data from the image, and obtain the coordinates of the center points of each target object in the image from the target object data;

[0032] Step S32: Obtain the spatial epipolar data between the camera unit of the image and the camera units of other images, obtain the coordinates of the center point of a target object in the image, and obtain the coordinates of the center points of several target objects in other images.

[0033] Step S33: Based on the spatial epipolar data between the camera unit of one image and the camera units of other images, obtain the epipolar line L of the coordinates of the center point of a target object in other images. △ =[γ,η,λ] T Obtain the coordinates (x, y) of the center point of the e-th target object in other images. ▽ e ,y ▽ e ), calculate polar line L △ The Euclidean distance d to the center point of the e-th target object (▽,e) :

[0034] ,

[0035] When the Euclidean distance d (▽,e) If the distance is less than the preset distance threshold, the e-th target object is recorded as a candidate matching object for a certain target object; otherwise, the e-th target object is not processed.

[0036] The candidate matching objects of each target object in the image are obtained in other images and aggregated to obtain target candidate matching data between the image and other images.

[0037] Furthermore, step S4 includes:

[0038] Step S41: Obtain target candidate matching data between the image and other images, obtain candidate matching objects of the target object in the image in other images from the target candidate matching data, obtain the Euclidean distance d´ between the epipolar line of the target object in the image and the center point coordinates of the candidate matching objects in other images, obtain the normalized Euclidean distance d´, and record it as the geometric cost value between the target object in the image and the candidate matching objects in other images.

[0039] Step S42: Use a preset object detection model to obtain the bounding boxes of each target object in the image and other images, crop and resize the bounding boxes in the image and other images, and input them into a preset feature extraction model to obtain the feature vectors of each target object in the image and other images, and normalize the feature vectors of each target object in the image and other images.

[0040] Step S43: Obtain the feature vector G of the w-th target object in the image and the v-th target object in the other image respectively. w and G ▽ v Calculate the feature value Q between the w-th target object and the v-th target object. (w,v) :

[0041] ,

[0042] Step S44: When the vth target object is not a candidate matching object of the wth target object, the combined cost of the wth target object and the vth target object is set to the preset maximum value.

[0043] When the v-th target object is a candidate matching object of the w-th target object, calculate the combined cost U of the w-th target object and the v-th target object. (w,v) :

[0044] ,

[0045] Among them, D (w,v) Let ψ be the geometrical cost between the v-th target object and the w-th target object; Q ψ is the preset feature cost weight coefficient; D ψ is the preset geometric cost weighting coefficient; Q +ψ D =1, ψ Q >0, ψ D >0;

[0046] Step S45: Obtain the comprehensive cost value between each target object in the image and several target objects in other images, construct the target cost matrix between the image and other images, obtain the total number m of each target object in the image, and obtain the total number n of several target objects in other images;

[0047] When m=n, the target cost matrix is ​​not processed. When m≠n, the target cost matrix is ​​filled in. The specific filling process is as follows:

[0048] When m>n, the last column of the target cost matrix is ​​filled with mn virtual columns, where the comprehensive cost value of each element in the mn virtual columns is a preset maximum value.

[0049] When m < n, fill n - m dummy rows in the last row of the target cost matrix, where the comprehensive cost values of all elements in the n - m dummy column rows are preset maximum values;

[0050] Step S46: Use the Hungarian algorithm to solve the target cost matrix, obtain the zero elements in each row and column of the feature target cost matrix, and obtain the matching pairs of the positions corresponding to the zero elements in the feature target cost matrix. Among them, one matching pair includes the target object in the image that matches the target object in other images;

[0051] Obtain the target objects in the image that match the target objects in other images, and gather them to obtain the target object matching data between the image and other images;

[0052] In the above steps, obtaining the target objects in the image that match the target objects in other images is because in practice, a certain target object in an image usually has only one matching target object in other images. However, due to the different shooting angles and parameters of different camera units, and for fast - moving target objects with high speed and small size, traditional methods are prone to incorrect matching. The above steps analyze the comprehensive cost of the matching between a certain target object in an image and another target object in other images from two aspects of spatial geometry and appearance features, and use the Hungarian algorithm to solve the target cost matrix through the target cost matrix between the image and other images, so as to accurately obtain the matching relationship between the target objects between the image and other images. This not only reduces the probability of target object loss but also avoids the generation of virtual coordinates of target objects, greatly improving the accuracy of target object coordinate positioning.

[0053] Further, step S5 includes:

[0054] Step S51: Obtain each image with the same shooting time in the multi - view panoramic camera, obtain the target object matching data between each image, and obtain the target object in a certain image that matches the target object in another image from the target object matching data between a certain image and another image;

[0055] Step S52: Obtain the camera parameters of each camera unit corresponding to each image, and combine the target object matching data between each image to perform image feature fusion on each image to obtain a fused image;

[0056] For example, the specific process of performing image feature fusion on each image to obtain a fused image is as follows:

[0057] The system obtains target object matching data between one image and another image, obtains the target object that matches between one image and another image, obtains the projection matrix of the camera unit corresponding to one image and another image, and calculates the three-dimensional coordinates of the target object by solving the linear equation system.

[0058] Image reconstruction is performed using one image and another image, and three-dimensional points are added to several images to obtain the three-dimensional coordinates of the target object in each image;

[0059] The semi-global matching (SGM) method is used to find the corresponding point in another image for every pixel in different images (rather than just the feature points corresponding to the target object);

[0060] Based on the matched pixels, the visual difference corresponding to the pixels is obtained, and the depth maps corresponding to different viewpoints are obtained. All the depth maps corresponding to different viewpoints are converted into a preset unified world coordinate system and fused into a fused image corresponding to each image.

[0061] Step S53: Locate each target object in the fused image to obtain the three-dimensional coordinates of each target object in the fused image. The coordinate system of each three-dimensional coordinate is in the preset world coordinate system. Collect the three-dimensional coordinates of each target object in the fused image to obtain the target coordinate data in the fused image.

[0062] To better implement the above method, a target coordinate localization system based on a multi-view panoramic camera is also proposed. The system includes a camera spatial relationship evaluation module, a target matching module, and a target localization module.

[0063] The camera spatial relationship evaluation module is used to evaluate the spatial geometric relationship between different camera units and obtain the spatial epipolar data between different camera units;

[0064] The target matching module is used to acquire target object data of images and other images, acquire candidate matching objects of the target object in other images based on the target object data, obtain target candidate matching data, construct and solve the target cost matrix between images and other images, and obtain target object matching data between images and other images.

[0065] The target localization module is used to acquire target object matching data between images captured at the same time in a multi-view panoramic camera, perform image feature fusion on each image to obtain a fused image, and acquire the three-dimensional coordinates of the target objects in the fused image to obtain target coordinate data in the fused image.

[0066] Furthermore, the camera spatial relationship evaluation module includes a basic matrix construction unit and a camera spatial relationship evaluation unit;

[0067] The basic matrix construction unit is used to acquire camera parameter data of different camera units in a multi-view panoramic camera and construct the basic matrix between different camera units.

[0068] The camera spatial relationship evaluation unit is used to evaluate the spatial geometric relationship between different camera units based on the fundamental matrix between different camera units, and to obtain the spatial epipolar data between different camera units.

[0069] Furthermore, the target matching module includes a target detection unit, a candidate matching acquisition unit, and a target matching unit;

[0070] The target detection unit is used to detect target objects in the image and filter the undetermined target objects in the image by combining historical images to obtain target object data in the image.

[0071] The candidate matching acquisition unit is used to acquire target object data from other images captured at the same time as the image capture, acquire spatial epipolar data between camera units of the image and camera units of other images, acquire candidate matching objects of the target object in other images, and obtain target candidate matching data.

[0072] The target matching unit is used to construct and solve the target cost matrix between the image and other images based on the target candidate matching data between the image and other images, so as to obtain the target object matching data between the image and other images.

[0073] Furthermore, the target localization module includes an image fusion unit and a target localization unit;

[0074] The image fusion unit is used to acquire images captured at the same time from the multi-view panoramic camera, acquire target object matching data between the images, and perform image feature fusion on the images to obtain a fused image.

[0075] The target localization unit is used to acquire and aggregate the three-dimensional coordinates of the target object in the fused image to obtain the target coordinate data in the fused image.

[0076] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention achieves precise target coordinate positioning based on a multi-view panoramic camera. Considering the actual need for multi-view panoramic cameras to detect fast-moving target objects, it not only uses a preset model to detect target objects, but also analyzes the differences between the pixel values ​​of the target object's location in historical images and the pixel values ​​of the target object's location in the current image to further verify the authenticity of the target object. Furthermore, it performs precise matching of target objects in the images captured by each camera unit in the multi-view panoramic camera, thereby greatly reducing the probability of misjudgment of target objects and enabling precise control of the target object's position. Finally, based on the target object matching relationship in the images captured by each camera unit, the images captured by each camera unit are fused, and the target coordinates of the target objects in the fused image are precisely located. This not only allows for a larger field of view in the acquired images, enabling the detection of more target objects, but also achieves precise positioning of target objects. Attached Figure Description

[0077] Figure 1 This is a flowchart of a target coordinate localization method based on a multi-view panoramic camera according to the present invention;

[0078] Figure 2 This is a flowchart of a target coordinate positioning system based on a multi-view panoramic camera according to the present invention. Detailed Implementation

[0079] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0080] Example: Figures 1-2 As shown, the present invention provides a technical solution, a target coordinate localization method based on a multi-view panoramic camera, the method comprising:

[0081] Step S1: Obtain camera parameter data of different camera units in the multi-view panoramic camera, construct the basic matrix between different camera units, evaluate the spatial geometric relationship between different camera units, and obtain spatial epipolar data;

[0082] For example, a camera unit refers to one of the camera lenses in a multi-view panoramic camera;

[0083] Step S1 includes:

[0084] Step S11: Obtain camera parameter data for each camera unit in the multi-view panoramic camera, wherein the camera parameter data includes the principal point coordinates and focal length values ​​of the camera unit in the multi-view panoramic camera.

[0085] Based on the camera parameter data, construct the intrinsic parameter matrix K of the camera unit, and construct the intrinsic parameter matrix of each camera unit;

[0086] For example, the intrinsic parameter matrix K of the camera unit is specifically:

[0087] ,

[0088] Among them, f x and f y These are the pixel values ​​of the focal length f of the camera unit in the horizontal and vertical directions of the image, respectively; (c x ,c y ) represents the principal point coordinates of the camera unit; c x ,c y These represent the x-axis and y-axis coordinates of the optical center in the image coordinate system, respectively. The optical center is the pixel coordinate of the point in the image sensor through which the optical axis of the camera unit passes.

[0089] For example, f x =f / dx, where dx is the physical width of each pixel in the horizontal direction of the image;

[0090] Step S12: Construct the fundamental matrix between each camera unit. The process for constructing the fundamental matrix between the a-th camera unit and the b-th camera unit is as follows:

[0091] Obtain the three-dimensional coordinates of physical point P in the coordinate systems of the a-th and b-th camera units respectively. a and P b ;

[0092] For example, physical point P is a point that exists in reality;

[0093] The rotation matrix R and translation vector t between the a-th camera unit and the b-th camera unit are calculated using the following formula: P a =P b ·R+t;

[0094] Obtain the antisymmetric matrix t´ of the translation vector t, and calculate the fundamental matrix F between the a-th camera unit and the b-th camera unit. (a,b) =K b -T ·R·K a -1 , where K b -T K is the transpose of the inverse of the intrinsic parameter matrix K of the b-th camera unit;a -1 K is the intrinsic parameter matrix of the a-th camera unit. a The inverse matrix;

[0095] For example, the antisymmetric matrix t´ is specifically:

[0096] ,

[0097] Where t=[t x ,t y ,t z ] T ;

[0098] Step S13: Evaluate the spatial geometric relationship between the a-th camera unit and the b-th camera unit. The specific evaluation process is as follows:

[0099] Obtain the coordinates (x, y) of image point α in the a-th camera unit. a α ,y a α ), obtain the homogeneous coordinates α´ of image point α. a =[x a α ,y a α ,1] T Calculate the epipolar line L of image point α in the b-th camera unit. b α =F (a,b) ·α´ a ;

[0100] Obtain the coordinates (x, y) of image point β in the b-th camera unit. b β ,y b β ), obtain the homogeneous coordinates β´ of image point β. b =[x b β ,y b β ,1] T Calculate the epipolar line L of image point β in the a-th camera unit. a β =F (a,b) T ·β´ b , of which F (a,b) T Basic matrix F (a,b) Transpose of;

[0101] Obtain the epipolar lines of each image point of the a-th camera unit in the b-th camera unit, obtain the epipolar lines of each image point of the b-th camera unit in the a-th camera unit, and aggregate them to obtain the spatial epipolar data between the a-th camera unit and the b-th camera unit.

[0102] Step S14: Evaluate the spatial geometric relationship between different camera units to obtain spatial epipolar data between different camera units;

[0103] Step S2: Acquire images captured by camera units in the multi-view panoramic camera, acquire historical images of the area where the images are located, acquire the target database in the platform, perform target object detection on the images, obtain the target objects to be determined, and filter the target objects to obtain target object data.

[0104] Step S2 includes:

[0105] Step S21: Acquire images captured by the camera unit in the multi-view panoramic camera during the current cycle, acquire historical images within the area where the images are located, construct a two-dimensional coordinate system for the images, preprocess the images, and acquire the pixel values ​​of each pixel in the images.

[0106] For example, preprocessing includes color space conversion, normalization, and image resizing;

[0107] Step S22: Obtain the preset target database in the platform, wherein the target database includes the image set corresponding to each preset target object;

[0108] For example, the preset target objects include sparrows, pigeons, etc.

[0109] Based on the target database and using a preset target detection model, target objects in the image are detected to obtain object region data for each target object to be determined in the image. The object region data is the smallest square region in the image where the target object to be determined is located.

[0110] Get the side length of the smallest square region and record it as the side length of the target region of the target object to be determined. Get the average value of the pixels in the smallest square region where the target object to be determined is located in the image and record it as the pixel value ζ of the marked region of the target object to be determined in the image.

[0111] For example, the preset object detection models include YOLO and Faster, which are trained using images corresponding to each object in the object database, and each training image is labeled with the object.

[0112] Step S23: Obtain the three-dimensional coordinates of the target object in the image, obtain the historical images of the region where the image is located, obtain the coordinates of the center point of the target object in the historical image based on the three-dimensional coordinates of the target object in the image, construct a square region with the side length of the target region of the target object as the side length and the coordinates of the center point as the center coordinates, obtain the average value of the pixels in the square region in the historical image, and obtain the pixel value of the feature region of the target object in the historical image.

[0113] Step S24: Filter the target objects to be determined. The specific filtering process is as follows: obtain the mean value ζ´ of the pixel values ​​of the feature regions of the target objects to be determined in each historical image. When the absolute value of the pixel value ζ minus the mean value ζ´ of the marked region is greater than the preset absolute value threshold, the target object to be determined is determined to be a target object in the target database, the target object to be determined is retained, and recorded as a target object in the image. Otherwise, the target object to be determined is determined not to be a target object in the target database, the target object to be determined is removed, the coordinates of the center points of each target object in the image are obtained and collected to obtain the target object data in the image.

[0114] Step S3: Obtain target object data from other images captured at the same time as the image capture, obtain spatial epipolar data between camera units of the image and camera units of other images, and obtain candidate matching objects of the target object in other images to obtain target candidate matching data.

[0115] Step S3 includes:

[0116] Step S31: Obtain other images from the multi-view panoramic camera at the same time as the image capture, obtain target object data from the other images, obtain target object data from the image, and obtain the coordinates of the center points of each target object in the image from the target object data;

[0117] For example, the coordinates of the images and other images here are based on a two-dimensional coordinate system.

[0118] Step S32: Obtain the spatial epipolar data between the camera unit of the image and the camera units of other images, obtain the coordinates of the center point of a target object in the image, and obtain the coordinates of the center points of several target objects in other images.

[0119] Step S33: Based on the spatial epipolar data between the camera unit of one image and the camera units of other images, obtain the epipolar line L of the coordinates of the center point of a target object in other images. △ =[γ,η,λ] T Obtain the coordinates (x, y) of the center point of the e-th target object in other images. ▽e ,y ▽ e ), calculate polar line L △ The Euclidean distance d to the center point of the e-th target object (▽,e) :

[0120] ,

[0121] When the Euclidean distance d (▽,e) If the distance is less than the preset distance threshold, the e-th target object is recorded as a candidate matching object for a certain target object; otherwise, the e-th target object is not processed.

[0122] The candidate matching objects of each target object in the image are obtained in other images and aggregated to obtain target candidate matching data between the image and other images;

[0123] Step S4: Obtain target candidate matching data between the image and other images, evaluate the comprehensive cost between the target object in the image and the candidate matching objects in other images, construct the target cost matrix between the image and other images and solve it to obtain the target object matching data;

[0124] Step S4 includes:

[0125] Step S41: Obtain target candidate matching data between the image and other images, obtain candidate matching objects of the target object in the image in other images from the target candidate matching data, obtain the Euclidean distance d´ between the epipolar line of the target object in the image and the center point coordinates of the candidate matching objects in other images, obtain the normalized Euclidean distance d´, and record it as the geometric cost value between the target object in the image and the candidate matching objects in other images.

[0126] For example, the value of the normalized Euclidean distance d´ is restricted to between 0 and 1;

[0127] Step S42: Use a preset object detection model to obtain the bounding boxes of each target object in the image and other images, crop and resize the bounding boxes in the image and other images, and input them into a preset feature extraction model to obtain the feature vectors of each target object in the image and other images, and normalize the feature vectors of each target object in the image and other images.

[0128] For example, the feature extraction model is a pre-defined model, a dedicated re-identification model, which is a model trained using a dataset and trained using metric learning techniques such as triplet loss, and can extract highly discriminative features.

[0129] For example, the feature vector G' of the ξ-th target object in the image ξAfter normalization, the eigenvector G is obtained. ξ， Eigenvector G ξ The specific calculation formula is as follows:

[0130] ,

[0131] Among them, ||G´ ξ ||2 is the feature vector G´ ξ The Euclidean paradigm;

[0132] Step S43: Obtain the feature vector G of the w-th target object in the image and the v-th target object in the other image respectively. w and G ▽ v Calculate the feature value Q between the w-th target object and the v-th target object. (w,v) :

[0133] ,

[0134] Step S44: When the vth target object is not a candidate matching object of the wth target object, the combined cost of the wth target object and the vth target object is set to the preset maximum value.

[0135] When the v-th target object is a candidate matching object of the w-th target object, calculate the combined cost U of the w-th target object and the v-th target object. (w,v) :

[0136] ,

[0137] Among them, D (w,v) Let ψ be the geometrical cost between the v-th target object and the w-th target object; Q ψ is the preset feature cost weight coefficient; D ψ is the preset geometric cost weighting coefficient; Q +ψ D =1, ψ Q >0, ψ D >0;

[0138] Step S45: Obtain the comprehensive cost value between each target object in the image and several target objects in other images, construct the target cost matrix between the image and other images, obtain the total number m of each target object in the image, and obtain the total number n of several target objects in other images;

[0139] When m=n, the target cost matrix is ​​not processed. When m≠n, the target cost matrix is ​​filled in. The specific filling process is as follows:

[0140] When m > n, fill m - n dummy columns in the last column of the target cost matrix, where the comprehensive cost values of all elements in the m - n dummy columns are preset maximum values;

[0141] When m < n, fill n - m dummy rows in the last row of the target cost matrix, where the comprehensive cost values of all elements in the n - m dummy rows are preset maximum values;

[0142] Step S46: Use the Hungarian algorithm to solve the target cost matrix, obtain the zero elements in each row and column of the characteristic target cost matrix, and acquire the matching pairs of the positions corresponding to the zero elements in the characteristic target cost matrix, where one matching pair includes the target object in the image that matches the target object in other images;

[0143] For example, use the Hungarian algorithm to solve the target cost matrix, obtain the zero elements in each row and column of the characteristic target cost matrix. The specific solution process is as follows:

[0144] Set the objective function of the target cost matrix:

[0145] ,

[0146] where U （i,j） is the value of the element corresponding to the i-th row and i-th column of the target cost matrix;

[0147] Set the constraint conditions of the target cost matrix:

[0148] 1. The target object in the image can only match one target object in other images:

[0149] ,

[0150] The target object in other images can only match one target object in the image:

[0151] ,

[0152] X(i, j) is a binary decision variable, and its value is only 0 or 1, indicating whether to assign i in the image to j in other images;

[0153] The specific steps of the Hungarian algorithm are as follows:

[0154] 1. Row subtraction: Subtract the minimum value of each row from each row in the target cost matrix;

[0155] 2. Column subtraction: Subtract the minimum value of each column from each column in the target cost matrix;

[0156] 3. Use the fewest horizontal and vertical lines to cover all zero elements in the target cost matrix;

[0157] 4. Adjust the target cost matrix, find the value of the smallest uncovered element, subtract the value of the smallest element from the uncovered elements, and add the value of the smallest element to all elements in the target cost matrix that are covered by horizontal and vertical lines.

[0158] 5. Repeat steps 3-4 until the objective function of the target cost matrix is ​​satisfied, thereby obtaining the feature target cost matrix;

[0159] 6. Obtain a set of zero elements from the feature target cost matrix, such that each row and each column of the feature target cost matrix has a corresponding zero element;

[0160] The target objects in each image are matched with the target objects in other images, and these matches are aggregated to obtain the target object matching data between the image and other images.

[0161] Step S5: Acquire images taken at the same time, obtain target object matching data between the images, perform image feature fusion to obtain a fused image, obtain the three-dimensional coordinates of the target objects in the fused image and aggregate them to obtain target coordinate data;

[0162] Step S5 includes:

[0163] Step S51: Acquire images from the multi-view panoramic camera that were captured at the same time, acquire target object matching data between the images, and from the target object matching data between one image and another image, acquire the target object that matches the target object in one image in another image.

[0164] Step S52: Obtain the camera parameters of the camera unit corresponding to each image, and combine the target object matching data between the images to perform image feature fusion on each image to obtain a fused image;

[0165] For example, various camera parameters include the intrinsic parameter matrix of the camera unit, distortion coefficients, rotation matrix, and translation vector;

[0166] Step S53: Locate each target object in the fused image to obtain the three-dimensional coordinates of each target object in the fused image. The coordinate system of each three-dimensional coordinate is in the preset world coordinate system. Collect the three-dimensional coordinates of each target object in the fused image to obtain the target coordinate data in the fused image.

[0167] To better implement the above method, a target coordinate localization system based on a multi-view panoramic camera is also proposed. The system includes a camera spatial relationship evaluation module, a target matching module, and a target localization module.

[0168] The camera spatial relationship evaluation module is used to evaluate the spatial geometric relationship between different camera units and obtain the spatial epipolar data between different camera units;

[0169] The target matching module is used to acquire target object data of images and other images, acquire candidate matching objects of the target object in other images based on the target object data, obtain target candidate matching data, construct and solve the target cost matrix between images and other images, and obtain target object matching data between images and other images.

[0170] The target localization module is used to acquire target object matching data between images captured at the same time in a multi-view panoramic camera, perform image feature fusion on each image to obtain a fused image, and acquire the three-dimensional coordinates of the target objects in the fused image to obtain target coordinate data in the fused image.

[0171] The camera spatial relationship evaluation module includes a basic matrix construction unit and a camera spatial relationship evaluation unit.

[0172] The basic matrix construction unit is used to acquire camera parameter data of different camera units in a multi-view panoramic camera and construct the basic matrix between different camera units.

[0173] The camera spatial relationship evaluation unit is used to evaluate the spatial geometric relationship between different camera units based on the fundamental matrix between different camera units, and to obtain the spatial epipolar data between different camera units.

[0174] The target matching module includes a target detection unit, a candidate matching acquisition unit, and a target matching unit.

[0175] The target detection unit is used to detect target objects in the image and filter the undetermined target objects in the image by combining historical images to obtain target object data in the image.

[0176] The candidate matching acquisition unit is used to acquire target object data from other images captured at the same time as the image capture, acquire spatial epipolar data between camera units of the image and camera units of other images, acquire candidate matching objects of the target object in other images, and obtain target candidate matching data.

[0177] The target matching unit is used to construct and solve the target cost matrix between the image and other images based on the target candidate matching data between the image and other images, so as to obtain the target object matching data between the image and other images.

[0178] The target localization module includes an image fusion unit and a target localization unit.

[0179] The image fusion unit is used to acquire images captured at the same time from the multi-view panoramic camera, acquire target object matching data between the images, and perform image feature fusion on the images to obtain a fused image.

[0180] The target localization unit is used to acquire and aggregate the three-dimensional coordinates of the target object in the fused image to obtain the target coordinate data in the fused image.

[0181] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A target coordinate localization method based on a multi-view panoramic camera, characterized in that, The method includes: Step S1: Obtain camera parameter data of different camera units in the multi-view panoramic camera, construct the basic matrix between different camera units, evaluate the spatial geometric relationship between different camera units, and obtain spatial epipolar data; Step S2: Acquire images captured by camera units in a multi-view panoramic camera, acquire historical images of the area where the images are located, acquire the target database in the platform, perform target object detection on the images, obtain the target objects to be determined, and filter the target objects to obtain target object data. Step S3: Obtain target object data from other images captured at the same time as the image; obtain spatial epipolar data between the camera unit of the image and the camera units of other images; and obtain candidate matching objects of the target object in other images to obtain target candidate matching data. Step S4: Obtain target candidate matching data between the image and other images, evaluate the comprehensive cost between the target object in the image and the candidate matching objects in other images, construct the target cost matrix between the image and other images and solve it to obtain the target object matching data; Step S5: Acquire images taken at the same time, obtain target object matching data between the images, perform image feature fusion to obtain a fused image, obtain the three-dimensional coordinates of the target objects in the fused image and aggregate them to obtain target coordinate data; Step S3 includes: Step S31: Obtain other images from the multi-view panoramic camera that were captured at the same time as the image, obtain target object data from the other images, obtain target object data from the image, and obtain the coordinates of the center points of each target object in the image from the target object data; Step S32: Obtain the spatial epipolar data between the camera unit of the image and the camera units of other images, obtain the coordinates of the center point of a target object in the image, and obtain the coordinates of the center points of several target objects in other images. Step S33: Based on the spatial epipolar data between the camera unit of the image and the camera units of other images, obtain the epipolar line L of the coordinates of the center point of a target object in other images. △ =[γ,η,λ] T Obtain the coordinates (x, y) of the center point of the e-th target object in other images. ▽ e ,y ▽ e ), calculate polar line L △ The Euclidean distance d to the center point of the e-th target object (▽,e) : ， When the Euclidean distance d (▽,e) If the distance is less than a preset distance threshold, the e-th target object is recorded as a candidate matching object for a certain target object; otherwise, the e-th target object is not processed. Candidate matching objects of each target object in the image are obtained in other images and aggregated to obtain target candidate matching data between the image and other images; Step S4 includes: Step S41: Obtain target candidate matching data of the image and other images, obtain candidate matching objects of the target object in the image in other images from the target candidate matching data, obtain the Euclidean distance d´ between the epipolar line of the target object in the image and the center point coordinates of the candidate matching objects in other images, obtain the normalized Euclidean distance d´, and record it as the geometric cost value between the target object in the image and the candidate matching objects in other images. Step S42: Use a preset object detection model to obtain the bounding boxes of each target object in the said image and other images, crop and resize the bounding boxes in the said image and other images, and input them into a preset feature extraction model to obtain the feature vectors of each target object in the said image and other images, and perform normalization processing on the feature vectors of each target object in the said image and other images; Step S43: Obtain the feature vector G of the w-th target object in the image and the v-th target object in the other image respectively. w and G ▽ v Calculate the feature value Q between the w-th target object and the v-th target object. (w,v) : ， Step S44: When the v-th target object is not a candidate matching object of the w-th target object, set the comprehensive cost value of the w-th target object and the v-th target object to a preset maximum value; When the v-th target object is a candidate matching object of the w-th target object, calculate the combined cost U of the w-th target object and the v-th target object. (w,v) : ， Among them, D (w,v) Let ψ be the geometrical cost between the v-th target object and the w-th target object; Q ψ is the preset feature cost weight coefficient; D ψ is the preset geometric cost weighting coefficient; Q +ψ D =1, ψ Q >0, ψ D >0; Step S45: Obtain the comprehensive cost values between each target object in the said image and several target objects in other images, construct a target cost matrix between the said image and other images, obtain the total number m of each target object in the said image, and obtain the total number n of several target objects in other images; When m = n, do not process the target cost matrix. When m ≠ n, fill the target cost matrix. The specific filling process is as follows: When m > n, fill m - n virtual columns in the last column of the target cost matrix, where the comprehensive cost values of each element in the m - n virtual columns are all preset maximum values; When m < n, fill n - m virtual rows in the last row of the target cost matrix, where the comprehensive cost values of each element in the n - m virtual columns are all preset maximum values; Step S46: Use the Hungarian algorithm to solve the target cost matrix, obtain the zero elements in each row and column of the feature target cost matrix, and obtain the matching pairs of the positions corresponding to the zero elements in the feature target cost matrix. Among them, a matching pair includes the target object in the said image that matches the target object in other images; Obtain the target objects in other images that match each target object in the said image, and perform aggregation to obtain the target object matching data between the said image and other images.

2. The target coordinate localization method based on a multi-view panoramic camera according to claim 1, characterized in that, The said Step S1 includes: Step S11: Obtain the camera parameter data of each camera unit in the multi-view panoramic camera. Among them, the camera parameter data includes the principal point coordinates and the values corresponding to the focal lengths of the camera units in the multi-view panoramic camera; According to the camera parameter data, construct the internal parameter matrix K of the camera unit, and construct the internal parameter matrix of each camera unit; Step S12: Construct the fundamental matrix between each camera unit. Among them, the process of constructing the fundamental matrix between the a-th camera unit and the b-th camera unit is as follows: Obtain the three-dimensional coordinates of physical point P in the coordinate systems of the a-th and b-th camera units respectively. a and P b ; The rotation matrix R and translation vector t between the a-th camera unit and the b-th camera unit are calculated using the following formula: P a =P b ·R+t; Obtain the antisymmetric matrix t´ of the translation vector t, and calculate the fundamental matrix F between the a-th camera unit and the b-th camera unit. (a,b) =K b -T ·R·K a -1 , where K b -T K is the transpose of the inverse of the intrinsic parameter matrix K of the b-th camera unit; a -1 K is the intrinsic parameter matrix of the a-th camera unit. a The inverse matrix; Step S13: Evaluate the spatial geometric relationship between the a-th camera unit and the b-th camera unit. The specific evaluation process is as follows: Obtain the coordinates (x, y) of image point α in the a-th camera unit. a α ,y a α ), obtain the homogeneous coordinates α´ of image point α. a =[x a α ,y a α ,1] T Calculate the epipolar line L of image point α in the b-th camera unit. b α =F (a,b) ·α´ a ; Obtain the coordinates (x, y) of image point β in the b-th camera unit. b β ,y b β ), obtain the homogeneous coordinates β´ of image point β. b =[x b β ,y b β ,1] T Calculate the epipolar line L of image point β in the a-th camera unit. a β =F (a,b) T ·β´ b , of which F (a,b) T Basic matrix F (a,b) transpose; Obtain the epipolar lines of each image point of the a-th camera unit in the b-th camera unit, obtain the epipolar lines of each image point of the b-th camera unit in the a-th camera unit, and perform aggregation to obtain the spatial epipolar line data between the a-th camera unit and the b-th camera unit; Step S14: Evaluate the spatial geometric relationship between different camera units to obtain the spatial epipolar line data between different camera units.

3. The target coordinate localization method based on a multi-view panoramic camera according to claim 2, characterized in that, The said Step S2 includes: Step S21: Obtain images captured by the camera unit in the multi-view panoramic camera within the current period, obtain historical images within the area where the images are located, construct a two-dimensional coordinate system for the images, preprocess the images, and obtain the pixel values ​​of each pixel in the images; Step S22: Obtain the preset target database in the platform, wherein the target database includes the image set corresponding to each preset target object; Based on the target database and using a preset target detection model, the target objects in the image are detected to obtain object region data for each undetermined target object in the image. The object region data is the smallest square region in the image where the undetermined target object is located. Obtain the side length of the smallest square region and record it as the side length of the target region of the target object to be determined. Obtain the average value of the pixels in the smallest square region where the target object to be determined is located in the image and record it as the pixel value ζ of the marked region of the target object to be determined in the image. Step S23: Obtain the three-dimensional coordinates of the target object to be determined in the image, obtain each historical image of the region where the image is located, obtain the coordinates of the center point of the target object to be determined in the historical image based on the three-dimensional coordinates of the target object to be determined in the image, construct a square region with the side length of the target region of the target object to be determined as the side length and the coordinates of the center point as the center coordinates, obtain the average value of the pixel points in the square region in the historical image, and obtain the feature region pixel value of the target object to be determined in the historical image. Step S24: Filter the target objects to be determined. The specific filtering process is as follows: obtain the mean value ζ´ of the pixel values ​​of the feature regions of the target objects to be determined in each historical image. When the absolute value of the pixel value ζ minus the mean value ζ´ is greater than the preset absolute value threshold, the target object to be determined is determined to be a target object in the target database, and the target object to be determined is retained and recorded as a target object in the image. Otherwise, the target object to be determined is determined not to be a target object in the target database, and the target object to be determined is removed. Obtain the coordinates of the center points of each target object in the image and collect them to obtain the target object data in the image.

4. The target coordinate localization method based on a multi-view panoramic camera according to claim 3, characterized in that, Step S5 includes: Step S51: Acquire images from the multi-view panoramic camera that were captured at the same time, acquire target object matching data between the images, and from the target object matching data between one image and another image, acquire the target object that matches the target object in one image in another image. Step S52: Obtain the camera parameters of the camera unit corresponding to each image, and combine the target object matching data between the images to perform image feature fusion on each image to obtain a fused image; Step S53: Locate each target object in the fused image to obtain the three-dimensional coordinates of each target object in the fused image. The coordinate system of each three-dimensional coordinate is in the preset world coordinate system. Collect the three-dimensional coordinates of each target object in the fused image to obtain the target coordinate data in the fused image.

5. A target coordinate positioning system based on a multi-view panoramic camera, used to execute the target coordinate positioning method based on a multi-view panoramic camera as described in any one of claims 1-4, characterized in that, The system includes a camera spatial relationship evaluation module, a target matching module, and a target localization module; The camera spatial relationship evaluation module is used to evaluate the spatial geometric relationship between different camera units and obtain spatial epipolar data between different camera units. The target matching module is used to acquire target object data of the image and other images, acquire candidate matching objects of the target object in other images based on the target object data, obtain target candidate matching data, construct and solve the target cost matrix between the image and other images, and obtain target object matching data between the image and other images. The target localization module is used to acquire target object matching data between images captured at the same time in a multi-view panoramic camera, perform image feature fusion on the images to obtain a fused image, and acquire the three-dimensional coordinates of the target objects in the fused image to obtain target coordinate data in the fused image.

6. A target coordinate positioning system based on a multi-view panoramic camera according to claim 5, characterized in that, The camera spatial relationship evaluation module includes a basic matrix construction unit and a camera spatial relationship evaluation unit; The basic matrix construction unit is used to acquire camera parameter data of different camera units in a multi-view panoramic camera and construct a basic matrix between different camera units. The camera spatial relationship evaluation unit is used to evaluate the spatial geometric relationship between different camera units based on the fundamental matrix between different camera units, and to obtain the spatial epipolar data between different camera units.

7. A target coordinate positioning system based on a multi-view panoramic camera according to claim 5, characterized in that, The target matching module includes a target detection unit, a candidate matching acquisition unit, and a target matching unit; The target detection unit is used to detect target objects in the image and filter the undetermined target objects in the image by combining historical images to obtain target object data in the image. The candidate matching acquisition unit is used to acquire target object data of other images at the same time as the image was captured, acquire spatial epipolar data between the camera unit of the image and the camera units of other images, acquire candidate matching objects of the target object of the image in other images, and obtain target candidate matching data. The target matching unit is used to construct and solve the target cost matrix between the image and other images based on the target candidate matching data between the image and other images, so as to obtain the target object matching data between the image and other images.

8. A target coordinate positioning system based on a multi-view panoramic camera according to claim 5, characterized in that, The target localization module includes an image fusion unit and a target localization unit; The image fusion unit is used to acquire images captured at the same time from the multi-view panoramic camera, acquire target object matching data between the images, and perform image feature fusion on the images to obtain a fused image. The target localization unit is used to acquire and collect the three-dimensional coordinates of the target object in the fused image to obtain target coordinate data in the fused image.

Images