Target identification method based on multiple cameras

Abstract

The invention discloses a multi-camera-based target identification method, a plurality of cameras are arranged at fixed positions around a target, the acquisition range of the plurality of cameras covers the target, and the method comprises the following steps: obtaining a spatial transformation matrix among the plurality of cameras at the fixed positions based on a checkerboard calibration method; synchronously acquiring corresponding local images from different visual angles through a plurality of cameras; wherein a certain overlapping area exists between adjacent camera view fields; performing target detection and identification on the local image acquired by each camera to obtain a bounding box of the target and obtain information representing the bounding box; based on the spatial transformation matrix between the cameras, perspective transformation is carried out on the multi-view local images, spatial alignment and geometric correction of the multi-view local images are achieved, and then a seamless overall spliced image is generated from the multiple local images through an image fusion and splicing algorithm. According to the method, multiple cameras are adopted for parallel acquisition and target recognition, and the target recognition speed and efficiency are improved.

Description

Technical Field

[0001] This invention relates to a target recognition method based on multiple cameras, belonging to the field of object detection and recognition technology. Background Technology

[0002] Currently, shelving systems are used in retail stores or factory warehouses. These systems have multiple aisles, and cameras are placed at the top of the aisles to detect and identify items (i.e., targets) on the aisles. Patent CN110705666A discloses an AI-powered cloud computing method for monitoring and storing goods on display shelves and their labels. It describes using an industrial digital camera mounted on a fixed platform, a mobile platform, or a mobile robot. The captured images are transmitted from the industrial digital camera to AI neural network software for analysis. This software identifies the positions of multiple target images, the categories and quantities of goods in the images, and performs barcode, QR code, or optical character recognition to identify information from product labels and the location labels of the storage areas. However, this patent does not specifically disclose whether it uses a single camera or multiple cameras to collect and identify targets. There is an urgent need for a multi-camera-based target recognition method to effectively identify and detect items on the aisles. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a target recognition method based on multiple cameras. The method uses multiple cameras to acquire and recognize targets in parallel, thereby improving the speed and efficiency of target recognition.

[0004] To solve the above-mentioned technical problems, the technical solution of the present invention is: a target recognition method based on multiple cameras, wherein multiple cameras are placed at fixed positions around the target, and the acquisition range of the multiple cameras covers the target, and the method includes the following steps: S01: Obtain the spatial transformation matrix between multiple cameras at fixed positions based on the checkerboard calibration method; S02: Simultaneously acquire corresponding local images from different perspectives using multiple cameras; where there is a certain overlap between the fields of view of adjacent cameras; S03: Perform target detection and recognition on the local images acquired by each camera to obtain the bounding box of the target and acquire information representing the bounding box; S04: Based on the spatial transformation matrix between cameras, perspective transformation is performed on local images from multiple perspectives to achieve spatial alignment and geometric correction of local images from multiple perspectives. Then, multiple local images are used to generate a seamless overall stitched image through image fusion and stitching algorithms. On the basis of stitching, the overlapping areas between adjacent local images are processed. The detection boxes under different camera perspectives are uniformly mapped to the global stitched image coordinate system or a certain reference camera coordinate system. The degree of overlap is determined by calculating the intersection-union ratio or Euclidean distance between the bounding boxes. The bounding boxes with an overlap rate higher than a set threshold are clustered and fused to retain the detection results with unique representation. S05: Construct a mapping model from the local camera coordinate system to the global stitched image coordinate system or a certain reference camera coordinate system. Based on the mapping model, calculate the unified position coordinates of the target in the overall stitched image through coordinate synthesis to achieve cross-camera target localization.

[0005] Furthermore, in step S01, a checkerboard calibration board is used for calibration. Multiple cameras acquire images of the checkerboard calibration board in multiple poses. Based on the checkerboard calibration board images, the rigid body transformation relationship between the camera coordinate systems of each pair of adjacent cameras is calculated, and finally a spatial transformation matrix is ​​formed, thereby establishing the mapping relationship between the camera coordinate systems.

[0006] Furthermore, the spatial transformation matrix includes a rotation matrix and a translation vector.

[0007] Furthermore, to ensure the accuracy of subsequent local image stitching and fusion, in step S02, the local images acquired by each camera have the same resolution and orientation.

[0008] Furthermore, in order to provide sufficient matching features for image alignment and coordinate mapping, in step S02, there is a certain overlap area between the fields of view of adjacent cameras, and the overlap rate of the overlap area is not less than 20%.

[0009] Furthermore, in order to better detect and identify targets, in step S03, the YOLOv8-OBB model based on deep learning performs target detection and identification on the local images captured by each camera.

[0010] Furthermore, in step S03, the information characterizing the bounding box includes the coordinates of the center point of the bounding box in the image coordinate system, the width and height of the bounding box, and the rotation angle of the bounding box.

[0011] Furthermore, in step S04, the image coordinate system of a local image is selected as the reference image coordinate system. Using the spatial transformation matrix, the remaining local images are subjected to perspective transformation and reprojected onto the reference image coordinate system to complete the preliminary geometric alignment of the local images from multiple perspectives.

[0012] Furthermore, bilinear interpolation is used during the alignment process to maintain image smoothness; and / or multi-band fusion is used in the image fusion stage to eliminate brightness differences and stitching gaps.

[0013] Furthermore, while performing step S03, the following steps are also performed: real-time identification of the identification code of the target in the local image, and extraction of the geometric center and orientation information of the identification code; while performing step S05, the following steps are also performed: the extracted identification code information is transformed into the global stitched image coordinate system according to the mapping model.

[0014] After adopting the above technical solution, the method of the present invention can identify targets by acquiring data in parallel with multiple cameras, which improves the speed and efficiency of target identification, and is especially suitable for identifying target goods on long cargo lanes. Detailed Implementation

[0015] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments.

[0016] A target recognition method based on multiple cameras, wherein multiple cameras are placed at fixed positions around the target. In this embodiment, the multiple cameras are fixedly positioned above the target and arranged side by side, with the acquisition range of the multiple cameras covering the target. The method includes the following steps: S01: Obtain the spatial transformation matrix between multiple cameras at fixed positions based on the checkerboard calibration method; S02: Simultaneously acquire corresponding local images from different perspectives using multiple cameras; where there is a certain overlap between the fields of view of adjacent cameras; S03: Perform target detection and recognition on the local images acquired by each camera to obtain the bounding box of the target and acquire information representing the bounding box; S04: Based on the spatial transformation matrix between cameras, perspective transformation is performed on local images from multiple perspectives to achieve spatial alignment and geometric correction of local images from multiple perspectives. Then, multiple local images are used to generate a seamless overall stitched image through image fusion and stitching algorithms. On the basis of stitching, the overlapping areas between adjacent local images are processed. The detection boxes under different camera perspectives are uniformly mapped to the global stitched image coordinate system or a certain reference camera coordinate system. The degree of overlap is determined by calculating the intersection-union ratio or Euclidean distance between the bounding boxes. The bounding boxes with an overlap rate higher than a set threshold are clustered and fused to retain the detection results with unique representation. S05: Construct a mapping model from the local camera coordinate system to the global stitched image coordinate system or a specific reference camera coordinate system. Based on this mapping model, calculate the unified position coordinates of the target in the overall stitched image through coordinate synthesis, achieving cross-camera target localization. Specifically, when a target is detected in the image of a certain camera, the system calls the pre-defined mapping relationship (spatial transformation matrix) between that camera and the global coordinate system to mathematically transform the target's local image coordinates (such as the center point of the bounding box). Through this "coordinate synthesis" calculation, the unified position of the target in the overall stitched image or the world coordinate system is finally determined.

[0017] Specifically, in step S05, let the homogeneous coordinates of a point under camera C_i be p_i=[u,v,1]^T, which corresponds to the point P_w=[X,Y,Z]^T in the world coordinate system. The mapping relationship is given by the projection matrix M_i: p_i≃K_i[R_i|t_i]P_w, where K_i is the camera intrinsic parameter matrix and [R_i|t_i] is the extrinsic parameter matrix. The coordinates of the common observation point of multiple cameras are optimized by bundle adjustment to further improve global consistency.

[0018] Specifically, in step S01, a checkerboard calibration board (a 15mm×15mm grid with a 7×5 layout) is used for calibration. Multiple cameras acquire images of the checkerboard calibration board in multiple poses. Based on the checkerboard calibration board images, the rigid body transformation relationship between the camera coordinate systems of each pair of adjacent cameras is calculated, and finally a spatial transformation matrix is ​​formed, thereby establishing the mapping relationship between the camera coordinate systems.

[0019] Specifically, in this embodiment, the spatial transformation matrix includes a rotation matrix R and a translation vector t.

[0020] Specifically, let the coordinates of a point be P in one camera coordinate system and P′ in another camera coordinate system. Then the transformation relationship is: P′=RP+t. This transformation will be used for the coordinate unification and fusion of subsequent multi-view bounding boxes and will be fixed in the device as the initial parameters of the system.

[0021] Specifically, in order to ensure the accuracy of subsequent local image stitching and fusion, in step S02, the local images acquired by each camera have the same resolution and orientation.

[0022] Specifically, in order to provide sufficient matching features for image alignment and coordinate mapping, in step S02, there is a certain overlap area between the fields of view of adjacent cameras, and the overlap rate of the overlap area is not less than 20%.

[0023] Specifically, in order to effectively detect and identify targets, in step S03, the YOLOv8-OBB model based on deep learning performs target detection and identification on the local images captured by each camera.

[0024] Specifically, in step S03, the information representing the bounding box includes the coordinates of the bounding box center point in the image coordinate system, the width and height of the bounding box, and the rotation angle of the bounding box. Specifically, it is represented as: B_i = (x_c, y_c, w, h, θ), where (x_c, y_c) represents the coordinates of the bounding box center point in the image coordinate system, w and h are the width and height of the bounding box, respectively, and θ is the rotation angle of the bounding box (usually defined as the angle of counterclockwise rotation from the horizontal axis to the first edge of the bounding box). This representation method is more suitable for objects with obvious directionality and can reduce alignment errors during subsequent coordinate mapping.

[0025] In this embodiment, in step S04, retaining the detection result with unique representation specifically means retaining the detection result with the highest confidence or whose geometric center is most representative.

[0026] Specifically, in step S04, the image coordinate system of a local image is selected as the reference image coordinate system. Using the spatial transformation matrix, the remaining local images are subjected to perspective transformation and reprojected onto the reference image coordinate system to complete the preliminary geometric alignment of the local images from multiple perspectives.

[0027] Specifically, bilinear interpolation is used during the alignment process to maintain image smoothness; multi-band fusion is used in the image fusion stage to eliminate brightness differences and stitching gaps.

[0028] Specifically, while performing step S03, the following steps are also performed: The ZBar algorithm is used to identify the identifier code of the target in the local image in real time, extracting the geometric center and orientation information of the identifier code. This information is based on the coordinate system of a single local image and is marked in the single local image along with the target identification. If multiple local images need to be stitched together, while performing step S05, the following steps are also performed: The extracted identifier code information is transformed to the global stitched image coordinate system according to a mapping model. Simultaneously, the system also compares the identified identifier code information with the item information pre-stored in the database to ultimately determine the uniqueness of the item. If no identifier code is identified, it will be highlighted on the icon.

[0029] Specifically, in this embodiment, the identification code includes a QR code and / or a barcode.

[0030] The above specific embodiments further illustrate the technical problems solved by the present invention, the technical solutions, and the beneficial effects. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-camera based target recognition method, a plurality of cameras are placed at fixed positions around a target, and the capturing ranges of the plurality of cameras cover the target, characterized in that: The steps of the method comprise: S01: obtaining a spatial transformation matrix between cameras at multiple fixed positions based on a chessboard calibration method; S02: synchronously collecting corresponding local images from different perspectives by multiple cameras; wherein there is a certain overlapping area between the fields of view of adjacent cameras; S03: performing target detection and recognition on the local images collected by each camera to obtain the bounding box of the target and information representing the bounding box; S04: based on the spatial transformation matrix between the cameras, performing perspective transformation on the multi-perspective local images to realize spatial alignment and geometric correction of the multi-perspective local images, and then generating a seamless overall stitched image through image fusion and stitching algorithm; based on the stitching, processing the direct overlapping area between adjacent local images, mapping the detection boxes under different camera perspectives to the global stitched image coordinate system or a certain reference camera coordinate system, judging the overlapping degree by calculating the intersection over union or the Euclidean distance between the bounding boxes, and clustering and fusing the bounding boxes with an overlapping rate higher than a certain threshold to retain the unique detection result; S05: constructing a mapping model from the local camera coordinate system to the global stitched image coordinate system or a certain reference camera coordinate system, and calculating the unified position coordinates of the target in the overall stitched image based on the mapping model through coordinate synthesis to realize cross-camera target positioning.

2. The method of claim 1, wherein, In step S01, a chessboard calibration board is used for calibration, multiple camera pose chessboard calibration board images are collected by multiple cameras, the rigid transformation relationship between the camera coordinate systems of each pair of adjacent cameras is calculated based on the chessboard calibration board images, and finally a spatial transformation matrix is formed to establish the mapping relationship between the camera coordinate systems.

3. The method according to claim 1 or 2, characterized in that, The spatial transformation matrix includes a rotation matrix and a translation vector.

4. The method of claim 1, wherein, In step S02, the local images collected by each camera have the same resolution and direction.

5. The method of claim 1, wherein, In step S02, the overlapping rate of the overlapping area is not less than 20%.

6. The method of claim 1, wherein, In step S03, a YOLOv8-OBB model based on deep learning is used to perform target detection and recognition on the local images collected by each camera.

7. The method of claim 1, wherein, In step S03, the information representing the bounding box includes the coordinates of the center point of the bounding box in the image coordinate system, the width and height of the bounding box, and the rotation angle of the bounding box.

8. The method of claim 1, wherein, In step S04, an image coordinate system of one local image is selected as a reference image coordinate system, and the spatial transformation matrix is used to perform perspective transformation on the remaining local images to project them all to the reference image coordinate system, completing the preliminary geometric alignment of the multi-perspective local images.

9. The method of claim 1, wherein, During the alignment process, bilinear interpolation is used to maintain image smoothness; and / or multi-band fusion is used in the image fusion stage to eliminate brightness differences and stitching gaps.

10. The method of claim 1, wherein, While step S03 is being performed, the step of real-time recognizing the identification code of the target in the local image and extracting the geometric center and direction information of the identification code is also being performed; while step S05 is being performed, the step of converting the extracted information of the identification code to the global stitched image coordinate system according to the mapping model is also being performed.