A cross-camera multi-target tracking method, system and electronic device

Abstract

The application discloses a cross-camera multi-target tracking method, system and electronic equipment. The method comprises the following steps: acquiring a first detection box corresponding to each target object in a current frame image captured by each camera; merging the first detection boxes containing the same target object to obtain H second detection boxes, and taking the second detection boxes and the first detection boxes which are not merged as candidate boxes in a candidate box set; finding a target candidate box containing the target object in all frame images before the current frame image to obtain a detection position of the target object corresponding to each target candidate box in the current frame image; updating a current position of the target object based on each detection position, and calculating a predicted position of the target object in a next frame image. Through the technical scheme provided in the embodiment of the application, the predicted positions of the target objects in the next frame image in the current frame images captured by different cameras are avoided to be predicted respectively, and the prediction accuracy of the positions of the target objects in the next frame image is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target tracking, in particular to a cross-camera multi-target tracking method, system and electronic device. BACKGROUND

[0002] With the development of science and technology, target tracking technology has also developed rapidly. Through target tracking technology, the position of a target object in the next frame of image can be predicted, and based on the position of the target object in each frame of image, the motion trajectory of the target object can be generated, which has been widely used in smart city, intelligent driving, robot and other fields.

[0003] In the target tracking technology, according to the number of target objects, the target tracking technology can be divided into single target tracking and multi-target tracking. And multi-target tracking can be divided into single-camera multi-target tracking and cross-camera multi-target tracking. However, due to the limited field of view of single-camera, occlusion and other problems, single-camera multi-target tracking has great limitations, for example, it cannot continuously track the target object, so currently cross-camera multi-target tracking is generally used to predict the position of each target object in the next frame of image.

[0004] In cross-camera multi-target tracking, referring to the schematic diagram of camera and target object as shown in Figure 1 , there is an angle overlap area between the first camera and the second camera installed on the vehicle, and the angle of view of the first camera and the angle of view of the second camera can both shoot the target object, i.e. A object, and then predict the position of A object in the next frame of image.

[0005] Firstly, the first current frame of image shot by the first camera is input into a target detection module to obtain a first detection box corresponding to A object, wherein the identity code (Identity Document, ID) of A object corresponding to the first detection box is A1, and then the first predicted position of A1 in the next frame of image is calculated based on the first detection box. Similarly, for the second current frame of image shot by the second camera, a second detection box corresponding to A object is obtained, and the ID of A object corresponding to the second detection box is A2, and then the second predicted position of A2 in the next frame of image is calculated.

[0006] Then, based on the similarity between the first detection box and the second detection box, it is determined whether A1 and A2 are the same A object, and after it is determined that A1 and A2 are the same A object, the first predicted position and the second predicted position are fused to obtain the predicted position of A object in the next frame of image.

[0007] However, when the first predicted position and the second predicted position are fused to obtain the predicted position of the A object in the next frame image, since the shooting angles of the first camera and the second camera to the A object are different, there is a large difference between the first detection box and the second detection box, so that the first predicted position and the second predicted position obtained based on the first detection box and the second detection box also have a large difference, and further, when the first predicted position and the second predicted position are fused to obtain the predicted position of the A object in the next frame image, the prediction accuracy of the position of the A object in the next frame image is not high, that is, the same target object in the current frame image photographed by different cameras at different angles has a large difference in the predicted position in the next frame image due to different angles of view, and further, when different predicted positions are fused to obtain the predicted position of the target object in the next frame image, the prediction accuracy of the position of the target object in the next frame image is not high. SUMMARY

[0008] The present application provides a cross-camera multi-target tracking method, system and electronic device to solve the problem that the same target object in the current frame image photographed by different cameras at different angles has a large difference in the predicted position in the next frame image due to different angles of view, and further, when different predicted positions are fused to obtain the predicted position of the target object in the next frame image, the prediction accuracy of the position of the target object in the next frame image is not high. The specific implementation scheme is as follows:

[0009] In a first aspect, the present application provides a cross-camera multi-target tracking method, which comprises:

[0010] Obtaining a first detection box corresponding to each target object in each current frame image photographed by each camera respectively;

[0011] For each first detection box, merging the first detection boxes containing the same target object to obtain H second detection boxes, and taking the H second detection boxes and the first detection boxes not merged as candidate boxes in a candidate box set, wherein H is an integer greater than zero;

[0012] In the candidate box set, finding a target candidate box containing the target object in all frame images before the current frame image, and taking the position of each target candidate box as the detection position of the target object corresponding to each target candidate box in the current frame image respectively;

[0013] Based on the detection position of the target object corresponding to each target candidate box, updating the current position of the target object corresponding to each target candidate box in the current frame image, and calculating the predicted position of the target object corresponding to each target candidate box in the next frame image.

[0014] In this way, the positions of the target objects in the next frame image are predicted by avoiding predicting the positions of the target objects in the next frame image in the current frame images captured by different cameras respectively, and then the prediction accuracy of the positions of the target objects in the next frame image is improved.

[0015] In a possible implementation, the obtaining of the first detection frame corresponding to each target object in each current frame image captured by each camera comprises:

[0016] obtaining each current frame image captured by each camera;

[0017] inputting each current frame image into the target detection module respectively, and outputting the first detection frame corresponding to each target object in each current frame image respectively.

[0018] In this way, the first detection frame corresponding to each target object in each current frame image captured by each camera is obtained based on the target detection module.

[0019] In a possible implementation, the merging of the first detection frames containing the same target object to obtain the H second detection frames, and the H second detection frames and the first detection frames not merged as the candidate frames in the candidate frame set comprises:

[0020] obtaining the feature vector corresponding to each first detection frame;

[0021] calculating the feature distance between each first detection frame based on the feature vector corresponding to each first detection frame;

[0022] converting the first coordinate corresponding to each first detection frame into the second coordinate in the same coordinate system to obtain the second coordinate corresponding to each first detection frame;

[0023] for each first detection frame, screening the first detection frames containing the same target object based on the feature distance between each first detection frame, the second coordinate corresponding to each first detection frame, and the size corresponding to each first detection frame;

[0024] merging the first detection frames containing the same target object to obtain the H second detection frames, and the H second detection frames and the first detection frames not merged as the candidate frames in the candidate frame set.

[0025] In the above manner, the feature distance between each first detection box is obtained based on the respective feature vector corresponding to each first detection box. Then, the first detection boxes containing the same target object are determined based on the feature distance between each first detection box, the coordinates of each first detection box in the same coordinate system, and the respective size of each first detection box, thereby improving the matching accuracy of the target objects between different first detection boxes. In addition, the candidate boxes in the candidate box set are obtained by merging the first detection boxes containing the same target object, thereby providing the candidate boxes for subsequent matching operations.

[0026] In a possible implementation, the searching, in the candidate box set, for the target candidate boxes containing the target objects in all the frame images before the current frame image, and taking the respective position of each target candidate box as the detection position of the target object corresponding to each target candidate box in the current frame image, comprises:

[0027] The first matching operation is sequentially performed on each candidate box in the candidate box set.

[0028] A candidate box in the candidate box set is taken as a matching candidate box.

[0029] The second matching operation is sequentially performed on each target object in all the frame images before the current frame image through the matching candidate box, to determine whether the matching candidate box contains the target objects in all the frame images before the current frame image.

[0030] If not, the target object in the matching candidate box is taken as a new target object.

[0031] If yes, the matching candidate box is taken as the target candidate box, and the position of the target candidate box is taken as the detection position of the target object corresponding to the target candidate box in the current frame image.

[0032] The first matching operation is sequentially performed on each candidate box in the candidate box set.

[0033] In the above manner, the target candidate boxes containing the target objects in all the frame images before the current frame image are searched in the candidate box set based on the first matching operation sequentially performed on each candidate box in the candidate box set, thereby obtaining the detection position of the target object corresponding to each target candidate box in the current frame image after searching for all the target candidate boxes containing the target objects in all the frame images before the current frame image in the candidate box set, and providing the detection position for updating the current position of the target object in the current frame image.

[0034] In a possible implementation, the second matching operation comprises:

[0035] taking one target object in all frame images before the current frame image as a matching object;

[0036] obtaining a third detection box corresponding to the matching object, a first image feature corresponding to the matching object, and a second image feature corresponding to a target object in the matching candidate box;

[0037] calculating an image matching score between the second image feature and the first image feature and a box matching score between the matching candidate box and the third detection box;

[0038] determining whether the image matching score is greater than an image preset threshold and the box matching score is greater than a box preset threshold simultaneously;

[0039] if not, confirming that the matching candidate box does not contain the matching object;

[0040] if yes, confirming that the matching candidate box contains the matching object.

[0041] In the above manner, after calculating the image matching score between the first image feature and the second image feature and calculating the box matching score between the matching candidate box and the third detection box, it is determined whether the matching candidate box contains the matching object based on the judgment of the image matching score and the image preset threshold and the judgment of the box matching score and the box preset threshold, and the matching precision between the target object in the current frame image and the target object in all frame images before the current frame image is further improved.

[0042] In a possible implementation, the updating of the current position of the target object corresponding to each target candidate box in the current frame image based on the detection position of the target object corresponding to each target candidate box, and the calculation of the predicted position of the target object corresponding to each target candidate box in the next frame image comprise:

[0043] obtaining a last frame predicted position of the target object corresponding to each target candidate box, wherein the last frame predicted position is a position of the target object corresponding to each target candidate box in the current frame image predicted in a last frame image;

[0044] updating the current position of the target object corresponding to each target candidate box in the current frame image based on the detection position of the target object corresponding to each target candidate box and the last frame predicted position;

[0045] According to the current position corresponding to each target object corresponding to each target candidate frame, the current position of each target object corresponding to each target candidate frame in the next frame image is calculated.

[0046] In the above manner, the current position of each target object corresponding to each target candidate frame in the current frame image is updated based on the detection position corresponding to each target object corresponding to each target candidate frame and the previous frame prediction position, so that the prediction position of each target object corresponding to each target candidate frame in the next frame image is calculated according to the current position, and the prediction accuracy of the prediction position of the target object in the next frame image is further improved.

[0047] In a possible implementation, the calculation of the prediction position of each target object corresponding to each target candidate frame in the next frame image according to the current position corresponding to each target object corresponding to each target candidate frame comprises:

[0048] According to the current position corresponding to each target object corresponding to each target candidate frame, it is determined that there is a pre-cross-view-angle target object in each target object corresponding to each target candidate frame, wherein the pre-cross-view-angle target object is a target object that will enter the view angle of the second camera from the view angle of the first camera;

[0049] For each of the pre-cross-view-angle target objects, the following prediction operation is performed:

[0050] The first prediction position of the pre-cross-view-angle target object in the next frame image captured by the first camera is calculated;

[0051] It is judged whether the pre-cross-view-angle target object is in the overlapping area of the view angles of the first camera and the second camera;

[0052] If yes, the first prediction position is converted into a position in the view angle of the second camera to obtain the prediction position of the pre-cross-view-angle target object in the next frame image;

[0053] If no, a fifth detection frame exceeding the view angle of the first camera is intercepted from the fourth detection frame corresponding to the first prediction position, a second prediction position in the view angle of the second camera is converted from the position corresponding to the fifth detection frame to obtain the prediction position of the pre-cross-view-angle target object in the next frame image;

[0054] Until each of the pre-cross-view-angle target objects performs the prediction operation to obtain the prediction position of the pre-cross-view-angle target object in the next frame image, and the prediction position of the non-pre-cross-view-angle target object in each target object corresponding to each target candidate frame in the next frame image is calculated.

[0055] In this way, whether there is a pre-cross-view-angle target object in each target object corresponding to each target candidate box is determined according to the current position of the target object corresponding to each target candidate box; then, when there is a pre-cross-view-angle target object, the specific calculation manner of the predicted position of the pre-cross-view-angle target object in the second camera is determined based on the judgment of whether the pre-cross-view-angle target object is in the view angle overlap region, and then the predicted position of the pre-cross-view-angle target object in all next frame images is obtained; in addition, the predicted position of the non-pre-cross-view-angle target object in the next frame image is calculated, and the predicted position of each target object corresponding to each target candidate box in the next frame image is obtained, thereby further improving the prediction accuracy of the predicted position of the target object in the next frame image.

[0056] In a second aspect, the present application also provides a cross-camera multi-target tracking system, the system comprising:

[0057] The acquisition module is configured to acquire a first detection box corresponding to each target object in each current frame image respectively captured by each camera.

[0058] The merging module is configured to merge the first detection boxes containing the same target object to obtain H second detection boxes, and take the H second detection boxes and the first detection boxes not merged as candidate boxes in a candidate box set, where H is an integer greater than zero.

[0059] The screening module is configured to find a target candidate box containing the target objects in all frame images before the current frame image in the candidate box set, and take the position of each target candidate box as the detection position of the target object corresponding to each target candidate box in the current frame image.

[0060] The processing module is configured to update the current position of the target object corresponding to each target candidate box in the current frame image based on the detection position of the target object corresponding to each target candidate box, and calculate the predicted position of the target object corresponding to each target candidate box in the next frame image.

[0061] In a possible implementation, the acquisition module is specifically configured to acquire the current frame image respectively captured by each camera.

[0062] Each current frame image is input into the target detection module respectively, and each first detection box corresponding to each target object in each current frame image is output respectively.

[0063] In a possible implementation, the merging module is specifically configured to acquire a feature vector corresponding to each first detection box.

[0064] calculate feature distances between each of the first bounding boxes based on the respective feature vectors corresponding to the first bounding boxes;

[0065] convert the respective first coordinates corresponding to each of the first bounding boxes into second coordinates in a same coordinate system to obtain second coordinates corresponding to each of the first bounding boxes;

[0066] for each of the first bounding boxes, filter the first bounding boxes containing the same target object based on the feature distances between each of the first bounding boxes, the second coordinates corresponding to each of the first bounding boxes, and the sizes corresponding to each of the first bounding boxes;

[0067] merge the first bounding boxes containing the same target object to obtain the H second bounding boxes, and use the H second bounding boxes and the first bounding boxes that are not merged as the candidate boxes in the candidate box set.

[0068] In a possible implementation, the screening module is specifically configured to, for each of the candidate boxes in the candidate box set, sequentially perform the following first matching operation:

[0069] use one of the candidate boxes in the candidate box set as a matching candidate box;

[0070] perform a second matching operation on each target object in all the frames of images before the current frame image and the matching candidate box to determine whether the matching candidate box contains the target object in all the frames of images before the current frame image;

[0071] if not, use the target object in the matching candidate box as a new target object;

[0072] if yes, use the matching candidate box as the target candidate box, and use the position of the target candidate box as the detection position of the target object corresponding to the target candidate box in the current frame image;

[0073] until each of the candidate boxes performs the first matching operation to obtain the detection position of the target object corresponding to each of the target candidate boxes in the current frame image.

[0074] In a possible implementation, the screening module is specifically configured to use one of the target objects in all the frames of images before the current frame image as a matching object;

[0075] obtain a third bounding box corresponding to the matching object, a first image feature corresponding to the matching object, and a second image feature corresponding to the target object in the matching candidate box;

[0076] calculate an image matching score between the second image feature and the first image feature and a frame matching score between the matching candidate frame and the third detection frame;

[0077] determine whether the image matching score is greater than an image preset threshold and the frame matching score is greater than a frame preset threshold simultaneously;

[0078] if not, confirm that the matching candidate frame does not contain the matching object;

[0079] if yes, confirm that the matching candidate frame contains the matching object.

[0080] In a possible implementation, the processing module is specifically configured to acquire a last frame predicted position of a target object corresponding to each target candidate frame, wherein the last frame predicted position is a position of the target object corresponding to each target candidate frame in a last frame image predicted in a last frame;

[0081] update a current position of the target object corresponding to each target candidate frame in the current frame image based on a detection position of the target object corresponding to each target candidate frame and the last frame predicted position;

[0082] calculate a predicted position of the target object corresponding to each target candidate frame in a next frame image according to the current position of the target object corresponding to each target candidate frame.

[0083] In a possible implementation, the processing module is specifically configured to determine that a pre-cross-view target object exists in the target object corresponding to each target candidate frame according to the current position of the target object corresponding to each target candidate frame, wherein the pre-cross-view target object is a target object that is about to enter a view of a second camera from a view of a first camera;

[0084] for each pre-cross-view target object, the following prediction operations are performed:

[0085] calculate a first predicted position of the pre-cross-view target object in a next frame image captured by the first camera;

[0086] determine whether the pre-cross-view target object is in an overlapping area of the views of the first camera and the second camera;

[0087] if yes, convert the first predicted position into a position in the view of the second camera to obtain a predicted position of the pre-cross-view target object in a next frame image;

[0088] If not, a fifth detection frame exceeding the visual angle of the first camera is intercepted from the fourth detection frame corresponding to the first predicted position, a second predicted position in the visual angle of the second camera is obtained by converting the position corresponding to the fifth detection frame, and the predicted position of the pre-cross-view target object in the next frame image is obtained.

[0089] Until each of the pre-cross-view target objects performs the prediction operation, the predicted position of the pre-cross-view target object in the next frame image is obtained, and the predicted position of the non-pre-cross-view target object in the next frame image is calculated for each target candidate frame.

[0090] In a third aspect, the present application provides an electronic device, comprising:

[0091] a memory for storing a computer program;

[0092] a processor for executing the computer program stored in the memory to realize the steps of the above-mentioned cross-camera multi-target tracking method.

[0093] In a fourth aspect, the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps of the above-mentioned cross-camera multi-target tracking method.

[0094] The above-mentioned second aspect to fourth aspect and the technical effects that can be achieved by each aspect are described above in the first aspect or the various possible schemes in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0095] Figure 1 A schematic diagram of the camera and the target object provided by the present application;

[0096] Figure 2 A flowchart of a cross-camera multi-target tracking method provided by the present application;

[0097] Figure 3a A camera visual angle diagram provided by the present application Figure 1 ;

[0098] Figure 3b A camera visual angle diagram provided by the present application Figure 2 ;

[0099] Figure 3c A camera visual angle diagram provided by the present application

[0100] Figure 3d A camera visual angle diagram provided by the present application Figure 4 ;

[0101] Figure 4 A processing process schematic diagram of a cross-camera multi-target tracking method provided for the present application is provided.

[0102] Figure 5 A schematic diagram of a cross-camera multi-target tracking system provided for the present application is provided.

[0103] Figure 6 An electronic device schematic diagram provided for the present application is provided. DETAILED DESCRIPTION

[0104] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. The specific operation method in the method embodiment can also be applied to the device embodiment or the system embodiment. It should be noted that in the description of the present application, "multiple" is understood as "at least two". The association relationship of the associated objects is described, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone. A is connected with B, which means that A is directly connected with B and A is connected with B through C. In addition, in the description of the present application, "first", "second", etc. are used only for the purpose of distinguishing the description, and cannot be understood as indicating or implying relative importance, nor can it be understood as indicating or implying order.

[0105] The embodiments of the present application will be described in detail below with reference to the drawings.

[0106] In cross-camera multi-target tracking, due to different viewing angles, the predicted positions of the same target object in the next frame image in the current frame image photographed by different cameras are quite different, which further leads to that when the predicted positions of the target object in the next frame image are obtained by fusing different predicted positions, the prediction accuracy of the position of the target object in the next frame image is not high.

[0107] Therefore, the present application provides a cross-camera multi-target tracking method, by acquiring a first detection box corresponding to each target object in each current frame image photographed by each camera; then for each first detection box, merging the first detection boxes containing the same target object to obtain H second detection boxes, wherein H is an integer greater than zero, and taking the H second detection boxes and the first detection boxes not merged as candidate boxes in a candidate box set; in the candidate box set, finding a target candidate box containing the target object in all frame images before the current frame image, and taking the position of each target candidate box as the detection position of the target object corresponding to each target candidate box in the current frame image; then updating the current position of the target object corresponding to each target candidate box in the current frame image based on the detection position of the target object corresponding to each target candidate box, and further calculating the predicted position of the target object corresponding to each target candidate box in the next frame image. By this method, the predicted positions of the target objects in the next frame image photographed by different cameras are avoided, thereby avoiding the fusion of different predicted positions to obtain the predicted position of the target object in the next frame image, and further improving the prediction accuracy of the position of the target object in the next frame image.

[0108] Referring to Figure 2 Fig. 1 shows a flowchart of a cross-camera multi-target tracking method provided by an embodiment of the present application, which comprises:

[0109] S1, acquiring a first detection box corresponding to each target object in each current frame image photographed by each camera;

[0110] In cross-camera multi-target tracking, when predicting the predicted positions of multiple target objects in the next frame image, it is necessary to first acquire a first detection box corresponding to each target object in each current frame image photographed by each camera.

[0111] Specifically, first, a current frame image photographed by each camera is acquired, wherein one camera corresponds to one current frame image.

[0112] It should be noted that in the embodiment of the present application, the union of the viewing angles of all cameras is 360°, that is, the sum of the viewing angles of all cameras can detect objects around the vehicle, thereby achieving omnidirectional detection of the surrounding environment and further improving the safety of intelligent driving.

[0113] For example, referring to the camera viewing angle diagram Figure 3a shown in Fig. 2, the camera viewing angle diagram shown in Fig. 2 shows the viewing angle of each camera, wherein the viewing angle of each camera is 90°, and the sum of the viewing angles of all cameras is 360°. Figure 1The vehicle is provided with six cameras, and each camera has a 90° field of view. The first camera is a front main camera, which can detect objects within a 90° range in front of the vehicle. The second camera is a rear main camera, which can detect objects within a 90° range behind the vehicle. The third, fourth, fifth, and sixth cameras are auxiliary cameras, which can detect objects within a 90° range in front of the left, right, back left, and back right of the vehicle, respectively.

[0114] After obtaining the current frame images, each current frame image is input into a target detection module, and each target object in each current frame image is output with a corresponding first detection box.

[0115] It should be noted that in the embodiments of the present application, the target detection module can be a detection model based on a fully convolutional one-stage 3D target detection (FCOS3D), or a region-based convolutional neural network (R-CNN) detection model. In the embodiments of the present application, the specific detection model in the target detection module is not limited, and the detection model in the target detection module can be adjusted according to the specific application scenario.

[0116] It should also be noted that one current frame image corresponds to N target objects, where N is an integer greater than zero. One target object corresponds to one first detection box, but the number of target objects included in each current frame image can be different, and the types of target objects included in each current frame image can also be different. In the embodiments of the present application, the number and types of target objects included in each current frame image can be determined according to the specific application scenario.

[0117] In addition, the target detection module outputs an ID corresponding to each first detection box when outputting the first detection box. One first detection box corresponds to one unique ID, so that the specific first detection box used in subsequent operations can be identified.

[0118] In the above manner, based on the target detection module, each target object in each current frame image captured by each camera is obtained with a corresponding first detection box.

[0119] S2, for each first detection box, merge the first detection boxes containing the same target object to obtain H second detection boxes, and take the H second detection boxes and the unmerged first detection boxes as candidate boxes in a candidate box set;

[0120] After obtaining each target object in each self-captured current frame image of each camera respectively corresponding first detection box, first, obtain each first detection box respectively corresponding feature vector.

[0121] It should be noted that in the embodiment of the present application, when obtaining the feature vector corresponding to the first detection box, the feature vector corresponding to the first detection box can be obtained based on the corner detection algorithm, or the feature vector corresponding to the first detection box can be obtained based on the neural network model. In the embodiment of the present application, the acquisition method of the feature vector corresponding to the first detection box can be adjusted according to the application scenario.

[0122] Then, based on each first detection box respectively corresponding feature vector, the feature distance between each first detection box is calculated, for example, there are 3 first detection boxes, and the IDs of the 3 first detection boxes are A, B and C respectively, wherein A corresponds to the first feature vector, B corresponds to the second feature vector, and C corresponds to the third feature vector. Then, the feature distance between the first feature vector and the second feature vector is calculated, the feature distance between the first feature vector and the third feature vector is calculated, and the feature distance between the second feature vector and the third feature vector is calculated.

[0123] Then, each first detection box respectively corresponding first coordinate is converted into second coordinate in the same coordinate system, and each first detection box respectively corresponding second coordinate is obtained.

[0124] It should be noted that in the embodiment of the present application, when each first detection box respectively corresponding first coordinate is converted into second coordinate in the same coordinate system, the world coordinate system can be used as the reference coordinate system, and then each first detection box respectively corresponding first coordinate is converted into second coordinate in the world coordinate system. Alternatively, the coordinate system in which the first coordinate of one first detection box is located can be used as the reference coordinate system, and then the first coordinates of the remaining first detection boxes are converted into second coordinates in the coordinate system. In the embodiment of the present application, the reference coordinate system can be adjusted according to the specific application scenario.

[0125] Further, for each first detection box, according to the feature distance between each first detection box, the second coordinate corresponding to each first detection box, and the size corresponding to each first detection box, the first detection box containing the same target object is screened; then the first detection box containing the same target object is merged to obtain H second detection boxes, and the H second detection boxes and the first detection boxes not merged are used as candidate boxes in the candidate box set, wherein H is an integer greater than zero.

[0126] It should be noted that when the first bounding boxes containing the same target object are combined to obtain H second bounding boxes, and the H second bounding boxes and the first bounding boxes that are not combined are taken as the candidate boxes in the candidate box set, the IDs corresponding to the second bounding boxes and the first bounding boxes that are not combined need to be renamed respectively.

[0127] For example, there are currently three first bounding boxes, and the IDs corresponding to the three first bounding boxes are A, B and C respectively, wherein only the first bounding boxes corresponding to A and B contain the same target object, at this time, A and B are combined to obtain a second bounding box, and the ID of the second bounding box is renamed as 001, and C is renamed as 002.

[0128] In the above manner, based on the feature vectors corresponding to each first bounding box, the feature distances between each first bounding box are obtained; then based on the feature distances between each first bounding box, the coordinates of each first bounding box in the same coordinate system and the sizes corresponding to each first bounding box, the first bounding boxes containing the same target object are determined, and the matching precision of the target objects between different first bounding boxes is improved; in addition, by combining the first bounding boxes containing the same target object, the candidate boxes in the candidate box set are obtained, and then the candidate boxes are provided for subsequent matching operations.

[0129] In a possible implementation, when the first bounding boxes containing the same target object are screened for each first bounding box according to the feature distances between each first bounding box, the second coordinates corresponding to each first bounding box and the sizes corresponding to each first bounding box, each two different first bounding boxes are taken as a group of first bounding boxes, for example, there are currently three first bounding boxes, and the IDs corresponding to the three first bounding boxes are A, B and C respectively, wherein A and B are the first group of first bounding boxes, A is the second group of first bounding boxes, and B and C are the third group of first bounding boxes.

[0130] Then each group of first bounding boxes is sequentially executed the following screening operation:

[0131] One first bounding box in a group of first bounding boxes is taken as a first screening box, and another first bounding box in the group of first bounding boxes is taken as a second screening box.

[0132] Then the feature distance between the first screening box and the second screening box is obtained; then the coordinate distance between the second coordinates corresponding to the first screening box and the second coordinates corresponding to the second screening box is calculated, and the size difference between the size of the first screening box and the size of the second screening box is calculated;

[0133] Secondly, weights are respectively assigned to the feature distance, the coordinate distance and the size gap, and then an object matching score is obtained, wherein the weights corresponding to the feature distance, the coordinate distance and the size gap can be consistent or inconsistent;

[0134] Then, it is determined whether the object matching score is less than an object preset threshold value;

[0135] If the object matching score is less than the object preset threshold value, it is indicated that the first screening frame and the second screening frame contain the same target object;

[0136] If the object matching score is not less than the object preset threshold value, it is indicated that the first screening frame and the second screening frame do not contain the same target object;

[0137] Until the above screening operation is performed on each group of first detection frames, the first detection frame containing the same target object in each first detection frame can be determined.

[0138] In a possible implementation, when screening the first detection frame containing the same target object, the first detection frame containing the same target object can also be screened based on a deep learning model. The deep learning model can be a convolutional neural network model (CNN) or a deep convolutional inverse graphics network model (DCIGN). In the embodiments of the present application, a specific deep learning model can be selected according to a specific application scenario.

[0139] Specifically, for each first detection frame, each two different first detection frames are taken as a group of first detection frames; then each group of first detection frames is input into the trained deep learning model, and the deep learning model directly outputs whether the two first detection frames in each group of first detection frames contain the same target object, so that the first detection frame containing the same target object in each first detection frame can be determined.

[0140] For example, the IDs corresponding to the first detection frames in the first group of first detection frames are A and B respectively. The first detection frames corresponding to A and B are input into the trained deep learning model, and then the deep learning model directly outputs that the first detection frames corresponding to A and B contain the same target object.

[0141] In a possible implementation, when screening the first detection frame containing the same target object, for each first detection frame, each two different first detection frames can be taken as a group of first detection frames;

[0142] Then, for each group of first detection frames, the following determination operations are sequentially performed:

[0143] First, the intersection of two first bounding boxes in a group of first bounding boxes and the union of the two first bounding boxes are obtained, and then the ratio of the intersection to the union is calculated to obtain the intersection over union (IOU) of the group of first bounding boxes;

[0144] Then, it is judged whether the IOU is greater than a preset intersection over union threshold value;

[0145] If the IOU is not greater than the preset intersection over union threshold value, it is indicated that the target objects contained in the two first bounding boxes in the group are not the same target object;

[0146] If the IOU is greater than the preset intersection over union threshold value, it is indicated that the target objects contained in the two first bounding boxes in the group are the same target object;

[0147] Until the above judgment operation is performed on each group of first bounding boxes, the first bounding box containing the same target object in each first bounding box is determined.

[0148] In a possible implementation, when the first bounding boxes containing the same target object are merged to obtain H second bounding boxes, the union of the first bounding boxes containing the same target object is obtained, the first bounding boxes in the union all contain the same target object, and then the union is taken as a second bounding box, and thus H second bounding boxes are obtained.

[0149] For example, there are currently four first bounding boxes, and the IDs corresponding to the four first bounding boxes are A, B, C and D, respectively, wherein only A and B contain the same target object, and only C and D contain the same target object. At this time, the union of A and B is obtained to obtain a first second bounding box, and the union of C and D is obtained to obtain a second second bounding box.

[0150] It should be noted that in the embodiments of the present application, when the first bounding boxes containing the same target object are merged to obtain H second bounding boxes, each first bounding box containing the same target object can also be retained at the same time, and then one of the first bounding boxes is taken as a second bounding box. For example, there are currently three first bounding boxes, and the IDs corresponding to the three first bounding boxes are A, B and C, respectively, wherein only A and B contain the same target object. At this time, A is taken as a second bounding box.

[0151] In the embodiments of the present application, the merging manner of the first bounding boxes containing the same target object can be adjusted according to specific application scenarios.

[0152] S3, in the candidate frame set, find the target candidate frame containing the target object in all the frame images before the current frame image, and take the position of each target candidate frame as the detection position of the target object corresponding to each target candidate frame in the current frame image respectively;

[0153] After screening the first detection frame containing the same target object in each first detection frame to obtain the candidate frame set, it is also necessary to screen out the target candidate frame containing the target object in all the frame images before the current frame image in the candidate frame set.

[0154] Specifically, for each candidate frame in the candidate frame set, the following first matching operation is performed in turn:

[0155] First, take one of the candidate frames in the candidate frame set as the matching candidate frame, for example, there are two candidate frames in the candidate frame set, and the IDs corresponding to the two candidate frames are 001 and 002 respectively, then select the candidate frame corresponding to 001 as the matching candidate frame from 001 and 002, but it needs to be explained here that after the first matching operation is performed on the candidate frame corresponding to 001, the next time the first matching operation is performed on the candidate frame corresponding to 002, and so on until the first matching operation is performed on the candidate frames corresponding to 001 and 002 respectively.

[0156] Then, the second matching operation is performed in turn by matching the matching candidate frame with each target object in all the frame images before the current frame image, for example, there are three frame images before the current frame image, among which, the first frame image has target objects A and B, the second frame image has target objects B and C, and the third frame image has target objects A, B and C, then the second matching operation is performed in turn by matching the matching candidate frame with A, the matching candidate frame with B and the matching candidate frame with C.

[0157] After performing the second matching operation, it is confirmed whether the matching candidate frame contains the target object in all the frame images before the current frame image;

[0158] If the matching candidate frame does not contain the target object in all the frame images before the current frame image, it means that the target object in the matching candidate frame does not appear in all the frame images before the current frame image, and then the target object in the matching candidate frame is taken as a new target object.

[0159] If the matching candidate frame contains the target object in all the frame images before the current frame image, the matching candidate frame is taken as a target candidate frame, and the position of the target candidate frame is taken as the detection position of the target object corresponding to the target candidate frame in the current frame image.

[0160] For example, there are three frame images before the current frame image, in which, the first frame image has target objects A and B, the second frame image has target objects B and C, and the third frame image has target objects A, B and C. After the first matching operation, it is confirmed that the matching candidate frame contains A, and the matching candidate frame is taken as the target candidate frame, and the position of the matching candidate frame is taken as the detection position of A in the current frame image.

[0161] Until the first matching operation is performed on each candidate frame, the detection position of each target object corresponding to each target candidate frame in the current frame image is obtained.

[0162] In addition, after the first matching operation is performed on each candidate frame, if all target objects in all frame images before the current frame image do not appear in all candidate frames in the candidate frame set, the missing frame number of the target object is recorded, for example, there are target objects A, B and C in all frame images before the current frame image, and all candidate frames in the candidate frame set do not contain C, then it is recorded that C is missing in the current frame image, and the missing frame number of C is increased by 1.

[0163] In the above manner, based on the first matching operation performed on each candidate frame in the candidate frame set in turn, it is confirmed whether the target candidate frame containing the target object in all frame images before the current frame image is contained in the candidate frame set, so that after all target candidate frames containing the target object in all frame images before the current frame image are found in the candidate frame set, the detection position of each target object corresponding to each target candidate frame in the current frame image is obtained, and then the detection position is provided for updating the current position of the target object in the current frame image.

[0164] Further, when the second matching operation is performed on the matching candidate frame and each target object in all frame images before the current frame image in turn, the second matching operation is as follows:

[0165] One target object in all frame images before the current frame image is taken as a matching object, for example, there are three frame images before the current frame image, and there are target objects A, B and C in the three frame images, when the second matching operation is performed on the matching candidate frame and A, A is taken as the matching object.

[0166] Then, the third detection frame corresponding to the matching object, the first image feature corresponding to the matching object and the second image feature corresponding to the target object in the matching candidate frame are obtained.

[0167] It should be noted that in the embodiments of the present application, when the first image features corresponding to the matching object and the second image features corresponding to the target object in the matching candidate box are obtained, that is, when the image features corresponding to the object are obtained, the image features corresponding to the object can be obtained based on the CNN, or the image features corresponding to the object can be obtained based on the DCIGN. In the embodiments of the present application, the specific acquisition mode of the image features can be adjusted according to the specific application scene.

[0168] After obtaining the third detection box corresponding to the matching object, the first image features corresponding to the matching object, and the second image features corresponding to the target object in the matching candidate box, the image matching score between the second image features and the first image features and the box matching score between the matching candidate box and the third detection box are calculated.

[0169] It is further judged whether the image matching score greater than the image preset threshold and the box matching score greater than the box preset threshold are satisfied simultaneously.

[0170] If the image matching score greater than the image preset threshold and the box matching score greater than the box preset threshold cannot be satisfied simultaneously, it is indicated that the target object in the matching candidate box is inconsistent with the matching object, that is, the matching candidate box does not contain the matching object.

[0171] If the image matching score greater than the image preset threshold and the box matching score greater than the box preset threshold are satisfied simultaneously, it is indicated that the target object in the matching candidate box is consistent with the matching object, that is, the matching candidate box contains the matching object.

[0172] Through the above manner, after the image matching score between the first image features and the second image features is calculated and the box matching score between the matching candidate box and the third detection box is calculated, whether the matching candidate box contains the matching object is determined based on the judgment of the image matching score and the image preset threshold and the judgment of the box matching score and the box preset threshold, and the matching precision between the target object in the current frame image and the target objects in all the frame images before the current frame image is further improved.

[0173] S4, based on the detection position of the target object corresponding to each target candidate box, updating the current position of the target object corresponding to each target candidate box in the current frame image, and calculating the predicted position of the target object corresponding to each target candidate box in the next frame image;

[0174] After obtaining the detection position of the target object corresponding to each target candidate box in the current frame image, the position of the target object corresponding to the target candidate box in the next frame image is predicted.

[0175] Specifically, first, a previous frame prediction position corresponding to each target object corresponding to each target candidate frame is acquired, wherein the previous frame prediction position is a position corresponding to each target object corresponding to each target candidate frame in a current frame image, which is predicted in a previous frame image.

[0176] Then, based on the detection position corresponding to each target object corresponding to each target candidate frame and the previous frame prediction position, a current position corresponding to each target object corresponding to each target candidate frame in the current frame image is updated.

[0177] It should be noted that in the embodiments of the present application, in the process of updating the current position based on the detection position and the previous frame prediction position, the detection position and the previous frame prediction position can be updated to the current position based on Kalman filtering, or the detection position and the previous frame prediction position can be updated to the current position based on particle filtering, or the detection position and the previous frame prediction position can be updated to the current position based on the mean, and in the embodiments of the present application, the specific updating method of the current position can be adjusted according to the specific application scenario.

[0178] For example, the target object corresponding to the target candidate frame is A, the detection position corresponding to A is (1, 3, 4), the previous frame prediction position of A is (2, 3, 4), the mean of the detection position corresponding to A and the previous frame prediction position corresponding to A is calculated, and the mean is (1.5, 3, 4), so that the mean (1.5, 3, 4) is taken as the current position corresponding to A, and the current position corresponding to A is updated.

[0179] Then, according to the current position corresponding to each target object corresponding to each target candidate frame, a prediction position corresponding to each target object corresponding to each target candidate frame in a next frame image is calculated.

[0180] In the above manner, based on the detection position corresponding to each target object corresponding to each target candidate frame and the previous frame prediction position, the current position of each target object corresponding to each target candidate frame in the current frame image is updated, so that the prediction position of each target object corresponding to each target candidate frame in the next frame image is calculated according to the current position, and the prediction accuracy of the prediction position of the target object in the next frame image is further improved.

[0181] Further, when calculating the prediction position of each target object corresponding to each target candidate frame in the next frame image according to the current position corresponding to each target object corresponding to each target candidate frame, first, it is determined that there is a pre-cross-view-angle target object in each target object corresponding to each target candidate frame according to the current position corresponding to each target object corresponding to each target candidate frame, wherein the pre-cross-view-angle target object is a target object that will enter the view angle of the second camera from the view angle of the first camera.

[0182] Then, for each target object across the pre-viewpoint, the following prediction operation is performed:

[0183] First, calculate the first predicted position of the target object in the next frame image captured by the first camera.

[0184] It should be noted that, in this embodiment, when calculating the first predicted position of the pre-cross-view target object in the next frame image captured by the first camera, the calculation can be based on Kalman filtering or deep learning. In this embodiment, the calculation method of the first predicted position of the pre-cross-view target object in the next frame image can be adjusted according to the specific application scenario.

[0185] Then determine whether the target object in the pre-cross-view angle is in the overlapping area of ​​the view angles of the first camera and the second camera from the second camera's view angle;

[0186] If the target object across the pre-viewpoint is in the overlapping area of ​​the viewpoints of the first and second cameras in the viewpoint of the second camera, the first predicted position is converted into the position in the viewpoint of the second camera to obtain the predicted position of the target object across the pre-viewpoint in the next frame image.

[0187] It should be noted that, in the embodiments of this application, when the view of the pre-crossing view target object is in the overlapping area of ​​the view of the first camera and the second camera, the pre-crossing view target object may be in the overlapping area of ​​the view of the first camera and the second camera from other overlapping areas of view.

[0188] For example, refer to Figure 3b The camera view shown is a schematic diagram. Figure 2 The ID of the target candidate box corresponding to the target object in the pre-cross view is A. The target object corresponding to A is about to enter the γ region from the β region. The β region is the view overlap area between the first camera and the third camera, and the γ region is the view overlap area between the first camera and the second camera. In other words, the target object corresponding to A is about to enter the other view overlap area from one view overlap area.

[0189] The first predicted position of the target object corresponding to A in the next frame image captured by the first camera is D2 in the γ region. Since the γ region is within the field of view of the second camera, it is predicted that the target object corresponding to A will appear in the next frame image captured by the second camera. Therefore, it is also necessary to determine the predicted position of the target object corresponding to A in the next frame image captured by the second camera.

[0190] The γ region is the visual angle overlap region of the first camera and the second camera, and thus the first predicted position D2 is directly converted into the D3 position in the visual angle of the second camera, to obtain the predicted position of the target object corresponding to A in the next frame of image captured by the second camera, and further to obtain the predicted positions of the target object corresponding to A in all the next frames of image.

[0191] In addition, when the pre-cross visual angle target object is in the visual angle of the second camera, the visual angle overlap region of the first camera and the second camera is the visual angle overlap region of the first camera and the second camera.

[0192] For example, referring to the camera visual angle diagram three as shown in FIG. 3, the ID of the target candidate box corresponding to the pre-cross visual angle target object is A, and the target object corresponding to A is about to enter the β region from the α region, wherein the β region is the visual angle overlap region of the first camera and the second camera, that is, the target object corresponding to A is about to enter the visual angle overlap region from the non-visual angle overlap region. Figure 3c

[0193] At this time, the first predicted position of the target object corresponding to A in the next frame of image captured by the first camera is still calculated first, which is D2 in the β region, and then the D2 position is converted into the D3 position in the visual angle of the second camera, to obtain the predicted position of the target object corresponding to A in the next frame of image captured by the second camera, and further to obtain the predicted positions of the target object corresponding to A in all the next frames of image.

[0194] If the pre-cross visual angle target object is not in the visual angle overlap region of the first camera and the second camera, the fourth detection box corresponding to the first predicted position is intercepted to obtain the fifth detection box beyond the visual angle of the first camera, the position corresponding to the fifth detection box is converted into the second predicted position in the visual angle of the second camera, to obtain the predicted position of the pre-cross visual angle target object in the next frame of image.

[0195] For example, referring to the camera visual angle diagram three as shown in FIG. 3, the ID of the target candidate box corresponding to the pre-cross visual angle target object is A, and the first predicted position of the target object corresponding to A in the next frame of image captured by the first camera is D2 in the α region, but the target candidate box corresponding to the D2 position has part of the content beyond the visual angle range of the first camera, and the beyond part is in the visual angle range of the second camera, and the ID corresponding to the beyond part is named as A1, and thus the second predicted position of the target object corresponding to A1 in the next frame of image captured by the second camera needs to be calculated. Figure 3d Figure 4

[0196] ​​​The D3 position corresponding to A1 is converted into a D4 position in the second camera, that is, a second predicted position of the target object corresponding to A1 in a next frame of image photographed by the second camera is obtained, and then a predicted position of the target object corresponding to A in all next frames of image is obtained.

[0197] Until each pre-cross-view-angle target object performs the above prediction operation, the predicted position of the pre-cross-view-angle target object in the next frame of image is obtained, and the predicted position of the non-pre-cross-view-angle target object in the next frame of image in each target candidate box corresponding target object is calculated.

[0198] It should be noted that in the embodiments of the present application, when calculating the predicted position of the non-pre-cross-view-angle target object in the next frame of image in each target candidate box corresponding target object, the calculation method is consistent with the calculation method when calculating the predicted position of the pre-cross-view-angle target object in the next frame of image, which can be a Kalman filtering based calculation method or a deep learning based calculation method. In the embodiments of the present application, the calculation method of the predicted position of the non-pre-cross-view-angle target object in the next frame of image can be adjusted according to the specific application scene. In addition, in the embodiments of the present application, the calculation methods of the predicted positions of the pre-cross-view-angle target object and the non-pre-cross-view-angle target object in the next frame of image can be inconsistent.

[0199] In the above manner, it is determined whether there is a pre-cross-view-angle target object in each target candidate box corresponding target object according to the current position of the target object corresponding to each target candidate box. Then, when there is a pre-cross-view-angle target object, the specific calculation method of the predicted position of the pre-cross-view-angle target object in the second camera is determined based on the judgment of whether the pre-cross-view-angle target object is in the view angle overlap area, and then the predicted position of the pre-cross-view-angle target object in all next frames of image is obtained. In addition, the predicted position of the non-pre-cross-view-angle target object in the next frame of image is calculated, and the predicted position of each target candidate box corresponding target object in the next frame of image is obtained, which further improves the prediction accuracy of the predicted position of the target object in the next frame of image.

[0200] In summary, the cross-camera multi-target tracking method provided in the application obtains a first detection box corresponding to each target object in each current frame image photographed by each camera; then, for each first detection box, the first detection boxes containing the same target object are merged to obtain H second detection boxes, and the H second detection boxes and the first detection boxes that are not merged are taken as candidate boxes in a candidate box set; in the candidate box set, a target candidate box containing the target object in all frame images before the current frame image is found, and the position of each target candidate box is taken as the detection position of the target object corresponding to each target candidate box in the current frame image; then, based on the detection position of the target object corresponding to each target candidate box, the current position of the target object corresponding to each target candidate box in the current frame image is updated, and the predicted position of the target object corresponding to each target candidate box in the next frame image is calculated. By this method, the predicted positions of the target objects in the next frame image photographed by different cameras are avoided, so that the predicted positions of the target objects in the next frame image are obtained by fusing different predicted positions, and the prediction accuracy of the positions of the target objects in the next frame image is improved.

[0201] The technical scheme of the application will be further described below in combination with a specific application process.

[0202] As shown in the following table, the cross-camera multi-target tracking method is used to track the target objects in the current frame images photographed by the N cameras. Figure 4 As shown in the following table, the cross-camera multi-target tracking method is used to track the target objects in the current frame images photographed by the N cameras.

[0203] In the target detection module, the target detection model is used to process the N current frame images to obtain a first detection box corresponding to each target object in each current frame image, and then each first detection box is input into the merging module.

[0204] In the merging module, the first detection boxes containing the same target object are selected from each first detection box; then, the first detection boxes containing the same target object are merged by obtaining the union of the first detection boxes containing the same target object to obtain H second detection boxes, wherein H is an integer greater than zero; secondly, the H second detection boxes and the first detection boxes that are not merged are taken as candidate boxes in a candidate box set; and each candidate box in the candidate box set is transmitted to the matching module.

[0205] In the matching module, target candidate boxes containing target objects in all frames of images before the current frame in the candidate box set are screened, and the position of each target candidate box is taken as the detection position of the target object corresponding to each target candidate box in the current frame of image, and then the detection position of the target object corresponding to each target candidate box is transmitted to the updating module;

[0206] In the updating module, the current position of the target object corresponding to each target candidate box in the current frame of image is updated by fusing the detection position of the target object corresponding to each target candidate box and the predicted position of the last frame, and the current position of the target object corresponding to each target candidate box is transmitted to the prediction module;

[0207] In the prediction module, the predicted position of the target object corresponding to each target candidate box in the next frame of image is calculated based on the current position of the target object corresponding to each target candidate box.

[0208] In this way, based on the target detection module, the first detection box corresponding to each target object in each current frame of image photographed by each camera is obtained; then in the merging module, for each first detection box, the first detection boxes containing the same target object are merged to obtain H second detection boxes, and the H second detection boxes and the first detection boxes not merged are taken as candidate boxes in the candidate box set; secondly, in the matching module, target candidate boxes containing target objects in all frames of images before the current frame of image are found from the candidate box set, and the position of each target candidate box is taken as the detection position of the target object corresponding to each target candidate box in the current frame of image; further, in the updating module, the current position of the target object corresponding to each target candidate box in the current frame of image is obtained based on the detection position of the target object corresponding to each target candidate box and the predicted position of the last frame; and then in the prediction module, the predicted position of the target object corresponding to each target candidate box in the next frame of image is calculated based on the current position of the target object corresponding to each target candidate box in the current frame of image. Through this method, the predicted positions of the target objects in the current frame of image photographed by different cameras in the next frame of image are avoided to be predicted respectively, so that the predicted positions of the target objects in the next frame of image are avoided to be fused to obtain the predicted position of the target object in the next frame of image, and the prediction accuracy of the position of the target object in the next frame of image is improved.

[0209] Based on the same inventive concept, the embodiment of the present application also provides a cross-camera multi-target tracking system, as shown in Figure 5 The structure of the cross-camera multi-target tracking system provided by the present application is shown in the figure, which comprises:

[0210] The acquisition module 501 is configured to acquire a first detection box corresponding to each target object in each current frame image respectively photographed by each camera.

[0211] The merging module 502 is configured to merge the first detection boxes containing the same target object to obtain H second detection boxes for each first detection box, and take the H second detection boxes and the first detection boxes not merged as candidate boxes in a candidate box set, where H is an integer greater than zero.

[0212] The screening module 503 is configured to find a target candidate box containing the target object in all frame images before the current frame image in the candidate box set, and take the position of each target candidate box as the detection position of the target object corresponding to each target candidate box in the current frame image.

[0213] The processing module 504 is configured to update the current position of the target object corresponding to each target candidate box in the current frame image based on the detection position of the target object corresponding to each target candidate box, and calculate the predicted position of the target object corresponding to each target candidate box in a next frame image.

[0214] In a possible implementation, the acquisition module 501 is specifically configured to acquire the current frame image respectively photographed by each camera.

[0215] Each current frame image is input into a target detection module respectively, and each target object in each current frame image outputs a first detection box corresponding to each target object respectively.

[0216] In a possible implementation, the merging module 502 is specifically configured to acquire a feature vector corresponding to each first detection box.

[0217] The feature distance between each first detection box is calculated based on the feature vector corresponding to each first detection box.

[0218] The first coordinate corresponding to each first detection box is converted into a second coordinate in the same coordinate system to obtain the second coordinate corresponding to each first detection box.

[0219] For each first detection box, the first detection boxes containing the same target object are screened according to the feature distance between each first detection box, the second coordinate corresponding to each first detection box, and the size corresponding to each first detection box.

[0220] The first detection boxes containing the same target object are merged to obtain H second detection boxes, and the H second detection boxes and the first detection boxes not merged are taken as candidate boxes in a candidate box set.

[0221] In a possible implementation, the screening module 503 is specifically configured to perform the following first matching operation for each candidate box in the candidate box set in sequence:

[0222] taking one candidate box in the candidate box set as a matching candidate box;

[0223] performing a second matching operation on the matching candidate box and each target object in all the frame images before the current frame image in sequence to determine whether the matching candidate box contains the target object in all the frame images before the current frame image;

[0224] if not, taking the target object in the matching candidate box as a new target object;

[0225] if yes, taking the matching candidate box as a target candidate box, and taking the position of the target candidate box as a detection position of the target object corresponding to the target candidate box in the current frame image;

[0226] until the first matching operation is performed on each candidate box, and a detection position of each target object corresponding to each target candidate box in the current frame image is obtained.

[0227] In a possible implementation, the screening module 503 is specifically configured to take one target object in all the frame images before the current frame image as a matching object;

[0228] obtaining a third detection box corresponding to the matching object, a first image feature corresponding to the matching object, and a second image feature corresponding to the target object in the matching candidate box;

[0229] calculating an image matching score between the second image feature and the first image feature, and a box matching score between the matching candidate box and the third detection box;

[0230] determining whether the image matching score is greater than an image preset threshold and the box matching score is greater than a box preset threshold at the same time;

[0231] if not, determining that the matching candidate box does not contain the matching object;

[0232] if yes, determining that the matching candidate box contains the matching object.

[0233] In a possible implementation, the processing module 504 is specifically configured to obtain a last frame prediction position of each target object corresponding to each target candidate box, where the last frame prediction position is a position of each target object corresponding to each target candidate box in the current frame image predicted in a last frame image;

[0234] updating a current position of each target object corresponding to each target candidate box in the current frame image based on the detection position of each target object corresponding to each target candidate box and the last frame prediction position.

[0235] According to the current position corresponding to each target object corresponding to each target candidate frame, a predicted position of each target object corresponding to each target candidate frame in a next frame image is calculated.

[0236] In a possible implementation, the processing module 504 is specifically configured to determine, according to the current position corresponding to each target object corresponding to each target candidate frame, whether a pre-cross-view-angle target object exists in each target object corresponding to each target candidate frame, wherein the pre-cross-view-angle target object is a target object that is about to enter the view angle of the second camera from the view angle of the first camera;

[0237] For each pre-cross-view-angle target object, the following prediction operation is performed:

[0238] A first predicted position of the pre-cross-view-angle target object in a next frame image captured by the first camera is calculated;

[0239] It is determined whether the pre-cross-view-angle target object is in the overlapping area of the view angles of the first camera and the second camera;

[0240] If yes, the first predicted position is converted into a position in the view angle of the second camera to obtain a predicted position of the pre-cross-view-angle target object in the next frame image;

[0241] If no, a fifth detection frame that is a fourth detection frame corresponding to the first predicted position and that is outside the view angle of the first camera is intercepted, a second predicted position corresponding to the fifth detection frame in the view angle of the second camera is converted, and a predicted position of the pre-cross-view-angle target object in the next frame image is obtained;

[0242] Until each pre-cross-view-angle target object has performed the prediction operation and obtained the predicted position of the pre-cross-view-angle target object in the next frame image, and a predicted position of a non-pre-cross-view-angle target object in each target object corresponding to each target candidate frame in the next frame image is calculated.

[0243] Based on the same inventive concept, the embodiment of the present application further provides an electronic device, which can realize the functions of the foregoing cross-camera multi-target tracking system, and the foregoing electronic device can be used as the cross-camera multi-target tracking system. Figure 6 The foregoing electronic device includes:

[0244] At least one processor 601 and a memory 602 connected with the at least one processor 601, and the specific connection medium between the processor 601 and the memory 602 is not limited in the embodiment of the present application, Figure 6 for example, the processor 601 and the memory 602 are connected through a bus 600. The bus 600 is used for Figure 6The connection between the other components is indicated by a thick line, which is only illustrative and not limited. The bus 600 can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity, Figure 6 The bus 600 is indicated by a thick line, but it does not mean that there is only one bus or only one type of bus. Alternatively, the processor 601 can also be referred to as a controller, and the name is not limited.

[0245] In the embodiment of the present application, the memory 602 stores instructions executable by the at least one processor 601, and the at least one processor 601 can execute the cross-camera multi-target tracking method discussed above by executing the instructions stored in the memory 602. The processor 601 can realize Figure 6 The functions of the modules in the system shown.

[0246] The processor 601 is the control center of the system, and can connect various parts of the control device through various interfaces and lines. By running or executing the instructions stored in the memory 602 and calling the data stored in the memory 602, the system can process various functions and data, thereby monitoring the system as a whole.

[0247] In a possible design, the processor 601 can include one or more processing units. The processor 601 can integrate an application processor and a modem processor, where the application processor mainly processes operating systems, user interfaces, and application programs, and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 601. In some embodiments, the processor 601 and the memory 602 can be implemented on the same chip, and in some embodiments, they can also be implemented on separate chips respectively.

[0248] The processor 601 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, which can realize or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the cross-camera multi-target tracking method disclosed in the embodiments of the present application can be directly embodied as execution completed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0249] The memory 602, as a non-volatile computer readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 602 can include at least one type of storage medium, for example, can include flash memory, hard disk, multimedia card, card type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. The memory 602 is any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but is not limited thereto. The memory 602 in the embodiments of the present application can also be a circuit or any other device capable of realizing a storage function, used for storing program instructions and / or data.

[0250] By designing and programming the processor 601, the code corresponding to the cross-camera multi-target tracking method introduced in the foregoing embodiments can be fixed into the chip, so that the chip can execute the steps of the cross-camera multi-target tracking method of the embodiments shown in FIG. 8 at runtime. Figure 5 How to design and program the processor 601 is a technology known to those skilled in the art, which will not be described here.

[0251] Based on the same inventive concept, the embodiments of the present application also provide a storage medium storing computer instructions, when the computer instructions run on a computer, the computer instructions make the computer execute the cross-camera multi-target tracking method discussed above.

[0252] In some possible implementations, various aspects of the cross-camera multi-target tracking method provided by the present application can also be implemented in the form of a program product, which includes program codes, when the program product runs on a device, the program codes are used to make the control device execute the steps in the cross-camera multi-target tracking method according to various exemplary embodiments of the present application described above in the specification.

[0253] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, and the like) embodying computer readable program code.

[0254] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0255] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0256] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0257] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A cross-camera multi-target tracking method, characterized in that, The method comprises: obtaining a first detection frame corresponding to each target object in each current frame image respectively photographed by each camera; obtaining a feature vector corresponding to each first detection frame respectively; calculating a feature distance between each first detection frame based on the feature vector corresponding to each first detection frame respectively; converting the first coordinate corresponding to each first detection frame into a second coordinate in the same coordinate system to obtain the second coordinate corresponding to each first detection frame respectively; for each first detection frame, screening the first detection frame containing the same target object according to the feature distance between each first detection frame, the second coordinate corresponding to each first detection frame respectively, and the size corresponding to each first detection frame respectively; merging the first detection frame containing the same target object to obtain H second detection frames, and taking the H second detection frames and the first detection frame not merged as a candidate frame in a candidate frame set, wherein H is an integer greater than zero; in the candidate frame set, finding a target candidate frame containing the target object in all frame images before the current frame image, and taking the position of each target candidate frame as the detection position of the target object corresponding to each target candidate frame in the current frame image respectively; updating the current position of the target object corresponding to each target candidate frame in the current frame image based on the detection position of the target object corresponding to each target candidate frame, and calculating the predicted position of the target object corresponding to each target candidate frame in the next frame image.

2. The method of claim 1, wherein, The method comprises: obtaining a first detection frame corresponding to each target object in each current frame image respectively photographed by each camera; inputting each current frame image into a target detection module respectively to output the first detection frame corresponding to each target object in each current frame image respectively.

3. The method of claim 1, wherein, The method comprises: for each candidate frame in the candidate frame set, sequentially performing the following first matching operation: taking one candidate frame in the candidate frame set as a matching candidate frame; confirming whether the matching candidate frame contains the target object in all frame images before the current frame image by sequentially performing a second matching operation on each target object in all frame images before the current frame image through the matching candidate frame; if not, taking the target object in the matching candidate frame as a new target object; if yes, taking the matching candidate frame as the target candidate frame, and taking the position of the target candidate frame as the detection position of the target object corresponding to the target candidate frame in the current frame image; until each candidate frame performs the first matching operation to obtain the detection position of the target object corresponding to each target candidate frame in the current frame image.

4. The method of claim 3, wherein, The second matching operation comprises: taking one target object in all frame images before the current frame image as a matching object; obtaining a third detection box corresponding to the matching object, a first image feature corresponding to the matching object, and a second image feature corresponding to the target object in the matching candidate box; calculating an image matching score between the second image feature and the first image feature and a box matching score between the matching candidate box and the third detection box; determining whether the image matching score is greater than an image preset threshold and the box matching score is greater than a box preset threshold at the same time; if not, confirming that the matching candidate box does not contain the matching object; if yes, confirming that the matching candidate box contains the matching object.

5. The method of claim 1, wherein, The updating of the current position of the target object corresponding to each target candidate box in the current frame image based on the detection position of the target object corresponding to each target candidate box, and the calculation of the predicted position of the target object corresponding to each target candidate box in the next frame image, comprises: obtaining a last frame predicted position corresponding to the target object corresponding to each target candidate box, wherein the last frame predicted position is the position of the target object corresponding to each target candidate box in the current frame image predicted in the last frame image; updating the current position of the target object corresponding to each target candidate box in the current frame image based on the detection position of the target object corresponding to each target candidate box and the last frame predicted position; calculating the predicted position of the target object corresponding to each target candidate box in the next frame image according to the current position of the target object corresponding to each target candidate box.

6. The method of claim 5, wherein, The calculation of the predicted position of the target object corresponding to each target candidate box in the next frame image according to the current position of the target object corresponding to each target candidate box, comprises: determining that there is a pre-cross-view-angle target object in the target object corresponding to each target candidate box according to the current position of the target object corresponding to each target candidate box, wherein the pre-cross-view-angle target object is a target object that will soon enter the view angle of a second camera from the view angle of a first camera; for each pre-cross-view-angle target object, the following prediction operations are performed: calculating a first predicted position of the pre-cross-view-angle target object in the next frame image shot by the first camera; determining whether the pre-cross-view-angle target object is in the overlapping area of the view angles of the first camera and the second camera; if yes, converting the first predicted position into a position in the view angle of the second camera to obtain the predicted position of the pre-cross-view-angle target object in the next frame image; if no, intercepting a fifth detection box that is a fourth detection box corresponding to the first predicted position and exceeding the view angle of the first camera, and converting a position corresponding to the fifth detection box into a second predicted position in the view angle of the second camera to obtain the predicted position of the pre-cross-view-angle target object in the next frame image; Until each of the pre-cross-view-angle target objects performs the prediction operation, a predicted position of the pre-cross-view-angle target object in a next frame image is obtained, and a predicted position of a non-pre-cross-view-angle target object in the next frame image is calculated for each target candidate box corresponding target object.

7. A cross-camera multi-target tracking system, characterized in that, The system comprises: An acquisition module is configured to acquire a first detection box corresponding to each target object in each frame image captured by each camera; A merging module is configured to acquire a feature vector corresponding to each first detection box, calculate a feature distance between each pair of first detection boxes based on the feature vector corresponding to each first detection box, convert the first coordinates corresponding to each first detection box into second coordinates in a same coordinate system to obtain the second coordinates corresponding to each first detection box, for each first detection box, screen the first detection boxes containing the same target object according to the feature distance between each pair of first detection boxes, the second coordinates corresponding to each first detection box, and the size of each first detection box, merge the first detection boxes containing the same target object to obtain H second detection boxes, and take the H second detection boxes and the first detection boxes not merged as candidate boxes in a candidate box set, wherein H is an integer greater than zero; A screening module is configured to find target candidate boxes containing target objects in all frame images before the current frame image in the candidate box set, and take the position of each target candidate box as the detection position of the target object corresponding to each target candidate box in the current frame image. A processing module is configured to update the current position of the target object corresponding to each target candidate box in the current frame image based on the detection position of the target object corresponding to each target candidate box, and calculate the predicted position of the target object corresponding to each target candidate box in a next frame image.

8. An electronic device, comprising: The memory is configured to store a computer program. The processor is configured to execute the computer program stored in the memory to implement the method steps in any one of claims 1-6. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method steps in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, ​

Images