Object re-identification adapted to image parameters in video stream

Abstract

The present invention relates generally to the field of object re-recognition in a video stream adapted to image parameters, and in particular to object re-recognition in a video stream captured by a camera (100). In particular, camera monitoring is disclosed.

Description

Technical Field

[0001] The present invention relates generally to the field of camera surveillance, and more particularly to object re-identification in a video stream captured by a camera. Background Art

[0002] In camera surveillance, object detection and object tracking are important functions. When tracking an object in a video stream, the object is observed in several instances and needs to be re-identified as the same object between instances by an algorithm.

[0003] Object re-identification is a technique that can be used to compare observations of an object to determine if two observations belong to the same object. Observations of an object instance can be accumulated in so-called tracks. The comparison can be done within one camera field of view or across different camera fields of view.

[0004] Object re-identification algorithms rely on offline training. Based on their training, the algorithms typically extract a feature vector from each object observation, sometimes called an appearance vector or a re-identification vector.

[0005] For re-identification, the task becomes comparing feature vectors accumulated over the trajectory, and if two feature vectors are considered similar, the corresponding objects are assumed to be the same. Thus, a detected object of the first instance can be re-identified as corresponding to the detected object of the second instance.

[0006] However, sudden changes such as light level, color, or other image-related parameters can cause problems for traditional object re-identification algorithms.

[0007] Therefore, there is room for improvement in object re-identification algorithms. Summary of the invention

[0008] In view of the above and other deficiencies in the prior art, an object of the present invention is to provide improvements in object re-identification to better address changes in image parameters.

[0009] Therefore, according to a first aspect of the present invention, there is provided a computer-implemented method for in-camera object re-identification, the method comprising: detecting, by an object detection algorithm, an object in a first set of image frames of a video stream captured by a camera; determining at least one image parameter of the first set of image frames, determining a first ReID feature vector describing the detection of the object in the first set of image frames; storing the at least one image parameter together with the first ReID feature vector in a data storage device; detecting, by the object detection algorithm, the object in a subsequent set of image frames of the video stream captured by the camera;

[0010] determining at least one image parameter of a subsequent set of image frames,

[0011] Determine at least one other ReID feature vector describing the detection of an object in a subsequent set of image frames, quantify the difference between at least one image parameter of the first set of image frames and at least one image parameter of the subsequent set of image frames, provide at least one re-identification algorithm configured to take the first ReID feature vector and the other ReID feature vector as input to determine whether the object in the first set of image frames is the same as the object in the subsequent set of image frames based on a re-identification threshold, adjust the re-identification threshold to take into account the quantized difference between the image parameter associated with the first ReID feature vector and the image parameter of the subsequent set of image frames, apply a re-identification algorithm to estimate whether the object in the first set of image frames is the same as the object in another track of image frames, and provide an output of the estimation result in the re-identification algorithm.

[0012] The present invention is based on the recognition that by quantifying the differences in image parameters, the re-identification threshold can be adjusted to take into account the differences. It is thus achieved that at least one image parameter associated with a first ReID feature vector is stored. The stored at least one image parameter can be compared with a corresponding at least one image parameter that is associated or linked to a subsequent ReID feature vector. In the case where the corresponding image parameters are relatively similar, the re-identification algorithm can operate as usual. However, if the quantified difference between the corresponding image parameters is too large, the re-identification threshold is adjusted to take into account the difference, thereby enabling the re-identification algorithm to perform re-identification despite the relatively large difference in image parameters.

[0013] A video stream is typically a collection of consecutive image frames captured over time. The consecutive image frames together form a video stream.

[0014] A feature vector is typically an n-dimensional vector representing the digital features of an object. This is a common representation of objects used, for example, in machine learning algorithms, which typically require digital representations of objects to achieve, for example, data processing and statistical analysis.

[0015] It should be understood that the object in this article can refer to a physical object, a person or an animal, in other words, any type of object that can be captured in a video stream and can be tracked. For example, the physical object can be a vehicle.

[0016] Re-ID typically involves labeling or otherwise identifying that two detections belong to the same object, such as an object ID.

[0017] Object detection algorithms are known per se and can be selected from a range of algorithms, including convolutional neural networks (CNNs), recursive neural networks, decision tree classifiers, such as random forest classifiers, which are also effective for classification. In addition, classifiers such as support vector machine classifiers and logistic regression classifiers are also possible. The object detection algorithm can be an object tracker using a convolutional neural network that is trained to detect and track objects according to its training.

[0018] In one embodiment, at least one re-identification algorithm is at least one neural network. Such a neural network may be a convolutional neural network (CNN).

[0019] A network or algorithm configured or trained for re-identification is applied to the crop in which the object is detected. Therefore, the object detection algorithm or network is preferably a separate network or algorithm.

[0020] Two ReID feature vectors are compared to each other to determine whether they belong to the same object. This matching step can be performed in different ways using different metrics and paradigms, but the goal is to determine the similarity between the ReID feature vectors. To this end, the image can be mapped to a Euclidean space, where the distance directly corresponds to a measure of similarity. For example, the L2 paradigm, which calculates the "Euclidean distance" between two vectors, can be used to determine similarity. The L2 norm of the difference between the vectors can then be compared to a re-identification threshold. Other norms may also be applicable, such as the Mahalanobis distance or paradigm.

[0021] There are multiple types of image parameters suitable for use with embodiments of the present invention. When performing differences between image parameters, quantization is performed between the same image parameter types.

[0022] Image parameters are parameters that affect the image quality or the ability to identify objects in the image.

[0023] For example, in an embodiment, at least one image parameter may include the light level in the image, such as brightness. Changes in the light level may be caused by gain changes or exposure time changes in the camera, or by changes in the monitored scene.

[0024] In another example, at least one image parameter includes a color map of the image. For example, when the white balance or color matrix in the camera is changed, the color map may also change. In addition, in the case where there are more local changes in the image frame, the local tone mapping may also change. In addition, if the day / night filter is changed, the color map may also be changed.

[0025] In an embodiment, the at least one image parameter may include a resolution of the image. In other words, if the resolution is caused by, for example, pixel density differences or scaling factor differences, the re-identification threshold may be adjusted to account for such differences.

[0026] In an embodiment, at least one image parameter may include a perspective or pose of an object in the image. That is, if the pose of the object or the perspective of the object changes between a first set of image frames and a subsequent set of image frames, the re-identification threshold may be adjusted to account for the perspective of such a pose change.

[0027] In one possible implementation, only one image parameter is used. That is, only the difference between one image parameter associated with a first ReID feature vector is compared with only one image parameter of a subsequent ReID feature vector. One quantized difference between two image parameters of the same type is used in subsequent steps.

[0028] However, in some embodiments, the method may include quantizing the difference between the set of image parameters of the first set of image frames and the set of image parameters of the subsequent set of image frames. That is, a combination of differences between the respective image parameters is used in the quantization. Then, the adjustment of the re-identification threshold is performed based on the combination of quantized differences between the image parameters.

[0029] In one embodiment, a data storage device may store multiple ReID feature vectors and associated corresponding at least one image parameter, and the method may include: when the quantized difference exceeds a threshold, selecting another ReID feature vector from the data storage device, the image parameter of the other ReID feature vector having the smallest deviation from the at least one image parameter among the stored multiple image parameters of a subsequent set of image frames, and replacing the first ReID feature vector and the linked image parameters with the selected ReID feature vector and the linked image parameters to adjust the re-identification threshold and apply the re-identification algorithm.

[0030] In other words, several subsequent ReID feature vectors are stored together with at least one image parameter to which they are linked. Selecting another first ReID feature vector having an image parameter that better matches one of the subsequent or current ReID feature vectors indicates that a subsequent adjustment of the re-identification threshold can be reduced. In some possible embodiments, the amount or degree of adjustment of the re-identification threshold can be estimated, and if the adjustment exceeds a certain threshold, another ReID feature vector can be selected and used as the first ReID feature vector to adjust the threshold and perform re-identification. "Replacement" does not mean replacement in a data storage device, but means that another ReID feature vector replaces the first ReID feature vector for subsequent steps of the method.

[0031] The re-identification threshold can be adjusted in different conceivable ways.

[0032] In one embodiment, the method may include providing multiple re-identification algorithms trained for different image parameter levels, wherein adjusting the re-identification threshold may include: selecting the re-identification algorithm that best fits the quantified difference between at least one image parameter. That is, different variations of the algorithm, such as different variations or trained networks, are prepared and stored. Different networks are trained under different image parameter conditions, thereby selecting the one network that best fits the set of image parameters. The selected algorithm and network operate with the re-identification threshold adjusted for the set of image parameters. This allows for better customized re-identification algorithms or networks.

[0033] In another embodiment, the method may include providing a re-identification algorithm and adjusting the re-identification threshold of the re-identification algorithm. That is, adjusting the re-identification threshold for a given re-identification algorithm or network. This advantageously requires only a single re-identification algorithm or network.

[0034] In a preferred embodiment, adjusting the threshold may increase the threshold. This advantageously reduces the risk of losing object tracking due to a bad match in the re-identification algorithm caused by differences in image parameter levels.

[0035] In an embodiment, adjusting the threshold may lower the threshold to reduce the risk of false positive re-identification. This avoids the re-identification algorithm from unintentionally generating similar ReID feature vectors for different objects.

[0036] According to a second aspect, a control device is provided for executing the method described in any one of the embodiments disclosed herein.

[0037] Further embodiments and effects of the second aspect of the present invention are substantially similar to the above contents of the first and second aspects of the present invention.

[0038] According to a third aspect of the present invention, there is provided a system comprising a camera for capturing an image of a scene comprising an object and a control unit according to the second aspect.The image acquisition device may be a camera, such as a surveillance camera.

[0039] Further embodiments and effects of the third aspect of the present invention are substantially similar to the above contents of the first and second aspects of the present invention.

[0040] According to a fourth aspect of the present invention, there is provided a computer program product, comprising a program code, which, when executed by a control unit, can execute a method according to any one of the embodiments discussed herein.

[0041] Further embodiments and effects of the fourth aspect of the present invention are very similar to the embodiments and effects of other aspects of the present invention described above.

[0042] A computer program product is further provided, comprising a computer-readable storage medium storing the computer program. The computer-readable storage medium may, for example, be non-transitory and provided as, for example, a hard disk drive (HDD), a solid state drive (SDD), a USB flash drive, an SD card, a CD / DVD and / or any other storage medium capable of non-transitory storage of data.

[0043] Further features and advantages of the present invention will become apparent when studying the appended claims and the following description.Those skilled in the art realize that different features of the present invention can be combined to create embodiments other than those described in the following, without departing from the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Various aspects of the present invention, including its particular features and advantages, may be readily understood from the following detailed description and accompanying drawings, in which:

[0045] Figure 1A Schematically illustrates an image capturing device according to an embodiment of the present invention;

[0046] Figure 1B A convolutional neural network is shown;

[0047] Figure 1C Schematically shows an overview of the neural network used for re-identification;

[0048] Figure 2A The first frame and object detection are shown;

[0049] Figure 2B A second frame and object detection are shown;

[0050] Figure 3 is a flow chart of method steps according to an embodiment of the present invention;

[0051] Figure 4A shows a first frame according to an embodiment of the present invention;

[0052] Figure 4B shows subsequent frames according to an embodiment of the present invention;

[0053] Figure 5 shows a data storage for storing a ReID feature vector linked to an image parameter according to an embodiment of the present invention;

[0054] Fig. 6A A control unit according to an embodiment of the present invention is shown;

[0055] Figure 6B shows a control unit according to an embodiment of the present invention; and

[0056] Figure 7is a flow chart of method steps according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The present invention will be described more fully below with reference to the accompanying drawings, in which currently preferred embodiments of the present invention are shown. However, the present invention may be implemented in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided for thoroughness and completeness and to fully convey the scope of the present invention to those skilled in the art. The same reference numerals refer to the same elements throughout.

[0058] Turning now to the drawings, in particular Figure 1A , shows a scene 1 monitored by an image capture device 100 (eg, a camera, or more specifically, a surveillance camera). In the scene 1, there is a set of objects 102a to 102c, which are exemplified herein as people present in the scene 1.

[0059] The camera 100 continuously monitors the scene 1 by capturing a video stream of images of the scene 1 and objects 102a to 102c therein. The camera 100 and the control unit 106 are part of a system 110, wherein the control unit 106 may be a separate stand-alone control unit or part of the camera 100. It is also conceivable that the control unit 106 is located remotely, such as on a server, and thus operates as a cloud-based service.

[0060] The camera 100 may be mounted on a building, on a utility pole, or in any other suitable location, depending on the particular application. Further, the camera 100 may be a fixed camera or a movable camera, such as a pan, tilt and zoom camera, or even a body-worn camera. Further, the camera 100 may be a visible light camera, an infrared (IR) sensitive camera, or a thermal (long wavelength infrared (LWIR)) camera. Further, image acquisition devices employing lidar and radar functionality are also possible. It is also conceivable that the camera 100 is a combination of the above-mentioned camera technologies.

[0061] The camera 100 further comprises an image capture module 202 , an image processing pipeline 204 , an encoder 206 , data storage 208 and an optional input and output interface 210 configured as a communication interface between the camera 100 and the network 114 via the radio link 112 .

[0062] The image capture module 202 includes various components, such as a lens and an image sensor, wherein the lens is adapted to project an image onto the image sensor including a plurality of pixels.

[0063] The image processing pipeline 204 is configured to perform various operations on image frames received from the image sensor. These operations may include filtering, demosaicing, color correction, noise filtering for removing spatial and / or temporal noise, distortion correction for removing the effects of, for example, barrel distortion, global and / or local tone mapping (e.g., enabling imaging of scenes containing a wide range of intensities), transformations (e.g., rotations), flat field corrections (e.g., for removing the effects of vignetting), overlay applications (e.g., privacy masks, explanatory text, etc.). However, it should be noted that some of these operations, such as transformation operations, such as correction of barrel distortion, rotation, etc., may be performed by one or more modules, components, or circuits disposed outside the image processing pipeline 204, such as in one or more units between the image processing pipeline 204 and the encoder 206.

[0064] After the image processing pipeline 204, the image frame is forwarded to the encoder 206, where the image frame is encoded according to the encoding protocol and forwarded to the receiver, such as the client 116 and / or the server 118, through the network 114 using the input / output interface 210. It should be noted that Figure 1A The camera 100 shown in FIG. 1 also includes a number of other components, such as a processor, memory, etc., which are common in conventional camera systems and whose purpose and operation are well known to those of ordinary skill in the art. Figure 1A These components are omitted from the illustration and description.

[0065] The camera 100 may also include a data storage 208 for optionally storing data related to the video stream capture. Thus, the data storage may store the captured video stream. The data storage may be a non-volatile memory, such as an SD card.

[0066] There are many traditional video coding formats. Some common video coding formats that work with various embodiments of the present invention include: JPEG, Motion JPEG (MJPEG), High Efficiency Video Coding (HEVC), also known as H.265 and MPEG-H Part 2, Advanced Video Coding (AVC), also known as H.264 and MPEG-4 Part 10, Versatile Video Coding (VVC), also known as H.266, MPEG-I Part 3 and Future Video Coding (FVC), VP9, ​​VP10, and AOMedia Video 1 (AV1), to name just a few examples.

[0067] Typically, the control unit 106 runs an algorithm for object detection and for determining a feature vector of a detected object. Such an algorithm can be selected from a range of algorithms, including convolutional neural networks (CNNs), recursive neural networks, decision tree classifiers, such as random forest classifiers that are also effective for classification. In addition, classifiers such as support vector machine classifiers and logistic regression classifiers are also conceivable. The algorithm for object detection and for determining a feature vector of a detected object can be run downstream of the image processing pipeline 204. However, it is conceivable that the algorithm runs within the image processing pipeline 204, or even runs upstream of the image processing pipeline 204, depending on the type of image data on which the algorithm is trained.

[0068] Figure 1B The schematic diagram shows an overview of the neural network prediction process. An image 101 acquired by a camera is input into a neural network model 107, which is exemplified herein as a convolutional neural network CNN model. The image 101 includes image data indicating objects 105a to 105c belonging to different object categories, such as persons 105a, 105c and vehicles 105b.

[0069] The operation of the neural network is considered to be known to those skilled in the art and will not be described in detail. However, typically, for a CNN model, convolution of the input is used to calculate the output. Local connections 109 are formed so that portions of the input layer are connected to nodes in the output. In each layer 111 of the convolutional neural network, a filter or kernel is applied, whereby the parameters or weights of the filter or kernel are learned during the training of the convolutional neural network.

[0070] The neural network 107a may be a single shot multi-box detector with enhanced blocks (SSD-EMB). In this case, the input to the EMB may be a feature map produced from a convolutional layer. The output of the EMB is used as the input to the next convolutional layer. The SSD-EMB network itself is known to those skilled in the art.

[0071] Based on its training, the neural network model 107a provides an output of detection, which can then be provided as input to another neural network 107b, which is trained for re-identification and Figure 1C The detection includes an object class and a crop or bounding box that includes the image patch 108a-c of the detected object.

[0072] The control unit 106 may run an object detection algorithm or network, which may be trained based on annotated training data representing different feature categories, for classifying features in the scene as belonging to a particular feature category based on its training. Object detection is typically performed downstream of the image processing pipeline 204 but upstream of the encoder 206, or even on the server 118. Classification may also be performed upstream of the image processing pipeline 204, depending on the type of image data the algorithm was trained on.

[0073] To train a neural network, such as CNN 107a, a data set is input to the network, the data set comprising images with image patches annotated with at least one corresponding class belonging to the class of the feature or object in the image patch. Images annotated with different classes are provided to the neural network, which feeds back its predictions and performs a validation step or a training step to improve its predictions. More specifically, the neural network can back-propagate the gradient of its loss function to improve the model accuracy performance.

[0074] Figure 1C An overview of a neural network 107b for identification is schematically illustrated. Image patches 108a and 108c of objects 105a, 105c that are expected to be tracked herein are fed to the neural network 107b. The neural network 107b generates ReID feature vectors for the detected objects 105a, c in the image patches 108a and 108c based on its training. Note that while the discussion herein is primarily focused on tracking people, it is also conceivable that the present invention is applicable to other objects, such as a vehicle 105b, in which case the image patch 108b is fed to the neural network 107b. Each image patch can be input to a respective neural network 107b, rather than having to be input to a common neural network for all image patches. The network 107b can be a classifier network, which can be a CNN.

[0075] Figure 2A A first image frame 112 is shown, and Figure 2B A subsequent image frame 114 is shown. For re-identification, subsequent feature vectors are compared, and if they are sufficiently similar, the corresponding objects are considered to be the same. Figure 2A The first ReID feature vector generated by the first detection of the person 102a shown in the frame 112 can be compared with Figure 2B14, where person 102d is in a new position compared to the position of object 102a in image frame 112. If the first ReID feature vector and the second ReID feature vector are similar according to the re-identification threshold, the detection of person 102a in frame 112 is considered to belong to the same person as the detection of person 102d in frame 114, whereby person 102d can be re-identified as being the same as 102a, and thus the person can be tracked.

[0076] Several factors affect the accuracy of a re-identification algorithm in correctly re-identifying an object. If, for example, brightness or other image parameters affect the algorithm's ability to accurately detect and correctly identify an object. Embodiments of the present invention aim to alleviate this problem.

[0077] Embodiments of the present invention will now be described in more detail with reference to the subsequent drawings.

[0078] Figure 3 1 is a flow chart of method steps according to an embodiment of the present invention. The method is a computer-implemented method for re-identifying an object in a camera (eg, camera 100).

[0079] In step S102, an object is detected in a first set of image frames of a video stream captured by a camera by an object detection algorithm (e.g., an object detection algorithm executed by the control unit 106). The first set of image frames can be considered as a track, i.e., a short instance of an object. An image frame 412 in the first set of image frames is Figure 4A 102a and 102c are shown as including objects 102a and 102c.

[0080] In step S104, at least one image parameter of the first set of image frames is determined by the control unit 106. The image parameter may be one or several different types of image parameters. For example, the at least one image parameter may include one or more of the illumination level of the image, the color mapping of the image, the resolution of the image, the object viewing angle, and the pose of the object in the image. The pose and viewing angle may be determined by finding and estimating skeletons / key points in the image.

[0081] Image parameters may be determined by analyzing the image or parameters of a camera imaging algorithm that controls, for example, exposure time, gain, white balance, color matrix, or day / night filters, to name a few examples.

[0082] In step S106 , a first ReID feature vector describing the detection of an object in a first set of image frames including frame 412 is determined.

[0083] The eigenvector A(x) can be expressed as:

[0084] A=[a1,a2,a3,a4,a5…,ax ..,a n ]

[0085] The feature vector A includes n feature indices x. Vector instances a1 to a n is a numerical value.

[0086] In step S108 , at least one image parameter is stored in the data storage 208 together with the first ReID feature vector. Figure 5 It shows the storage of ReID feature vectors ReID1 to ReID N and related image parameters P1 to P N The data storage 208 of the structured table 402 is stored in the image parameter symbol P. In this way, the image parameter symbol P is associated with the ReID feature vector ReID. Note that the image parameter symbol P may include one or more image parameters associated with the corresponding ReID feature vector.

[0087] In step S110, an object in a subsequent set of image frames of a video stream captured by the camera is detected by an object detection algorithm running on the control unit 106. The subsequent set of image frames may be a second track. Figure 4B One image frame 414 from the set of subsequent image frames is shown in FIG.

[0088] Further, image crops 416 and 418 of the respective detection objects 102a and 102b are shown.

[0089] In step S112 , at least one image parameter of a subsequent set of image frames is determined by the control unit 106 .

[0090] In step S114, the control unit 106 determines at least one other ReID feature vector, ReID N+1 , which describes the detection of objects in a subsequent set of image frames.

[0091] In step S116, the difference between at least one image parameter of the first image frame set and at least one image parameter of the subsequent image frame set is quantified. Figure 5 As shown, the subsequent ReID feature vector ReID N+1 The image parameter P N+1 Compared with the previous first ReID feature vector (e.g. ReID N )’s image parameter P N Make a comparison.

[0092] For example, one may consider using brightness statistics to quantify brightness levels, such as summing the brightness of a block of pixels (eg, a 16×16 pixel block).

[0093] The brightness or light level of one or more image frames in a first trajectory may be calculated across all pixels, or more preferably only for a crop of detected objects. Similarly, the brightness of a subsequent set of image frames or trajectories is calculated, preferably only for a crop of detected objects. The quantized difference may be a subtraction between two calculated brightnesses or brightnesses extracted from brightness statistics. In a similar manner, the cumulative light level across image crops may be considered.

[0094] As another example of an image parameter, the resolution of the crops 416 and 418 and / or 420 and 422 may be determined. The resolution may be affected by, for example, the scaling between subsequent detections required to provide a suitable image size for a neural network performing re-identification. The difference or ratio between the crop resolutions may be used to quantify the difference.

[0095] Additionally, quantifying color mapping differences can be accomplished by calculating the sum of absolute differences (SAD) between the color matrices required to perform color mapping. Color mapping algorithms typically use color matrices to transform image colors to target colors. By comparing color matrices, quantification of color mapping differences can be obtained.

[0096] In addition, the pose or viewpoint of the detected object can be estimated by analyzing the skeleton points of the detected object. In this article, the quantized difference can include analyzing the image frame in a pose-specific neural network to model the pose and calculate the transformation required to compensate for the different poses. The quantization of the required transformation can be used as the quantized difference.

[0097] Although quantifying the difference between image parameters is preferably applicable to one image parameter type at the time, it is conceivable to consider sets of image parameters simultaneously or together. That is, the control unit 106 may quantify the difference between the image parameter set of a first image frame set and the image parameter set of a subsequent image frame set.

[0098] For example, consider crop 416 in image frame 412 and crop 418 in image frame 414. Between frames 412 and 416, the brightness or color mapping, hue, or resolution has changed, such that the appearance or image condition of image crop 418 is different from the appearance or image condition of crop 416. Figure 4A It is shown in Figure 4A In the crop 420 of the frame 412 and in Figure 4B The object 102 c captured in the crop 422 in the frame 414 of FIG. 4 , the pose and perspective of the detected object 102 c have changed.

[0099] In step S118, at least one re-identification algorithm is provided, which is configured to take the first ReID feature vector and another ReID feature vector as input to determine whether the object in the first set of image frames is the same as the object in the subsequent set of image frames according to a re-identification threshold.

[0100] The at least one re-identification algorithm may be at least one neural network 107b.

[0101] In step S120, the re-identification threshold is adjusted by the control unit 106 to take into account the quantized differences between the image parameters associated with the first re-identification feature vector and the image parameters of the subsequent set of image frames.

[0102] The threshold may be adjusted according to the quantized difference q between the image parameters. For example, the adjustment D of the threshold may be:

[0103] D(q)=q(P1,…Pk,..Pn)

[0104] Where q(Pk) is the quantified difference as a function of the image parameter Pk. This function can be established mathematically or determined empirically from testing. A lookup table can store the re-ID thresholds versus the quantified differences between multiple sets of image parameters. For example, using test video sequences of a single known person at different imaging parameter settings, the statistics of the qualification and required thresholds can be used to generate a lookup table of re-ID thresholds versus the quantified differences.

[0105] As another example, the adjustment of the threshold can be determined or calibrated by first considering different use cases for a particular neural network, i.e., implementation settings or objects to be tracked. For a given use case, the network may receive images that are identical but have different image parameter settings, such as different brightness, resolution, color, etc. Preferably, one image parameter is considered at a time. For one image parameter, this can be performed for a series of known image parameter settings to achieve image parameter differences. The neural network calculates a ReID feature vector for each different image parameter setting. The image parameter differences can be associated with their respective ReID feature vector differences, for example in a graph or table, as determined by the distance or paradigm discussed herein. The distance or paradigm can be, for example, an L2 paradigm, or any desired paradigm. In this way, it is possible to investigate image parameter differences for which the ReID algorithm (i.e., the neural network) can no longer correctly re-identify a person or object in an image. This is an indication of image parameter differences for which the threshold needs to be adjusted to account for image parameter differences. Therefore, the threshold should be adjusted until the neural network reliably re-identifies the person or object. The process can also be advantageously repeated under the current use case in a real setting.

[0106] Typically, relatively larger quantized differences result in relatively larger thresholds, and relatively smaller quantized differences result in relatively smaller thresholds.

[0107] In one embodiment, adjusting the threshold may increase the threshold. This is advantageous when one wishes to maintain the track even when image conditions change. For example, the threshold may be a so-called L2 norm. Consider another ReID feature vector B represented by:

[0108] B=[b1,b2,b3,b4,b5…,b x, .,b n ]

[0109] For re-identification, the feature vectors B and A are input into a metric function or a norm function to estimate the similarity between the ReID feature vectors B and A. The L2 norm provides the Euclidean distance between the vectors, which is given by:

[0110]

[0111] The norm d is compared to a threshold to determine whether the feature vectors are considered to belong to the same object. For example, if d is less than the threshold, two object detections represented by ReID feature vectors B and A are considered to be detections of the same object, while if d is greater than the re-identification threshold, the objects are considered different. In cases where there are relatively large quantitative differences between image parameters, the re-identification threshold can be increased, allowing for larger deviations between the ReID feature vectors B and A but still resulting in a positive identification.

[0112] In one embodiment, Fig. 6A The control unit 106 is shown running a single re-identification algorithm 602, thereby providing only one re-identification algorithm. Adjusting the re-identification threshold means adjusting the threshold 602 of the re-identification algorithm.

[0113] In one embodiment, Figure 6B The control unit 106 shown may use multiple re-identification algorithms 604a to 604d trained for different image parameter levels. Adjusting the re-identification threshold herein includes selecting the re-identification algorithm that is most suitable for the quantified difference between at least one image parameter among the re-identification algorithms 604a to 604d.

[0114] It is further conceivable that the control unit 106 may lower the threshold, in particular to reduce the risk of false positive re-identification when necessary.

[0115] In step S122, a re-identification algorithm, ie, the re-identification algorithm 602 with an adjusted threshold or a selected one of the re-identification algorithms 604a to 604d, is applied by the control unit to estimate whether the object in the first set of image frames is the same as the object in the other set of image frames 416.

[0116] In step S124, the control unit provides an output of the estimation result in the re-identification algorithm, which may be a signal of a graphical user interface indicating the tracking process.

[0117] See also Figure 7 The flowchart and Figure 5 In some cases, it may occur that the quantized difference between at least one image parameter of the current (ie, subsequent) image frame set and at least one image parameter of the first image frame set is relatively high, exceeding a certain threshold. Figure 5 , where the ReID feature vector and at least one image parameter of the subsequent image frame set are represented by ReID N+1 and P N+1 Represented by, and the ReID feature vector and at least one image parameter of the first image frame set can be represented by ReID N and P N If the image parameter P N+1 and P N If the quantized difference between them exceeds a threshold, then in step S117a, another ReID feature vector is selected from the data storage device 208 storing multiple ReID feature vectors and at least one corresponding linked image parameter, and the image parameter of the other ReID feature vector deviates minimally from the at least one image parameter among the multiple stored image parameters of the subsequent image frame set.

[0118] That is, the control unit 106 may search the database 402 for the best matching image parameter P k , which means that at least one image parameter P N+1 With at least one selected image parameter P k The quantitative difference between all image parameter sets P N minimized.

[0119] Once at least one new image parameter P is found k In step S117b, the selected ReID feature vector ReID k and at least one image parameter P to which it is linked k The first ReID feature vector and the linked image parameters are replaced to adjust the re-identification threshold and apply the re-identification algorithm. That is, the method uses the selected ReID feature vector ReID in step S118. kand at least one image parameter P to which it is linked k conduct.

[0120] A computer program comprising instructions is further provided, which, when executed by a computer, can cause the computer to perform the method of any embodiment disclosed herein. The computer program can be stored or distributed on a data carrier. As used herein, a "data carrier" can be a temporary data carrier, such as a modulated electromagnetic wave or light wave, or a non-temporary data carrier. Non-temporary data carriers include volatile and non-volatile memories, such as permanent and non-permanent storage media of magnetic, optical or solid-state type. Still within the scope of "data carrier", such memory can be fixedly mounted or portable.

[0121] The control unit comprises a microprocessor, a microcontroller unit, a programmable digital signal processor or another programmable device. The control unit may also or alternatively comprise an application specific integrated circuit, a programmable gate array or programmable array logic, a programmable logic device or a digital signal processor. Where the control unit comprises a programmable device, such as the microprocessor, microcontroller unit or programmable digital signal processor described above, the processor may further comprise computer executable code for controlling the operation of the programmable device.

[0122] The control functions of the present disclosure can be implemented using an existing computer processor, or implemented by a dedicated computer processor for an appropriate system, for this purpose or another purpose, or implemented by a hard-wired system. Embodiments within the scope of the present disclosure include a program product, which includes a machine-readable medium for carrying or storing machine-executable instructions or data structures. This machine-readable medium can be any available medium that can be accessed by a general or special-purpose computer or other machine with a processor. For example, this machine-readable medium can include RAM, ROM, EPROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to carry or store the required program code in the form of machine-executable instructions or data structures and can be accessed by a general or special-purpose computer or other machine with a processor. When information is transmitted or provided to a machine via a network or another communication connection (hardwired, wireless or a combination of hardwired or wireless), the machine regards the connection as a machine-readable medium. Therefore, any such connection is appropriately referred to as a machine-readable medium. The above combination is also included in the scope of machine-readable media. Machine-executable instructions include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer or a special-purpose processing machine to perform a specific function or group of functions.

[0123] Although the accompanying drawings may show a sequence, the order of the steps may be different from that described. In addition, two or more steps may be performed simultaneously or partially simultaneously. This variation will depend on the selected software and hardware system and the designer's choice. All of these variations are within the scope of the present disclosure. Similarly, software implementations can be completed using standard programming techniques, which have rule-based logic and other logic to complete various connection steps, processing steps, comparison steps, and decision steps. In addition, although the present invention has been described with reference to specific exemplary embodiments of the present invention, it will be apparent to those skilled in the art that a number of different changes, modifications, etc. will become apparent.

[0124] Furthermore, by studying the drawings, the disclosure and the appended claims, those skilled in the art can understand and implement variations of the disclosed embodiments in practicing the claimed invention. Furthermore, in the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality.

Claims

1. A computer-implemented method for object re-identification in a surveillance camera, the method comprising: detecting, by an object detection algorithm, an object in a first set of image frames of a video stream captured by the camera; determining at least one image parameter of the first set of image frames; determining a first ReID feature vector describing the detection of the object in the first set of image frames; storing the at least one image parameter together with the first ReID feature vector in a data storage device; detecting, by the object detection algorithm, an object in a subsequent set of image frames of the video stream captured by the camera; determining at least one image parameter of the set of subsequent image frames; determining at least one further ReID feature vector describing the detection of the object in the set of subsequent image frames; quantifying a difference between the at least one image parameter of the first set of image frames and the at least one image parameter of the subsequent set of image frames; providing at least one re-identification algorithm, the re-identification algorithm being configured to take the first ReID feature vector and the another ReID feature vector as input to determine whether the object in the first set of image frames is the same as the object in the subsequent set of image frames according to a re-identification threshold; adjusting the re-identification threshold to account for quantized differences between the image parameters associated with the first ReID feature vector and the image parameters of the subsequent set of image frames; applying a re-identification algorithm to estimate whether the object in the first set of image frames is the same as an object in another track of image frames; as well as An output is provided in the re-identification algorithm for evaluating the result.

2. The method according to claim 1, wherein: The data storage device stores a plurality of ReID feature vectors and the linked corresponding at least one image parameter, and the method comprises: When the quantized difference exceeds a threshold, selecting from the data storage device another ReID feature vector whose image parameter deviates minimally from the at least one image parameter of the plurality of stored image parameters of the subsequent set of image frames, and The first ReID feature vector and the linked image parameters are replaced by the selected ReID feature vector and the linked image parameters to adjust the re-identification threshold and apply the re-identification algorithm.

3. The method of claim 1, comprising providing a plurality of re-ID algorithms trained for different image parameter levels, wherein: Adjusting the re-identification threshold includes: A re-identification algorithm is selected that is most suitable for the quantified difference between the at least one image parameter.

4. The method of claim 1, comprising providing a re-identification algorithm and adjusting the re-identification threshold for the re-identification algorithm.

5. The method according to claim 1, wherein: The at least one re-identification algorithm is at least one neural network.

6. The method according to claim 1, wherein: The at least one image parameter comprises a brightness level of the image.

7. The method according to claim 1, wherein: The at least one image parameter comprises a color map of the image.

8. The method according to claim 1, wherein: The at least one image parameter comprises a resolution of the image.

9. The method according to claim 1, wherein: The at least one image parameter comprises a viewpoint or a pose of an object in the image.

10. The method according to claim 1, wherein: Adjusting the threshold is increasing the threshold.

11. The method according to claim 1, wherein: Adjusting the threshold is to lower the threshold to reduce the risk of false positive re-identification.

12. The method according to claim 1, comprising: A difference between the set of image parameters of the first set of image frames and the set of image parameters of the subsequent set of image frames is quantified.

13. A control unit for executing the method according to claim 1.

14. A system comprising a surveillance camera for capturing an image of a scene including an object and a control unit according to claim 13.

15. A non-transitory computer readable medium comprising program code for performing the method of claim 1 when executed by a control unit.

Images