Evaluation device for re-identification and corresponding method, system and computer program

Abstract

Examples of the present disclosure relate to an evaluation device for re-identification and a corresponding method, system, and computer program. The evaluation device (10) includes a processing circuit (14) configured to obtain a plurality of transformed re-identification codes, each transformed re-identification code being associated with a timestamp and location information. Each transformed re-identification code is transformed based on a similarity-preserving transformation of the re-identification codes, the re-identification code representing at least a portion of a sample of media data, the media data originating from two or more different sources located at two or more different locations. The processing circuit is configured to match the transformed re-identification codes among the plurality of transformed re-identification codes using a similarity metric to generate one or more transformed re-identification code tuples that are similar according to the similarity metric. The processing circuit is configured to determine, based on the timestamp and location information associated with the transformed re-identification codes in the one or more transformed re-identification code tuples, one or more position sequences associated with the transformed re-identification codes in the corresponding tuples. The processing circuit is configured to provide information about the one or more position sequences.

Description

Technical Field

[0001] Examples of the present disclosure relate to an evaluation device for re-identification (re-ID) and a corresponding method, system, and computer program. Background Art

[0002] Many methods for visually tracking people rely on dedicated cameras with overlapping fields of view (e.g., using proprietary stereo vision sensors). However, such methods typically require dedicated cameras and may not support tracking between cameras with non-overlapping fields of view. Furthermore, these methods may only support downward-pointing camera orientations within a limited range of heights. Another approach to tracking people is based on using Bluetooth or Wi-Fi to track the person's mobile phone. Another approach uses infrared light for obstacle avoidance. Summary of the Invention

[0003] Various examples of the present disclosure are based on the discovery that a re-identification system that does not typically track the identity of a recorded person (or object) can be used to track the movement of said person or object across locations in a closed system with multiple non-overlapping cameras. In contrast to identification systems that seek to determine the absolute identity of a person (typically from facial features), a visual person re-identification system aims to distinguish or re-identify a person solely from their appearance. For example, the use of re-identification in a system may arise from a lack of a priori knowledge of the actual identities of all relevant persons, but the use of re-identification may also be due to externally enforced privacy policies. Tracking a person can be accomplished by generating so-called re-identification codes from images that represent the person (or object) perceptible within the image. In a re-identification system, for a given person, these re-identification codes should be similar in multiple images taken across multiple locations, allowing the evaluation device to track the sequence of locations that the person has visited.

[0004] Another observation is that some re-identification systems can be abused retroactively if a given re-identification code is associated with an absolute identity of a person and if the re-identification code remains unchanged across time and / or locations. Therefore, additional effort may be required (e.g. using strongly protected and closed systems) to securely store and transmit the re-identification codes, which may lead to additional implementation complexity, especially in systems with many cameras where the re-identification codes are transmitted to a central server for re-identification. It is also possible to store the re-identification codes for future analysis. This additional effort can be avoided if, instead of using the known re-identification codes directly, transformed re-identification codes are used (which are based on transformation functions that can vary over time and / or across locations). These transformed re-identification codes are still suitable for re-identification and also for tracking the sequence of locations visited by a person (or object), but since the generated transformation codes can be designed to be dissimilar across time and / or locations, the risk of the person being identified retroactively is avoided.

[0005] Various aspects of the present disclosure relate to an evaluation device for re-identification. The evaluation device includes processing circuitry configured to obtain a plurality of transformed re-identification codes. Each transformed re-identification code is associated with a timestamp and location information. Each transformed re-identification code is based on a similarity-preserving transformation of a re-identification code representing at least a portion of a media data sample. The media data originates from two or more different sources located at two or more different locations. The processing circuitry is configured to match the transformed re-identification codes among the plurality of transformed re-identification codes using a similarity metric to generate one or more transformed re-identification code tuples that are similar according to the similarity metric. The processing circuitry is configured to determine, based on the timestamp and location information associated with the transformed re-identification codes in the one or more transformed re-identification code tuples, one or more position sequences associated with the transformed re-identification codes in the corresponding tuples. The processing circuitry is configured to provide information regarding the one or more position sequences. By using the transformed re-identification codes, retroactive identification of a person can be prevented, enabling the transmission, processing, and storage of the re-identification codes in environments with less stringent security requirements. In an alternative example, the proposed concept can also be implemented directly using re-identification codes that have not been transformed using a similarity-preserving transformation function. In this case, the storage and / or transmission of the re-identification codes may use additional security measures.By grouping the re-identification codes according to the similarity measure, a path across locations (e.g., a path of a person, animal, or object) may be determined, and a corresponding sequence may be generated accordingly.

[0006] For example, each re-ID code and the corresponding transformed re-ID code may represent a person or object perceptible in the media data sample. In other words, the proposed evaluation device can be used to track the movement of people in a large space (e.g., an airport or a store), or to track the movement of objects (e.g., vehicles, for example, at a complex intersection).

[0007] In general, the proposed method can be used to derive patterns and statistics from the movement of people or objects, for example, to track the average time required to reach an airport gate or estimate the additional time required in traffic. A major factor in such statistics or patterns is the time it takes to access the sequence of locations. Therefore, the processing circuit can be configured to determine information about the time span associated with the one or more sequence of locations based on the timestamp associated with the transformed re-identification code in the one or more transformed re-identification code tuples.

[0008] In some cases, the proposed concept can be used for a space where both the entrance and exit are covered by two or more different sources. In this case, the proposed system can be used to track the complete presence of a person or object represented by the transformed re-identification code. For example, the two or more different locations can be part of a delimited space. The two or more different locations can cover at least the entrance and exit of the delimited space.

[0009] In this disclosure, the proposed concepts are primarily presented in the context of visual person re-identification. Thus, each re-identification code and the corresponding transformed re-identification code may represent a person perceived in the media data sample. However, the proposed concepts may also be used for other purposes, such as object re-identification or animal re-identification. Thus, each re-identification code and the corresponding transformed re-identification code may represent an object or animal perceived in the media data sample.

[0010] As described above, the proposed concept can be used for statistical purposes, for example to provide information about the average time required to visit the sequence of locations. However, in addition to time, additional information can also be taken into account, such as demographic information about the person represented by the transformed re-identification code. For example, each transformed re-identification code can be associated with demographic information about the person. The processing circuit can be configured to compile statistical information related to one or more location sequences and / or statistical information related to the time span associated with one or more location sequences. The statistical information can be aggregated based on the demographic information. For example, people with different demographic information may take different paths, stay at different locations, or require different amounts of time between locations.

[0011] The concepts described herein have various applications, for example, for safety reasons. For example, the processing circuitry may be configured to determine the presence of one or more persons within the delimited space based on the one or more transformed re-identification code tuples. This information may be used to determine whether the delimited space is full (e.g., in terms of the maximum number of persons allowed in the delimited space) or, in the event of a fire, to determine whether a person is present in the delimited space.

[0012] Additionally or alternatively, the processing circuit can be configured to detect the presence of a queue of people within the delimited space based on the determined one or more position sequences. The queue of people can be detected to determine whether to open another counter (e.g., at a security checkpoint at an airport, or another cashier counter in a store).

[0013] In some examples, the processing circuitry can be configured to generate an alarm signal based on the determined one or more position sequences. For example, an alarm can be issued if someone breaches a barrier at an airport or if someone proceeds to an exit without paying for goods obtained from another department.

[0014] Another application of the proposed concept is to find missing persons. For example, a child may enter a delimited space with a parent, such as a store or an airport. Within the delimited space, the child becomes lost. Since two people entered the space together, two re-identification codes can be linked together (i.e., associated with each other), and one of the re-identification codes can be used to search for the other (e.g., in video surveillance material). In other words, the processing circuitry can be configured to store an association between transformed re-identification codes representing a first person and a second person who have entered the delimited space together. The processing circuitry can be configured to retrieve a transformed re-identification code representing a second person based on the transformed re-identification code of the first person and based on the stored association. The processing circuitry can be configured to locate the second person based on the retrieved transformed re-identification code of the second person.

[0015] If the exact time at which two persons entered the space is unknown, or if one wishes to obtain all surveillance video showing them in the space, a transformed re-identification code for the query person can be generated and used to locate the query person and, optionally, the second person in the video surveillance footage. For example, the processing circuitry can be configured to generate the transformed re-identification code based on another media data representing the person. The processing circuitry can be configured to locate the person within the bounded space based on a determined sequence of positions associated with the transformed re-identification code in a tuple that includes a transformed re-identification code similar to the generated transformed re-identification code.

[0016] To retrieve the surveillance footage (ie, media data), the following tasks may be performed: The processing circuit may be configured to compile the media data on which the transformed re-ID code in the tuple (which includes a transformed re-ID code similar to the generated transformed re-ID code) is based.

[0017] In general, the proposed evaluation device can be used for surveillance, security monitoring, or in-store analytics concepts. The results of the evaluation device can be presented to a user of one of these systems, for example, via a visualization on a computer dashboard or as part of an alert. Thus, the processing circuitry can be configured to generate a display signal including a visualization of information regarding the one or more position sequences.

[0018] Typically, the re-identification code can be transformed by the similarity-preserving transformation such that if the re-identification code is similar to another re-identification code according to a similarity metric, then the transformed re-identification code is also similar to another transformed re-identification code that is a transformed version of the other re-identification code. In other words, the transformation can be performed such that subsequent re-identification is unbiased and satisfies the equivalence class preservation property.

[0019] In various examples, the re-identification code is transformed based on a time-dependent transformation parameter. For example, the transformation parameter can be gradually changed or periodically changed over time. This can prevent the transformed re-identification code from being misused across time and / or space.

[0020] The re-identification system can be used with various types of media. In addition to image data of people, the re-identification system can also be applied to images of vehicles or animals, or other types of media can be used together. For example, the media data can be one of image data, video data, audio data, a three-dimensional representation of an object's motion, and text-based media data. In various examples, the re-identification code is generated using a hashing algorithm, wherein the hashing algorithm is constructed such that if the hashing algorithm is applied to several media data samples representing the same person, animal, or object, similar re-identification codes are obtained. The above examples of media data types can be used with a suitable re-identification system and hashing algorithm.

[0021] For example, the processing circuit may be configured to obtain the plurality of transformed re-identification codes from two or more camera devices, or the processing circuit may be configured to obtain the plurality of transformed re-identification codes from a central device connected to the plurality of camera devices for generating the transformed re-identification codes.

[0022] Various aspects of the present invention relate to corresponding (computer-implemented) evaluation methods for re-identification. The method includes obtaining a plurality of transformed re-identification codes. Each transformed re-identification code is associated with a timestamp and position information. Each transformed re-identification code is based on a similarity-preserving transformation of re-identification codes representing at least a portion of a media data sample. The media data originates from two or more different sources located at two or more different locations. The method includes matching the transformed re-identification codes among the plurality of transformed re-identification codes using a similarity metric to generate one or more transformed re-identification code tuples that are similar according to the similarity metric. The method includes determining one or more position sequences associated with the transformed re-identification codes in the corresponding tuples based on the timestamp and position information associated with the transformed re-identification codes in the one or more transformed re-identification code tuples. The method includes providing information about the one or more position sequences.

[0023] Various aspects of the present disclosure relate to a computer program having a program code for performing the above-mentioned method when the computer program is executed on a computer, a processor or a programmable hardware component.

[0024] Various aspects of the present disclosure relate to a system comprising at least one apparatus for generating transformed re-identification codes and the aforementioned evaluation device. The at least one apparatus is configured to generate the plurality of transformed re-identification codes based on media data originating from two or more different sources. The evaluation device is configured to provide information about the one or more position sequences based on the plurality of transformed re-identification codes provided by the at least one apparatus. For example, each source may include a separate apparatus for generating the transformed re-identification codes, or a central apparatus for generating transformed re-identification codes may be used to generate transformed re-identification codes for multiple sources. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Some examples of apparatus and / or methods will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0026] Figure 1a A schematic block diagram illustrating an example of an evaluation device for re-identification is shown;

[0027] Figure 1b A schematic flow chart illustrating an example of an evaluation method for re-identification;

[0028] Figure 1c shows a schematic diagram of a system with three cameras and an evaluation device;

[0029] Figure 1d An example of a visualization provided by an evaluation device is shown;

[0030] Figure 1e Another example of a visualization provided by an evaluation device is shown;

[0031] Figure 2a and Figure 2b A schematic block diagram showing an example of an apparatus for re-identification or an apparatus for generating a transformed re-identification code; and

[0032] Figure 3a and Figure 3b A schematic block diagram shows an example of a system including at least one device for re-identification or at least one device for generating a transformed re-identification code. DETAILED DESCRIPTION

[0033] Some examples are now described in more detail with reference to the accompanying drawings. However, other possible examples are not limited to the features of the embodiments described in detail. Other examples may include modifications of these features as well as equivalent and alternative features of these features. In addition, the terms used in this article to describe certain examples should not limit other possible examples.

[0034] Throughout the description of the drawings, the same or similar reference numerals refer to the same or similar elements and / or features, which may be implemented in the same or modified form while providing the same or similar functions. For clarity, the thickness of lines, layers and / or regions in the drawings may also be exaggerated.

[0035] Unless explicitly defined otherwise in individual cases, when "or" is used to combine two elements A and B, this will be understood to disclose all possible combinations, namely only A, only B, and A and B. As alternative wording for the same combination, "at least one of A and B" or "A and / or B" can be used. The same applies to combinations of more than two elements.

[0036] If singular forms such as "a", "an", and "the" are used, and there is no explicit or implicit definition that the use of only a single element is mandatory, alternative examples may also use several elements to achieve the same function. If a function is described below as being implemented using multiple elements, alternative examples may use a single element or a single processing entity to achieve the same function. It should also be understood that the terms "comprises" and / or "comprising" when used to describe the presence of specific features, integers, steps, operations, processes, elements, components and / or groups thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, processes, elements, components and / or groups thereof.

[0037] In the following, an example is given that illustrates the basic principles of re-identification. In this example, two surveillance cameras are used. Two images are obtained from these cameras, in each of which a single person can be seen. These images can be taken from different angles and under different lighting conditions (or even from the same camera but at different points in time). Even without knowing any actual identity, the re-identification system seeks to infer whether the two images depict the same person, or whether the images actually show two different people. Furthermore, re-identification is not limited to the detection of people in images. For example, re-identification can be performed on animals or objects, and other types of media such as three-dimensional models, text or audio can also be used.

[0038] There are multiple use cases where re-ID can be used instead of a system that creates absolute identities, such as for filtering duplicate detections when counting unique visitors at a specific location during a day, for estimating travel time between two different locations using sensors at each endpoint, for calculating the dwell time of a customer at a single location, for wide-area people tracking using non-overlapping cameras, etc.

[0039] In some systems, re-ID is performed by applying a hash function to each image to generate a so-called re-ID code. The generated hash code represents the person, animal, or object visible within the corresponding image and can be compared using a similarity metric.

[0040] Various examples of the present disclosure extend the concept of re-identification codes by adding an "encryption layer" on top of the re-identification code to generate a transformed re-identification code. The hash function applied, along with the transformation, exhibits a so-called "equivalence class preserving" (ECP) property, which can be an isometric or "(almost) distance preserving" property, in that the distances between the transformed re-identification codes are at least similar, if not equal, to the distances between the re-identification codes on which the transformed re-identification codes are based. The ECP property is similar to that of locality-sensitive hashing (LSH), described below.

[0041] Expressed mathematically, let f be such a hash function, e be an ECP-preserving transformation function (encryption function), and let h1 = e(f(I1)) and h2 = e(f(I2)) be the re-ID codes for two images I1 and I2, respectively. The ECP property ensures that the two transformed re-ID codes are approximately similar in some suitable similarity metric—that is, h1 ≈ h2 if and only if the person in image I1 is the same person as in image I2 (even if the images were taken from different angles). Therefore, after evaluating the hash function for two images, if the distance between the two re-ID codes is small, it can be inferred that they are the same person. On the other hand, if the distance is large, then it is likely that the images contain different people.

[0042] Various methods can be used to implement such a visual re-ID system. Many systems use hand-crafted visual features (such as gender, age, facial features, clothing color, hairstyle, body shape, etc.), but to achieve the highest possible accuracy, many methods rely on deep learning-based techniques such as triplet loss. However, for the purpose of explaining the concept, the exact method used to calculate the re-ID code is not important.

[0043] In some systems, the re-ID code (short for re-identification code) for a specific person may be the same at different times, potentially allowing for absolute identification of the individual. This may be because traditional re-ID codes remain consistent over time. For example, if a person is imaged by a specific camera at a specific time t1, they may be identified at a much later time t2, potentially leading to knowledge of their personal identity, where an image may be obtained from their passport or driver's license, for example. If re-ID is applied in this manner, it may inadvertently enable non-anonymous identification of individuals. In many systems, this identification feature is undesirable because it may lead to data leakage. It is not possible to mitigate this disadvantage using direct cryptographic methods, such as encrypting the re-ID code, without losing the desired re-ID capability. Because individual devices relying on previous approaches may not provide privacy by design, distributed re-ID systems can, for example, maintain confidentiality of all communications and securely store re-ID values ​​by using secure devices or communicating over a secure network. In a distributed multi-camera system with multiple communicating devices, this can add additional burden to the overall system design. Furthermore, this problem may not be mitigated using traditional cryptographic methods. Almost all existing encryption methods lack the "equivalence class preservation" property due to the cryptographic avalanche effect. A small change in the unencrypted data results in a large change in the encrypted data, and thus re-identification capabilities may not be maintained. In other words, some re-identification methods may not provide "privacy by design."

[0044] Various examples of the present disclosure employ different approaches, where re-identification across multiple locations or over time may be limited by design. Examples of the present disclosure can provide a method for anonymous re-identification, and thus, can be "privacy by design" (by periodically or locally, efficiently, and irreversibly changing the re-identification hash function). This can enable insecure transmission and storage of re-identification codes without violating privacy regulations, as privacy can be preserved even when the transformed codes are stored in arbitrary storage systems without privacy guarantees. The proposed concept is based on augmenting existing (non-privacy-enhancing, non-anonymous, commonly used) re-identification hash functions with a transformation capability (e.g., using additional specialized cryptographic primitives). In contrast to traditional cryptographic schemes for confidentiality, the proposed concept does not eliminate the ability to perform re-identification even after the re-identification code is encrypted (by maintaining the ECP property). In contrast to some other systems, matching of re-identification codes can be performed on unsecured servers or even by untrusted third parties. Furthermore, the transformed re-identification codes can be stored indefinitely in untrusted databases without sacrificing privacy. The proposed concept can improve the concept of re-identification since it allows to enhance existing re-identification algorithms with "privacy by design" by adding new additional primitives on top of them. The proposed concept can combine existing or novel re-identification systems with (dynamic) cryptographic primitives (i.e., transformation functions) having ECP properties (on top of the re-identification system).

[0045] In the following, it is assumed that the system for calculating and subsequently encrypting the re-identification code is embedded in the camera. This is not necessary for the proposed concept to work, but is useful for illustrating a use case. While the method can be useful for embedded or edge devices, it can also be used for cloud-based implementations, where the subsequently presented apparatus or evaluation device is implemented.

[0046] Upon transforming the re-identification code, an evaluation device is provided that uses the transformed re-identification code to determine a sequence of locations visited by a person, animal, or object. Various aspects of this disclosure provide an apparatus for anonymous customer analysis, for example, using multiple non-overlapping cameras. This disclosure proposes several applications of anonymous re-identification, with in-store analysis being a primary example.

[0047] Various examples of the present disclosure are based on the use of information sources (e.g. cameras) that do not require stereo vision or overlapping fields of view. By using re-identification, analysis can be performed across spatial locations without having to cover the entire space. Therefore, in various examples, existing camera installations can be used. The proposed concept only analyzes the current feed (which can also be used for its original purpose or for new purposes). Existing camera feeds can be used simultaneously for other purposes, such as theft monitoring, advanced behavioral analysis, or demographic analysis (age / gender). Moreover, if cameras are used, these do not need to be dedicated (e.g., stereo vision is not required). The cameras can also be installed at wider angles and over a wider range of heights. Existing concepts and / or installations can be improved by using anonymous re-identification based on machine learning / deep learning.

[0048] In the following, it is assumed that a functional system for (visual) anonymous re-identification is available. For example, in combination with Figures 2a to 3b Such a system is shown, wherein a device for generating a transformed re-identification code for anonymous re-identification is given, and a system having such a device and a corresponding evaluation device (for example in combination with Figures 1a to 1e given) system.

[0049] Figure 1a A schematic block diagram of an example of an evaluation device 10 for re-identification is shown. The evaluation device includes processing circuitry 14. Optionally, the evaluation device includes an interface 12 and one or more storage devices 16. The processing circuitry is coupled to the optional interface and one or more storage devices. Typically, the functionality of the evaluation device is provided by, for example, the processing circuitry in conjunction with the interface (for exchanging information) and / or one or more storage devices (for storing information).

[0050] The processing circuit is configured to obtain a plurality of transformed re-identification codes (e.g., via interface 12 and / or from one or more storage devices 16). Each transformed re-identification code is associated with a timestamp and position information. Each transformed re-identification code is transformed based on a similarity-preserving transformation of re-identification codes representing at least a portion of a media data sample. The media data originates from two or more different sources located at two or more different locations. The processing circuit is configured to match the transformed re-identification codes among the plurality of transformed re-identification codes using a similarity metric to generate one or more transformed re-identification code tuples that are similar according to the similarity metric. The processing circuit is configured to determine one or more position sequences associated with the transformed re-identification codes in the one or more transformed re-identification code tuples based on the timestamp and position information associated with the transformed re-identification codes in the corresponding tuples. The processing circuit is configured to provide information about the one or more position sequences (e.g., via interface 12).

[0051] Figure 1b A schematic flow chart of an example of a corresponding (computer-implemented) evaluation method for re-identification is shown. The method includes providing a plurality of transformed re-identification codes 110. The method includes matching transformed re-identification codes among the plurality of transformed re-identification codes using a similarity metric 120 to generate one or more transformed re-identification code tuples that are similar according to the similarity metric. The method includes determining one or more position sequences associated with the transformed re-identification codes in the one or more transformed re-identification code tuples based on timestamps and position information associated with the transformed re-identification codes in the corresponding tuples 130. The method includes providing information about the one or more position sequences 140.

[0052] The following description involves Figure 1a The evaluation device 10 and Figure 1b The corresponding method.

[0053] Various aspects of the present disclosure relate to evaluation devices, evaluation methods, and corresponding computer programs for re-identification (i.e., for use in re-identification). Typically, a re-identification system includes multiple components, namely, components for generating re-identification codes (or, in various examples of the proposed concept, transformed re-identification codes), and components for evaluating these re-identification codes to perform the actual re-identification. In this document, the term re-identification refers to the re-identification (but not identification) of something (e.g., a person, animal, or object), i.e., re-recording a previously recorded person, animal, or object and matching it with the previous recording. In the proposed concept, re-identification is based on a so-called re-identification code, which is a code representing a person, animal, or object perceptible in some type of media data. In the context of the present disclosure, the primary focus is on visual person re-identification systems, i.e., systems in which visual media data (e.g., image data) is recorded and the person visible in the visual image data is being re-identified. However, the same concepts can also be applied to other types of media data, and can also be applied to animals or objects, provided that a suitable hash function is selected for generating the re-identification code. For example, each re-identification code and the corresponding transformed re-identification code may represent a person, animal, or object perceptible in the media data sample. On the other hand, a person, animal, or object may be represented by a set of similar re-identification codes. Thus, the media data may be one of image data, video data, audio data, and a three-dimensional representation of a person or object's motion. However, in various examples, the media data itself is not analyzed by the evaluation device. Instead, the media data is processed by an entity that generates a re-identification code representing at least a portion of the media data (e.g., a person, animal, or object perceptible in the media data). Thus, in combination Figure 2a The analysis of media data is described in more detail in the apparatus for generating a transformed re-identification code.

[0054] The processing circuit is configured to obtain a plurality of transformed re-identification codes. As described above, the means for generating such transformed re-identification codes may be, for example, a combination of Figures 2a to 2b As shown below, such a device can be integrated into the camera device 200, such as Figure 2b and Figure 1c As shown (in cameras 151, 154, 157). The apparatus may accept media data at its input and provide a transformed re-identification code based on the media data provided at the input. If such an apparatus is integrated into a camera device, the transformed re-identification code may be obtained (e.g., received) from the respective camera device. Thus, the processing circuitry may be configured to obtain a plurality of transformed re-identification codes from two or more camera devices 200. Alternatively, a central apparatus may be used to process media data from a plurality of cameras, and a plurality of transformed re-identification codes may be obtained from the central apparatus. In some examples, the evaluation device may comprise a central apparatus (e.g., in combination with Figure 2a and Figure 2b As already pointed out above, the same principle can also be applied to other sources, such as microphones or 3D sensors.

[0055] exist Figure 1c , a system is shown, wherein the transformed re-identification code is provided by a corresponding camera device. Figure 1c The system is shown comprising at least one device for generating a transformed re-identification code (integrated in the camera devices 151 , 154 and 157 ) and an evaluation device 10 . Figure 1c In FIG, a person 150 moves from an entrance to an exit in a space. The person is recorded by a camera device 151, 154, 157, which generates image data 152, 155, 158 of the person, which image data is then used to generate corresponding transformed re-identification codes 153, 156, 159, which are transmitted to the evaluation device 10. In other words, at least one device (i.e. a device integrated in a camera device) can be configured to generate a plurality of transformed re-identification codes based on media data originating from two or more different sources (i.e. two or more camera devices). With reference to the more specific example of a camera, the media data can be one of image data or video data. The media data can originate from two or more camera devices 151, 154, 157 (or as Figure 3aIn this document, the term "source" indicates that the media data is generated by two or more different sources. The evaluation device can be configured to provide information about one or more position sequences based on a plurality of transformed re-identification codes provided by at least one device.

[0056] In various examples, a plurality of transformed re-identification codes may be obtained from the database 18 (e.g., Figure 3a and / or Figure 3b As shown. For example, database 18 can be a database accessible by multiple devices via a defined interface (e.g., via a computer network and / or according to a predefined protocol). For example, multiple devices can be configured to provide the transformed re-identification codes generated by the multiple devices to the database. In some examples, the database can be external to the evaluation device and accessed, for example, via a computer network. However, in some examples, the evaluation device can include database 18.

[0057] As noted above, the re-identification code used in the context of the present disclosure is a transformed re-identification code, i.e., a re-identification code that has been transformed based on a similarity-preserving transformation of the re-identification code. In some examples, the transformation may be applied directly by the device that generated the re-identification code on which the transformed re-identification code is based. Figure 2a and Figure 2b An example of a transformation is given, wherein an apparatus for generating such a transformed re-identification code is shown. Typically, the transformation is a similarity-preserving transformation, i.e., the re-identification code is transformed by a similarity-preserving transformation such that if a re-identification code is similar to another re-identification code according to a similarity metric, then the transformed re-identification code is also similar to the transformed re-identification code which is a transformed version of the other re-identification code. Figure 2a and Figure 2b In more detail, the re-identification code may be transformed based on a transformation parameter that depends on one of time and location. For example, the transformation parameter may be changed gradually or periodically over time or across locations, i.e., gradually based on a linear interpolation between two transformation parameters, or at a fixed time or location. Figure 2a and Figure 2b Discover more details.

[0058] Each transformed re-identification code is associated with a timestamp and location information. In this document, the term "associated" indicates that the transformed re-identification code is received together with the corresponding timestamp and location information, or that the transformed re-identification code is annotated by the processing circuit with the corresponding timestamp and location information based on its arrival time and source. Typically, the timestamp relates to the time when the media data on which the corresponding transformed re-identification code is based is recorded, or the time when the corresponding transformed re-identification code is received (for example, if the transformed re-identification code is obtained substantially in real time). Therefore, the processing circuit can be configured to determine the timestamp of the corresponding transformed re-identification code based on the time when the transformed re-identification code is obtained. Similarly, the location information indicates the location of the media data on which the corresponding transformed re-identification code is based. For example, the location identifier can include an identifier for identifying the location (for example, an identifier of a camera providing media data, which is linked to actual information by the evaluation device) or coordinates according to a coordinate system. For example, if the transformed re-identification code is obtained from a known source (for example, a device of a known camera device), the processing circuit can be configured to determine the location information based on the source from which the transformed re-identification code is obtained.

[0059] The processing circuit is configured to match the transformed re-identification code among the plurality of transformed re-identification codes using a similarity metric to generate one or more transformed re-identification code tuples that are similar according to the similarity metric. Typically, a tuple is a set of values ​​containing zero or more elements in an ordered manner. However, in some examples, since only the adherence of samples to the tuple may be relevant to the generation of the position sequence, the order of the samples within the tuple is not meaningful. Therefore, the described tuples may also be understood as "sets" or "groups" of samples. Alternatively, the order within the tuple may be defined by a time order based on a timestamp associated with the transformed re-identification code. Furthermore, each tuple may contain one or more samples (unless a non-zero tuple has one or more samples removed).

[0060] The processing circuit is thus configured to group the transformed re-identification codes together using a similarity metric. Specifically, the processing circuit may be configured to match transformed re-identification codes having a similarity that exceeds or matches a predefined similarity threshold according to the similarity metric. In this document, the term "matching" indicates comparing corresponding transformed re-identification codes and grouping similar transformed re-identification codes together to form a tuple. For example, the transformed re-identification code may be represented as a string of hexadecimal values, such as Figure 1cAs shown. The processing circuit can be configured to compare each pair of transformed re-identification codes digit by digit (digit means individual hexadecimal values) and determine another string of hexadecimal values ​​consisting of the differences between the individual digits. The sum of the values ​​of the other string can then indicate the similarity between the two transformed re-identification codes, and the larger the sum, the larger the total difference and the lower the similarity. Alternatively, another system is selected, such as a system that considers the entire hexadecimal value instead of evaluating the digits separately. In some examples, the difference also represents the "distance" between the two transformed re-identification codes. By matching the transformed re-identification codes using a similarity metric, the transformed re-identification codes can be grouped together when grouped into one or more tuples so that the re-identification codes assigned to the same tuple are similar according to the similarity metric.

[0061] The obtained transformed re-identification codes are used to match similar transformed re-identification codes together, while the timestamp and location are used to create a location sequence. The processing circuit is configured to determine one or more location sequences associated with the transformed re-identification codes in the one or more transformed re-identification code tuples based on the timestamp and location information associated with the transformed re-identification codes in the corresponding tuples. For example, the processing circuit can be configured to sort the transformed re-identification codes in the one or more tuples according to the associated timestamps and generate a location sequence in an order defined by the associated timestamps based on the location information associated with the transformed re-identification codes in the tuples. Alternatively, the transformed re-identification codes may have been added to the tuples in an order defined by the timestamps associated with the corresponding transformed re-identification codes (e.g., based on the order and time at which the transformed re-identification codes were obtained by the evaluation device). In general, each location sequence can be associated with the transformed re-identification code used to generate the location sequence, and therefore also associated with the corresponding person, animal, or object represented by the corresponding transformed re-identification code. In short, each location sequence can be associated with a person, animal, or object represented by the transformed re-identification code used to generate the corresponding location sequence.

[0062] Once the location sequences are determined, they can be processed to compile knowledge about the location sequences, such as the time it takes for a person (or object) to visit the location sequences. For example, in a scenario where the behavior of a person in a store or another location (e.g., an airport, an oil rig, or an amusement park) is evaluated, the time spent in the store can be calculated. In other words, the processing circuitry can be configured to determine information about a time span associated with one or more location sequences based on the timestamps associated with the transformed re-identification codes in the one or more transformed re-identification code tuples, for example, to determine the time it takes for one or more people (or animals or objects) associated with the one or more location sequences to visit the location sequences, thereby determining the dwell time.

[0063] The system is not only applicable to retail stores, but also to airports, oil drilling platforms, chemical plants, factories, companies, amusement parks, etc. The system can be used for public order (tracking intruders), security (holding people accountable in emergencies and other situations), health (for example, the number of people present in a room, which can be limited to limit the spread of infectious diseases), traffic analysis of smart buildings and smart cities, and general analysis by architects and building owners. In general, this concept can be applied to any delimited (i.e., closed) space. In other words, two or more different locations can be part of a delimited space. In the context of the present disclosure, a delimited space or closed space can be defined as a space in which the entrance and / or exit are in two or more different locations covered by two or more sources (e.g., two or more cameras). Therefore, as long as the entrance and exit are in two or more different locations, the delimited space can also be an outdoor space. In other words, two or more different locations can at least cover one (or each) entrance and one (or each) exit of the delimited space.

[0064] In general, the proposed concept can be used to track people within a delimited space (e.g., as they travel from an entrance to an exit). If dwell time is the only statistic required, then two cameras per store entrance are sufficient, one facing outward and one facing inward. Customer re-identification hashes (i.e., re-identification codes) can be calculated upon entry and exit. Combined, this provides information about the dwell time (i.e., the time span associated with the corresponding location sequence). Thus, the processing circuitry can be configured to determine the dwell time of the individual represented by the transformed re-identification code based on the time span associated with the corresponding location sequence. The same approach can be used to calculate the number of unique visitors. This number can correspond to the number of tuples that include a first transformed re-identification code based on media data originating from the entrance of the delimited space and a second transformed re-identification code based on media data originating from the exit of the delimited space. After a person is detected at the exit, the person can be counted as a unique visitor, and the tuple can be discarded (or, alternatively, discarded at the end of the day).

[0065] To determine more granular information about the locations visited, more than two cameras can be set up in a bounded space. For example, cameras can be set up in strategic locations in a bounded space (e.g., a retail store). In addition to important locations in the bounded space, all entrances and exits can also be covered.

[0066] By using the anonymous re-identification concept, a person (e.g., a customer) can be re-identified across multiple non-overlapping cameras. If the camera locations are known in advance and the time and re-ID code for each customer are recorded, the customer's plausible movement path through the store can be reconstructed. Thus, two or more cameras can have non-overlapping fields of view, or the field of view of at least one of the two or more cameras can have zero overlap with any field of view of the other two or more cameras. Furthermore, because the system is anonymous, it provides "privacy by design" and therefore does not violate privacy regulations.

[0067] The proposed concept can be combined with image-based demographic analysis to give results based on age or gender. For example, if combined with age estimation, a histogram of different dwell times for various age groups can be generated. In other words, each transformed re-identification code can be associated with demographic information about the person (e.g. (estimated) age group or (estimated) gender). For example, the transformed re-identification code can be obtained together with the associated demographic information. For example, another machine learning-based function (which can be provided by the device providing the transformed re-identification code) can be used to estimate the age group or gender of the person represented by the transformed re-identification code. The processing circuit can be configured to compile statistical information related to one or more position sequences and / or statistical information related to time spans associated with one or more position sequences. For example, the statistical information may include a histogram of the time spans that individuals spent within a delimited space (e.g. as Figure 1d Alternatively or additionally, the statistical information may include a statistical breakdown of the paths (or routes) taken by individuals within the bounded space (e.g., as shown in FIG. Figure 1e ), or a statistical breakdown of the aisles visited within the bounded space (if the bounded space is a store). For example, the statistical information can be aggregated based on demographic information. In other words, individual histograms can be generated within the statistical information for different age groups and / or genders. Furthermore, the paths or aisles visited can be broken down by age group or gender within the statistical information.

[0068] A general system for many use cases can be created by exporting all camera-specific, anonymized, timestamped re-identification codes to an external database. This allows application programmers without any image analysis or deep learning experience to create advanced applications in which customer movement can be analyzed or even used for real-time alerts such as queue detection, theft alarms, emergency situations, etc. For example, the processing circuit can be configured to determine the presence of one or more people within the delimited space based on one or more transformed re-identification code tuples (e.g., by identifying that at least one of the tuples does not contain a transformed re-identification code based on media data originating from an exit of the delimited space). Additionally or alternatively, the processing circuit can be configured to detect the presence of a queue of people within the delimited space based on one or more determined location sequences (e.g., by identifying that multiple tuples include transformed re-identification codes associated with the same location information and the same timestamp). In some examples, for example, if it is determined that too many unique people exist within a portion of the delimited space, or if the location sequence includes a high-value location (e.g., a jewelry counter) and an exit but no cash register, the processing circuit can be configured to generate an alarm signal based on the determined one or more location sequences. Thanks to the anonymized re-ID codes, the database can be stored with a lower security level while complying with privacy laws and regulations, such as the General Data Protection Regulation (GDPR).

[0069] Another specific application of the system is tracking children who have gone missing in an amusement park or shopping mall. For example, if both a mother and a child arrive but the child is missing, the mother's re-ID code (from her driver's license or by taking a photo of her) can first be used to obtain the child's re-ID code, and then the child can be tracked. In other words, the processing circuit can be configured to store an association between the transformed re-identification codes of a first person (e.g., a mother) and a second person (e.g., a child) who have entered a delimited space together. For example, if two transformed re-identification codes with the same timestamp and the same location information indicating the entrance to the delimited space are detected during matching of the transformed re-identification codes, the processing circuit can be configured to store the association. The processing circuit can be configured to retrieve the transformed re-identification code representing the second person based on the transformed re-identification code of the first person and based on the stored association. For example, the association can be stored together with two tuples including the two transformed re-identification codes. The processing circuit can be configured to locate the second person based on the retrieved transformed re-identification code of the second person. For example, the processing circuit may be configured to retrieve the latest transformed re-identification code from the tuple including the transformed re-identification code representing the second person, and obtain the location information associated with the transformed re-identification code.

[0070] In order to search for a re-identification code tuple, a transformed re-identification code for the first person may be generated. For example, the processing circuit may be configured to generate a transformed re-identification code based on another media data representing a person (e.g., a second person). The transformed re-identification code may then be used to retrieve a transformed re-identification code for the second person. Alternatively, the transformed re-identification code may be used to locate the person within the delimited space based on a determined sequence of positions associated with the transformed re-identification codes in the tuple (which includes transformed re-identification codes similar to the generated transformed re-identification code). In this way, an image of the missing person may be sufficient to locate the person within the delimited space. Alternatively, the position of the person may not be determined, and instead the media data on which the corresponding transformed re-identification code is based may be retrieved.

[0071] In some examples, the system can be used to comply with another aspect of privacy laws and regulations. In many cases, privacy laws and regulations may allow a customer to request all video material recorded by a store where the customer is located. Such requests typically result in a significant amount of manual work, where the store owner must review the video material for a given day. Using re-ID (not necessarily anonymous re-ID), a system can be created to automatically perform this process. For example, the processing circuit can be configured to compile the media data on which the transformed re-identification code is based in a tuple (which includes a transformed re-identification code similar to the generated transformed re-identification code). For example, each transformed re-identification code can be associated with information about the media data on which the transformed re-identification code is based, which can be used to retrieve the media data. Alternatively, the media data can be retrieved based on the timestamp and location information associated with the corresponding transformed re-identification code.

[0072] In the previous sections, some applications of the proposed concept have been introduced. These applications have in common that one or more position sequences are analyzed and information about the person, animal, or object represented by the corresponding re-identification code (i.e., the results of the analysis) is compiled. Once the information is compiled, it can be provided by the evaluation device. In other words, the processing circuit is configured to provide information about one or more position sequences. Generally, the information about the one or more sequences can include the results of any of the above applications, such as statistical information, information about time spans, the location of a person, or retrieved media data.

[0073] Typically, information about one or more position sequences can be provided as a visualization of the results. In other words, the processing circuit can be configured to generate a display signal that includes a visualization of the information about the one or more position sequences. Typically, the display signal can be a control signal for a monitor or display, including a control instruction that causes the monitor or display to display a visualization. Alternatively, the display signal can be any type of digital signal that includes a visualization. In particular, the display signal can include an interpretable code for generating a visualization on a display device. For example, the display signal can include an interpretable code for generating a visualization in a web browser of a computing device. For example, the visualization can show the results of the re-identification performed by the evaluation device 10.

[0074] There are various types of results that can be visualized. For example, the evaluation device can be configured to determine the time span associated with the sequence of positions of different persons, animals, or vehicles between two different locations using the transformed re-identification codes generated based on the image data of the sensors at each endpoint. Thus, the visualization can show a visual representation of the time spans of different persons, animals, or vehicles and / or a statistical evaluation thereof.

[0075] Alternatively or additionally, the evaluation device may be configured to filter duplicate detections when counting unique visitors at a particular location during the day based on the transformed re-identification code. The visualization may show a visual representation of the (filtered) unique visitor count at the particular location and / or a statistical evaluation thereof, such as a histogram over time.

[0076] In some examples, the evaluation device can be configured to calculate the dwell time of the customer at a single location based on the transformed re-ID code (e.g., using a time span associated with the sequence of locations). Accordingly, the visualization can show a visual representation of the dwell time of the customer at the location and / or a statistical evaluation thereof, such as a histogram.

[0077] Figure 1d An example of a visualization provided by an evaluation device is shown. Figure 1d In FIG, the visualization shows anonymized store statistics and includes two parts: a first part 160, which includes text information such as the number of unique visitors and the median length of stay; and a second part 165, which has a histogram of the length of stay. Figure 1d The histogram is generated in four-minute time intervals (intervals of at least 1 minute and at most 5 minutes, more than 5 minutes and at most 9 minutes, more than 9 minutes and at most 13 minutes, etc.).

[0078] The evaluation device can be configured to perform wide-area person tracking using non-overlapping cameras based on the transformed re-identification code. For example, a two-dimensional image and a three-dimensional representation of a person can be used for tracking. Accordingly, the visualization can show a visual representation of the tracked person. Figure 1e Another example of a visualization provided by an evaluation device is shown. Figure 1e , an anonymized in-store customer flow 170 is shown. In the visualization, two separate customer flows (#1 and #2) are shown, leading from an entrance to an exit along two or more locations.

[0079] In summary, the proposed concept uses an (anonymous) re-ID system for generating (transformed) re-identification codes. Based on this, multi-camera tracking can be performed by recording a time-stamped re-ID code for each camera. All events can be exported to a common database of all cameras. On the basis of the database, various analytical applications can then be created. The various aspects of the evaluation device involve multi-camera tracking and, optionally, analytical applications, while Figure 2a and Figure 2b The device presented in [1] involves the generation of a transformed re-identification code. Specialized analytical applications can tailor the product to specific use cases.

[0080] Interface 12 may correspond to one or more inputs and / or outputs for receiving and / or transmitting information within a module, between modules, or between modules of different entities. This information may be a digital (bit) value according to a specified code. For example, interface 12 may include interface circuitry configured to receive and / or transmit information. For example, interface 12 may be suitable for communication within camera device 200. Additionally or alternatively, interface 12 may be suitable for communication via a computer network, such as a wireless or wired computer network.

[0081] The processing circuit 14 may be implemented using one or more processing units, one or more processing devices, any means for processing (e.g., a processor), a computer, or a programmable hardware component that can operate in conjunction with corresponding adapted software. In other words, the functions of the processing circuit 14 may also be implemented in software, which is then executed on one or more programmable hardware components. Such hardware components may include general-purpose processors such as a central processing unit (CPU), a digital signal processor (DSP), a microcontroller, and the like.

[0082] In at least some embodiments, the one or more storage devices 16 may include at least one element from the group of computer-readable storage media such as magnetic or optical storage media, for example, a hard drive, a flash memory, a floppy disk, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electronically erasable programmable read-only memory (EEPROM), or a network memory.

[0083] In combination with the proposed concept or one or more examples described above or below (e.g., Figures 2a to 3b ) Briefly describe further details and aspects of the evaluation apparatus, evaluation method, and corresponding computer program. The evaluation apparatus, evaluation method, and corresponding computer program may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0084] Figure 2a and Figure 2b A schematic block diagram of an example of an apparatus 20 for re-identification or an apparatus 20 for generating a transformed re-identification code is shown. The apparatus includes an interface 22 and processing circuitry 24. Optionally, the apparatus includes one or more storage devices 26. The processing circuitry is coupled to the interface and, optionally, the one or more storage devices. Typically, the functionality of the apparatus is provided by, for example, the processing circuitry in conjunction with the interface (for exchanging information) and / or one or more storage devices (for storing information).

[0085] The processing circuit 24 is configured to obtain media data via the interface 22. The processing circuit is configured to generate a re-identification code representing at least a portion of the media data using a hash algorithm. The processing circuit is configured to transform the re-identification code using a transformation function to obtain a transformed re-identification code. The transformation function is configured to transform the re-identification code so that if the re-identification code is similar to another re-identification code generated by the hash algorithm according to a similarity metric, then the transformed re-identification code is similar to another transformed re-identification code that is a transformed version of the other re-identification code. The transformation function is configured to transform the re-identification code based on transformation parameters. For example, the transformation parameters can depend on time and / or location. The processing circuit is configured to provide the transformed re-identification code (e.g., via the interface 22). Figure 2b Also shown is a camera device 200 comprising apparatus 22. For example, the camera device 200 may also comprise an imaging sensor for generating media data (eg, as image data).

[0086] Various aspects of the present disclosure relate to an apparatus for re-identification, specifically, an apparatus for generating a re-identification code for use in a re-identification system. Compared to various other systems, the present apparatus builds upon the general concept of re-identification, adding a transformation layer that preserves re-identification capabilities while preventing the ability to retroactively track a person or object across time or locations. For example, by transforming the re-identification code, the transformation function can encrypt the re-identification code. Thus, various aspects of the present disclosure relate to an apparatus for anonymous re-identification.

[0087] The proposed concept addresses this problem by adding a dynamically changing encryption layer (i.e., a transformation function) on top of the existing re-identification function that maintains the ECP properties. In other words, the proposed concept allows the construction of re-identification codes with built-in time intervals (or location restrictions), outside of which the re-identification code is intentionally rendered ineffective. This approach results in anonymous re-identification and "privacy by design", so that the re-identification value can even be transmitted to insecure devices or over insecure networks, and the re-identification code matching process can be performed at a later point in time by an untrusted server or even a third party. In general, the proposed concept is based on (dynamically) changing the re-identification scheme in an efficient manner (e.g., daily or across locations), such that the re-identification code is inconsistent between different days and / or locations, while still maintaining its consistency during each day or within each location.

[0088] In general, anonymous re-identification is achieved using at least two novel aspects, namely a transformation of the re-identification code that maintains the equivalence class preserving property and the dependence of the transformation on time or location. The first aspect aims to retain the re-identification functionality only between transformed re-identification codes that originate from the same (or similar) transformation parameters, while the second aspect aims to specify the cases where different transformation parameters are used. For example, if different transformation parameters are used for different times, tracking of people across different times can be prevented. Similarly, if different transformation parameters are used for different locations, tracking of people across different locations can be prevented. Therefore, the resulting transformed re-identification codes can be stored and / or transmitted on potentially unsecure systems, thereby reducing the required implementation effort.

[0089] The re-identification code is generated based on media data. Typically, the media data can be video, audio, or audiovisual media data. For example, the media data can be one of image data, video data, audio data, a three-dimensional representation of an object's motion (i.e., three-dimensional body motion), and text-based media data / output. Accordingly, the media data can originate from various types of media data generating devices, such as a camera or camera sensor, a microphone, a three-dimensional scanner, or a text acquisition system.

[0090] The processing circuitry can be configured to obtain media data from one or more media data generating devices, for example, locally or via a computer network. Thus, interface 22 can be or include a local interface for intra-device communication, or interface 22 can be or include an interface for communication via a computer network, such as the Internet or a local network. For example, in some cases, the apparatus can be used to generate re-identification codes for media data from more than one media data generating device, where the media data generating devices are located in different locations. Thus, media data can be obtained from or originate from two or more media data generating devices located in different locations.

[0091] For example, as described above, the apparatus may be part of a camera device 200 that includes a camera sensor 28. In this case, the media data generating device may be the camera device or the camera sensor 28 of the camera device, and the processing circuitry may obtain the media data from the camera sensor 28. However, in some cases, the apparatus may be external to the camera device, with the processing circuitry configured to obtain the media data from a camera external to the apparatus. For example, the apparatus may be implemented in a network device, an edge device (i.e., a network device located near the media data generating device), or a cloud server. In both cases, typically, the processing circuitry is configured to obtain the media data from the imaging sensor of the camera, where the media data is one of image data and video data.

[0092] Depending on the type of media data, a suitable hash algorithm can be selected to generate the re-identification code. The processing circuit is configured to generate a re-identification code representing at least a portion of the media data using a hash algorithm, wherein the hash algorithm is applicable to the media data at hand. The proposed system can also be used in areas other than person re-identification. For example, the proposed concept can be applied to cyclists, cars, luggage, and other objects or animals. For example, "Deep Learning for Person Re-identification: A Survey and Outlook" (2020) by Ye et al. provides an example of a hash algorithm for re-identification based on deep learning. Therefore, the processing circuit can be configured to use a machine learning model (e.g., a deep learning network) to generate the re-identification code. Newer technologies can also be adopted instead of using previously used re-identification systems, or the re-identification systems can be improved to work with (dynamic) encryption primitives.

[0093] There are various types of re-identification codes. In the following, the examples given assume a re-identification code based on a hash value. In particular, an example is given in which the re-identification code is a vector of 128 values. However, other types of re-identification codes can also be used, with appropriate adjustments to the transformation function.

[0094] The re-identification code represents at least a portion of the media data. For example, the re-identification code may represent an object or person represented by the media data. Therefore, the transformed re-identification code may represent the object or person represented by the media data.

[0095] In order to perform the above process without having to periodically rebuild the entire basic re-identification system (which is a very laborious process), an additional layer, namely a transformation function, is added on top of the existing re-identification system. Thus, the processing circuit is configured to transform the re-identification code (generated using an existing or novel re-identification system / algorithm) using the transformation function to obtain a transformed re-identification code. The additional layer of transformation function can, for example, be a key-dependent dynamic encryption primitive with ECP properties, which allows the re-identification properties to be maintained even when the re-identification code is encrypted (in contrast to the classical encryption schemes described above, which lose the re-identification properties).

[0096] Expressed mathematically, f represents an existing (conventional) re-identification system for generating re-identification codes. A new key k is distributed to all devices (e.g., using a conventional public key infrastructure) according to a predefined schedule, such as daily or per location. For example, the key k can be a transformation parameter, or k can be a cryptographic secret from which the transformation parameter is derived. Let e k is a cryptographically secure bijective transformation function with ECP properties, which depends on the shared key k. Then, through the dynamic encryption layer e k The combination of the proposed final anonymous dynamic re-identification function c that operates on the image I is given by k , that is, c k (I) = e k (f(I)).

[0097] In the following, we assume that the key changes every day, that is, the transformation function is time-based. Let k t-1 Represents yesterday's key, k t represents today’s key. Every day, all devices can ensure that the key k from yesterday t-1 and the corresponding encryption function is securely destroyed (e.g., by overwriting the relevant memory and storage area). Because the key is changed every day, it is impossible to compare the re-identification code from yesterday with the re-identification code from today. In other words, the following anonymous re-identification properties are met:

[0098] ck t (I1)≈ck t (I2) and ck t+1 (I1)≈ck t+1 (I2)

[0099] but

[0100] ck t (I1)≠ck t+1 (I2) and ck t (I2)≠ck t+1 (I1)

[0101] In other words, similarity preservation or equivalence class preservation characteristics are satisfied. Therefore, the transformation function is configured to transform the re-identification code so that if the re-identification code is similar to another re-identification code generated by the hash algorithm according to a similarity metric, the transformed re-identification code is similar to another transformed re-identification code that is a transformed version of another re-identification code. On the other hand, if the re-identification code is not similar to another re-identification code generated by the hash algorithm according to a similarity metric, the transformed re-identification code should (or more precisely, in fact) be not similar to another transformed re-identification code. More generally, the transformation function can be configured to transform the re-identification code so that the similarity level between the re-identification code and another re-identification code is equal to the similarity level between the transformed re-identification code and another transformed re-identification code, and the similarity level is based on the similarity metric. For example, if the similarity level is high, the two re-identification codes and the two transformed re-identification codes can be similar respectively, and if the similarity level is low, the two re-identification codes and the two transformed re-identification codes can be dissimilar respectively.

[0102] In various examples, the transformation function can be used to perform a linear transformation on the re-identification code. In other words, the transformation function can be configured to perform a linear transformation on the re-identification code based on transformation parameters. One specific implementation of the linear transformation is a transformation based on a rotation matrix. In other words, the transformation function can be configured to transform the re-identification code using a rotation matrix, where the rotation matrix is ​​based on the transformation parameters. In general, a rotation matrix is ​​a matrix used to perform a rotation (e.g., a rotation of a vector) in a given coordinate space by multiplying the vector by the rotation matrix.

[0103] In the following, a specific construction of a dynamic encryption function with ECP properties is described. For simplicity, it can be assumed that the original re-identification code h is a vector of 128 floating-point values, each floating-point value is in the range -1 to +1. In general, the assumption of using a 128-dimensional vector with floating-point based components can be changed to any dimension and other numeric domains, such as integers or Boolean values. Select e k As the encryption function for the transformation function, the function is composed of e k (h) = R k ×h is given, where R kis a 128·128 random rotation matrix that depends irreversibly on the secret key k, and where the operation × represents a matrix product. Since the rotation is distance-preserving, the resulting function still has the ECP properties required to keep the re-identification process working. R can be generated in a cryptographically secure manner by initializing a secure random number generator with k and then sampling a random 128-dimensional rotation matrix (e.g., by sampling each matrix component from a normal distribution using a cryptographically secure random number generator and then normalizing the matrix by performing a Gram-Schmidt orthogonalization). k . In other words, the elements of the rotation matrix can be based on pseudo-random numbers derived from the transformation parameters (e.g., a cryptographic secret derived from the transformation parameters). The pseudo-random numbers of the generated matrix can then be normalized using Gram-Schmidt orthogonalization. For example, the processing circuit can be configured to generate a rotation matrix based on the transformation parameters by generating pseudo-random numbers and normalizing the resulting matrix. Due to the large number of free parameters in the encryption operator matrix and due to its cryptographically strong construction process, even using a method like R k Such linear operators can also provide sufficient security in most possible attack scenarios.

[0104] Alternatively, the transformation function can be configured to perform a nonlinear transformation of the re-identification code. For example, a more complex hash function can be used instead of matrix multiplication. In some embodiments, deep learning can be used to create more complex, more nonlinear functions (while still maintaining the equivalence class preservation property). In other words, the transformation function can be configured to perform a nonlinear transformation using a machine learning model. For example, the machine learning model can take the re-identification code and transformation parameters as input and provide the transformed re-identification code at the output. In addition, end-to-end training of the combined deep network (the lower-level re-identification system combined with the dynamic encryption scheme) can be used to achieve higher robustness and accuracy.

[0105] Typically, in order to prevent tracking of people or objects across time and / or location, the transformation parameters and therefore the transformation itself depend on time and / or location. Typically, time can refer to the time when the transformation of the re-identification code is performed, or the time when the media data is obtained, as the apparatus can be used for near-instantaneous generation and transformation of the re-identification code. In some cases, these two time instances can be decoupled, for example, the transformation can be applied retroactively to previously generated media data. On the other hand, location can relate to the location where the media originated (for example in a system where an apparatus is used to generate transformed re-identification codes for media data from different media data generating devices located in different locations), or to the location of the apparatus itself (for example, if media data from different media data generating devices are to be processed using the same transformation parameters) or to the location of the apparatus that is co-located with the media data generating device.

[0106] In the following, the implementation of time-dependent transformation parameters is described, followed by the implementation of position-dependent transformation parameters.

[0107] For example, to avoid re-identification across dates and thus the risk of absolute identification, various examples of the present disclosure may utilize transformation parameters that vary over time. Thus, to apply new or adapted transformation parameters over time, the transformation parameters may be adapted based on a schedule. In other words, the processing circuitry may be configured to adapt the transformation parameters according to a predefined schedule. For example, the processing circuitry may be configured to adapt the transformation parameters daily, semi-daily, or weekly, depending on the desired level of privacy. For example, the processing circuitry may be configured to generate or select new transformation parameters according to a predefined schedule. After adapting the transformation parameters, the previously used parameters may be discarded (or more precisely, deleted) to prevent the retroactive regeneration of the transformed re-identification code. In other words, the processing circuitry may be configured to delete the previously used transformation parameters after adapting the transformation parameters. For example, if the transformation parameters or underlying keys are periodically destroyed, the original re-identification codes from previous days may be unavailable even if the device is accidentally compromised at some point in time. Consequently, the resulting transformed re-identification code can be freely shared with anyone without revealing anyone's identity, thereby providing an anonymous re-identification code.

[0108] In some examples, incremental / differential key changes can be employed in the transformation function. For example, if the transformation function depends on time, this will result in a sliding time window of anonymization, rather than the current fixed window, for example, from midnight to midnight. In other words, the processing circuit can be configured to generate two transformation parameters for two time points (e.g., midnight on two different days). The two transformation parameters can be sufficiently different to make it impossible to track a person or object across the two transformation parameters. The processing circuit can be configured to gradually adapt the transformation parameters between the two time points based on linear interpolation between the two transformation parameters. In other words, between the two time points, the transformation parameters used to transform the re-identification code can be gradually adapted based on linear interpolation from the first of the two transformation parameters to the second of the two transformation parameters. Taking the rotation matrix as an example, a first rotation matrix and a second rotation matrix each having multiple elements can be generated. Using linear interpolation, a third rotation matrix can be generated, wherein each element of the third matrix is ​​based on a linear interpolation between corresponding elements of the first matrix and the second matrix. The third matrix can then be normalized.

[0109] Similarly, incremental / differential key changes can be used across various locations within a region. The further apart the locations within a region are, the less similar the resulting re-identification codes may be. For example, as described above, the processing circuitry can be configured to obtain media data from two or more media data generating devices located at different locations. Typically, the processing circuitry can be configured to perform the transformation using different transformation parameters for the two or more media data generating devices located at different locations, for example to make re-identification between transformed re-identification codes generated for media data originating from two locations impossible. However, a third media data generating device can be located between the two media data generating devices. The transformation parameters for the third media data generating device can be selected so that re-identification can be performed on the transformed re-identification codes generated for the media data originating from the first two locations. In other words, the processing circuitry can be configured to obtain media data from first, second, and third media data generating devices located at different locations, wherein the second media data generating device is located between the first media data generating device and the third media data generating device. Similar to the example of gradual adaptation over time, the processing circuitry can be configured to generate transformation parameters for media data obtained from the second media data generating device based on linear interpolation between the transformation parameters for media data obtained from the first and third media data generating devices. As a result, re-identification can be performed between transformed re-identification codes generated based on media data from the first and second media data generating devices, and between transformed re-identification codes generated based on media data from the second and third media data generating devices, but not between transformed re-identification codes generated based on media data from the first and third media data generating devices. In various examples, an even more fine-grained system can be used, in which two transformation parameters are generated for two arbitrary locations located on either side of a group of media data generating devices, and the transformation parameters for the media data generating devices are generated based on position-based linear interpolation between the two transformation parameters.

[0110] In general, there are various options for obtaining suitable transformation parameters. For example, the transformation parameters can be generated on the device side (i.e., by processing circuitry) based on a cryptographic secret, which can be shared between devices configured to generate the same transformation parameters (e.g., depending on time). In other words, the transformation parameters can be derived from a cryptographic secret. Therefore, the processing circuitry can be configured to generate transformation parameters based on a cryptographic secret and based on time and / or position, the transformation parameters being, for example, a rotation matrix or input parameters of a machine learning model employed for the transformation function. For example, a cryptographic secret can be used together with time and / or position to generate a seed for generating a pseudo-random number for the transformation parameters. For example, a cryptographically strong method can be used to construct a new key (i.e., transformation parameter) or encryption method from a single shared secret. Irreversible key generation can be used to improve security. For example, in some examples, irreversible, shared dynamic transformation parameters (i.e., key values) can be used in the absence of a persistent network connection. After sharing the initial secret seed, the following method can be applied: k t =AES(k t-1 , k t-1 ), where the AES(x, k) operation applies the Advanced Encryption Standard to the text x using a key (i.e., a cryptographic secret) k, i.e., today’s key k t , yesterday's key k t-1 Can be encrypted itself and destroyed afterwards.

[0111] Finally, the processing circuit is configured to provide the transformed re-identification code, for example, via interface 22. Typically, the processing circuit can be configured to store the re-identification code, for example, using one or more storage devices or using a database, such as a database accessible by multiple devices via a defined interface (e.g., via a computer network and / or according to a predefined protocol) and / or external to the apparatus and / or camera device. In other words, the processing circuit is configured to provide the transformed re-identification code to the database. For example, the transformed re-identification code can be processed by an evaluation device that accesses the database.

[0112] The proposed concept can provide a combination of computer vision and cryptography. Specifically, secure video analytics can be used on distributed, embedded camera-based systems to provide privacy-enhanced visual person re-identification.

[0113] Interface 22 may correspond to one or more inputs and / or outputs for receiving and / or transmitting information within a module, between modules, or between modules of different entities. This information may be a digital (bit) value according to a specified code. For example, interface 22 may include interface circuitry configured to receive and / or transmit information. For example, interface 22 may be suitable for communication within camera device 200. Additionally or alternatively, interface 22 may be suitable for communication via a computer network, such as a wireless or wired computer network.

[0114] The processing circuit 24 may be implemented using one or more processing units, one or more processing devices, any means for processing (e.g., a processor), a computer, or a programmable hardware component that can operate in conjunction with corresponding adapted software. In other words, the functions of the processing circuit 24 may also be implemented in software, which is then executed on one or more programmable hardware components. Such hardware components may include general-purpose processors such as a central processing unit (CPU), a digital signal processor (DSP), a microcontroller, and the like.

[0115] In at least some embodiments, the one or more storage devices 26 may include at least one element from the group of computer-readable storage media such as magnetic or optical storage media, for example, a hard drive, a flash memory, a floppy disk, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electronically erasable programmable read-only memory (EEPROM), or a network memory.

[0116] In combination with the proposed concept or one or more examples described above or below (e.g. Figure 1a 、 Figure 1e 、 Figure 3a and Figure 3b ), briefly describes the combination Figure 2a and Figure 2b More details and aspects of the camera device or method, apparatus and computer program described herein. The camera device and method, apparatus and computer program may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0117] Figure 3a and Figure 3b Schematic block diagrams of examples of systems 300, 310 including at least one device 20 for re-identification or at least one device 20 for generating a transformed re-identification code are shown. Figure 2a and Figure 2b While introducing a device for generating and transforming a re-identification code, Figure 3a and Figure 3bThe context in which the device may be used is shown. Typically, the system may include a single device 20, such as part of a camera device 200. As shown, Figure 3a The system 300 includes two camera devices 200 (or more generally, two media data generating devices 200 ), each including an apparatus 20 . Figure 3b The system 310 includes a plurality of camera devices 200, each including an apparatus 20. In other words, the system 300, 310 includes two or more apparatuses 20 such as Figure 3b As shown, the apparatus 20 may include one or more additional components, such as a random access memory (RAM) 312 or one or more optional components 314 . Figure 3b The device 20 shown includes a network interface 22a, which may be a combination of Figure 2a and Figure 2b Part of the interface 22 is introduced.

[0118] In addition to the apparatus 20 or a camera device having the apparatus 20, the system may optionally include an evaluation device 10 (e.g. Figures 1a to 1e The evaluation device can be used to perform re-identification using the transformed re-identification code. In other words, Figure 3a and Figure 3b The system shown includes a database 18 (e.g., in conjunction with Figure 2a and Figure 2b The optional evaluation device 10 is coupled to the apparatus 20. Figure 3b As shown, the evaluation device 10 can be implemented as a re-identification code matching server 10 that can communicate with a database 18. The evaluation device can be configured to obtain a transformed re-identification code from at least one device 20 and compare the transformed re-identification code provided by the at least one device according to a similarity metric. For example, the evaluation device can be configured to obtain a transformed re-identification code from at least one device 20 via the database 18. Generally, the evaluation process can be configured to perform re-identification based on the transformed re-identification code of the device. Generally, the evaluation device 10 can be external to the device 20. However, in some examples, one of the devices can include the evaluation device 10. Figure 3b As shown, the evaluation device 10 can provide a visualization 316 to the end user (e.g., as combined with Figures 1a to 1e shown).

[0119] Depending on whether the transformation parameters used by one or more devices are time-dependent or location-dependent, different criteria may be met by the transformation parameters used. For example, the transformation parameters may be time-dependent (i.e., the current time at the device). Two or more devices may be configured to use the same transformation parameters at the same time, for example, so that transformed re-identification codes generated at the same time are suitable for re-identification.

[0120] On the other hand, if the transformation parameters depend on the location, the transformation parameters may be different depending on the location from which the respective media data originate. Figure 2a and Figure 2b As described, one or more (or two or more) devices can be configured to process media data originating from two or more locations and use different transformation parameters for the media data originating from the two or more locations. In addition, for example, in a scenario with three or more media data generating devices, gradual adaptation of the transformation parameters based on the respective locations from which the media data originates can also be applied.

[0121] In combination with the proposed concept or one or more examples described above or below (e.g. Figures 1a to 2b ) to briefly describe more details and aspects of the system. The system may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0122] In general, the concepts presented involve computer vision, machine learning, people counting, in-store analytics, and / or people flow monitoring.

[0123] The aspects and features described with respect to a specific example in the foregoing examples may also be combined with one or more aspects and features of another example to replace the same or similar features of the other example, or to additionally introduce these features into the other example.

[0124] Examples may also be or relate to a (computer) program including program code to perform one or more of the above methods when the program is executed on a computer, processor, or other programmable hardware component. Therefore, the steps, operations, or processes of the different methods in the above methods may also be performed by a programmed computer, processor, or other programmable hardware component. Examples may also encompass program storage devices (e.g., digital data storage media) that are machine, processor, or computer readable and encode and / or contain machine-executable, processor-executable, or computer-executable programs and instructions. Program storage devices may, for example, include or may be digital storage devices, magnetic storage media such as disks and tapes, hard drives, or optically readable digital data storage media. Other examples may also include systems of computers, processors, control units, (field) programmable logic arrays ((F)PLAs), (field) programmable gate arrays ((F)PGAs), graphics processor units (GPUs), application specific integrated circuits (ASICs), integrated circuits (ICs), or systems on chips (SoCs) that are programmed to perform the steps of the above methods.

[0125] It should also be understood that, unless explicitly stated in individual cases or necessary for technical reasons, the disclosure of several steps, processes, operations or functions disclosed in the specification or claims should not be interpreted as implying that these operations must rely on the order described. Therefore, the above description does not limit the execution of several steps or functions to a certain order. In addition, in other examples, a single step, function, process or operation may include and / or be decomposed into several sub-steps, sub-functions, sub-processes or sub-operations.

[0126] If aspects have been described with respect to a device or system, these aspects should also be understood as descriptions of the corresponding method. For example, functional aspects of a block, device, or device or system may correspond to features (e.g., method steps) of the corresponding method. Thus, aspects described with respect to a method should also be understood as descriptions of properties or functional features of the corresponding block, element, device, or system.

[0127] The following claims are hereby incorporated into the detailed description, where each claim can stand on its own as a separate example. It should also be noted that although dependent claims are referred to in the claims as specific combinations with one or more other claims, other examples may also include combinations of dependent claims with the subject matter of any other dependent or independent claims. Such combinations are expressly set forth herein unless a specific combination is stated not to be intended. Furthermore, even if a claim is not directly defined as dependent on any other independent claim, the features of that claim should be included with respect to that other independent claim.

Claims

1. An evaluation device (10) for re-identification, the evaluation device comprising a processing circuit (14), the processing circuit (14) being configured to: Obtain multiple transformed re-identification codes, each transformed re-identification code is associated with a timestamp and location information, in, Each transformed re-identification code is transformed based on the similarity preservation of the re-identification code, The re-identification code represents at least a portion of a sample of media data originating from two or more different sources located at two or more different locations, wherein the re-identification code is transformed based on a transformation parameter that depends on one of time and location; matching transformed re-identification codes among the plurality of transformed re-identification codes using a similarity metric to generate one or more transformed re-identification code tuples that are similar according to the similarity metric; determining one or more position sequences associated with the transformed re-identification codes in the corresponding tuples based on the timestamps and position information associated with the transformed re-identification codes in the one or more transformed re-identification code tuples; and Information about the one or more position sequences is provided.

2. The evaluation device according to claim 1, wherein Each re-ID code and the corresponding transformed re-ID code represent a person or object perceivable in the sample of the media data.

3. The evaluation device according to claim 1 or 2, wherein The processing circuit is configured to determine information about a time span associated with the one or more position sequences based on the timestamp associated with the transformed re-identification code in the one or more transformed re-identification code tuples.

4. The evaluation device according to any one of claims 1 to 3, wherein The two or more different positions are part of a delimited space, and the two or more different positions at least cover an entrance and an exit of the delimited space.

5. The evaluation device according to claim 4, wherein Each re-ID code and the corresponding transformed re-ID code represents a person perceivable in the sample of the media data.

6. The evaluation device according to claim 5, wherein Each transformed re-identification code is associated with demographic information about the person, wherein the processing circuit is configured to compile statistical information related to the one or more position sequences and / or statistical information related to a time span associated with the one or more position sequences, the statistical information being aggregated based on the demographic information.

7. The evaluation device according to claim 5 or 6, wherein The processing circuit is configured to store an association between transformed re-identification codes representing a first person and a second person who have entered the delimited space together, retrieve the transformed re-identification code representing the second person based on the transformed re-identification code of the first person and based on the stored association, and locate the second person according to the retrieved transformed re-identification code of the second person.

8. The evaluation device according to any one of claims 5 to 7, wherein The processing circuit is configured to generate a transformed re-identification code based on another media data representing a person, and to locate the person within the bounded space based on a determined sequence of positions associated with a transformed re-identification code in a tuple that includes a transformed re-identification code similar to the generated transformed re-identification code.

9. The evaluation device according to claim 8, wherein The processing circuit is configured to compile the media data on which the transformed re-identification code in the tuple including a transformed re-identification code similar to the generated transformed re-identification code is based.

10. The evaluation device according to any one of claims 1 to 9, wherein The processing circuit is configured to generate a display signal comprising a visualization of information about the one or more position sequences.

11. The evaluation device according to any one of claims 1 to 10, wherein The re-identification code is transformed by the similarity-preserving transformation such that if the re-identification code is similar to another re-identification code according to the similarity metric, then the transformed re-identification code is similar to another transformed re-identification code that is a transformed version of the other re-identification code.

12. The evaluation device according to any one of claims 1 to 11, wherein The media data is one of image data and video data, wherein the media data originates from two or more camera devices (200) located at two or more different locations.

13. A method for evaluating re-identification, the method comprising: obtaining a plurality of transformed re-identification codes (110), each transformed re-identification code being associated with a timestamp and location information, wherein each transformed re-identification code is transformed based on a similarity-preserving transformation of the re-identification codes, the re-identification code representing at least a portion of a sample of media data originating from two or more different sources at two or more different locations, wherein the re-identification code is transformed based on a transformation parameter that depends on one of time and location; matching transformed re-identification codes among the plurality of transformed re-identification codes using a similarity metric (120) to generate one or more transformed re-identification code tuples that are similar according to the similarity metric; determining, based on the timestamp and position information associated with the transformed re-identification codes in the one or more transformed re-identification code tuples, one or more position sequences associated with the transformed re-identification codes in the corresponding tuples (130); and Information about the one or more position sequences is provided (140).

14. A re-identification system comprising at least one device (20) for generating a transformed re-identification code and an evaluation device (10) according to any one of claims 1 to 12, wherein: The at least one apparatus is configured to generate the plurality of transformed re-identification codes based on the media data originating from the two or more different sources, and wherein the evaluation device is configured to provide information about the one or more position sequences based on the plurality of transformed re-identification codes provided by the at least one apparatus.

Images