Apparatus and method for identifying at least one person in image data and people counting system

Abstract

In one embodiment, an apparatus for identifying at least one person in image data comprises a first data input, a first pre-processing unit and a main processing unit. The first data input is adapted to supply a first sequence of individual images. The first pre-processing unit is configured to generate, in each case, a first motion trail image from a first settable number of consecutive individual images of the first sequence. The main processing unit is configured to evaluate the first motion trail image and to provide an output signal therefrom. The output signal is indicative of a person and / or a number of persons identified in the first motion trail image during the evaluation.

Description

Technical Field

[0001] The present invention relates to an apparatus, a people counting system, and a method for identifying at least one person in image data. Background Technology

[0002] Due to factory automation, both areas become safe for humans and free of dangerous machines or automated vehicles, and areas become unsafe for humans and contain dangerous machines or self-propelled automated vehicles. Special protection must be provided to humans from these dangerous machines or self-propelled automated vehicles. In such environments, it may be necessary to monitor several personnel in hazardous areas.

[0003] This functionality requires the ability to reliably distinguish between people and other objects. Object classification in images is typically achieved based on individual images, aided by an AI-supported image classifier. In dynamic scenes, image series can also be used as input data, as object tracking can be performed simultaneously within them. For example, AI in the form of a neural network searches for relevant objects in individual images, classifies them, and outputs the corresponding locations and associated bounding boxes—the bounding frames surrounding the identified objects or people. Alternatively, other recognition can be performed, such as segmenting pixel regions belonging to objects or people. In classic post-processing steps, these output data from consecutive image datasets are correlated. Correlation is achieved by incorporating the detection results of individual images into a temporal motion sequence (“trajectory”) and performing consistency checks on the image series. For example, such consistency checks relate to the stability of the classification.

[0004] However, known methods of extracting relevant information from individual images and then simply placing the abstract or generalized target information into a temporal context inevitably lead to a certain degree of information loss. Therefore, the distinguishing features of people and other objects are lost.

[0005] A further drawback of the known methods is that the images used (especially two-dimensional (2D) images) may contain personal data, which cannot be meaningfully anonymized without removing relevant features or introducing spurious features. Therefore, there are problems with the collection, storage, and use of this data within the European Economic Area. Summary of the Invention

[0006] Therefore, the object of the present invention is to provide an apparatus and an associated method for identifying at least one person in image data, which achieves an improvement over the prior art. In particular, the identification should meet functional safety requirements.

[0007] The subject matter of the independent claims fulfills this objective. Further developments and embodiments are the subject matter of the dependent claims.

[0008] In a first embodiment, the apparatus for identifying at least one person in image data has a first data input, a first preprocessing unit, and a main processing unit. The first data input is adapted to supply a first sequence of individual images. The first preprocessing unit is configured to generate a first motion trajectory image from a first settable number of consecutive individual images in the first sequence in each case. The main processing unit is configured to evaluate the first motion trajectory image and thereby provide an output signal. The output signal indicates the number of persons and / or the number of persons identified at least in the first motion trajectory image during the evaluation.

[0009] In the apparatus according to the invention, a first sequence of individual images is prepared in a first preprocessing unit for evaluation in a downstream main processing unit, so as to combine multiple consecutive individual images to form a first motion trajectory image. This additional processing step makes the distinguishing features between objects and people (e.g.) more clearly highlighted, thereby ensuring that people and objects can be reliably distinguished in the evaluation.

[0010] For example, a sequence of individual images is generated in a known manner using a suitable digital camera. These individual images are, for example, two-dimensional individual images, particularly grayscale or color images. Alternatively, the individual images can be three-dimensional individual images containing distance information for corresponding pixels in their third dimension. Therefore, the camera can be a depth camera, such as a LiDAR camera, which can, for example, use time-of-flight measurements to generate depth information. This distance information can be displayed in a color-coded form. Such individual images can be referred to as depth maps. Figure 5 An exemplary individual image that can be used in one embodiment of the device according to the invention is shown. This is a grayscale image of the scene.

[0011] According to the present invention, the motion trajectory image is formed from a time series, i.e., from multiple individual images recorded sequentially by a camera, and is supplied to the first data input of the device. Since information from multiple individual images is combined and displayed in a single motion trajectory image of a corresponding sequence, for example, target motion occurring in the sequence of individual images used for the motion trajectory image becomes visible in that motion trajectory image and is thus identified by automatic image analysis in the main processing unit. Therefore, during the evaluation period, the main processing unit can reliably distinguish whether the motion trajectory image contains people and, if necessary, determine the number of people in the motion trajectory image. If the motion trajectory image contains only objects or moving objects, the number is zero.

[0012] For example, an output signal can indicate the corresponding location of the identified person and / or their bounding box. The value indicated by the output signal and the value representing the identified person can also be designed in different forms, such as a trajectory determined from a motion trajectory image, as long as it ensures that conclusions about the person can be drawn from it, and subsequent or downstream counting can be performed as needed to determine the number.

[0013] The number of individual images combined in the preprocessing unit to form the motion trajectory image can be set according to the application. Because the number of individual images is small, motion features may not be very obvious, but on the other hand, the delay (i.e., the time offset between the recording of the first volume image or its supply to the device according to the invention and the presence of the output signal) is still very short. In applications where delay is not critical, more individual images can be used to generate the motion trajectory image. Therefore, in some cases, long motion trajectories can be seen in the motion trajectory image, thereby making the features and differences of people compared to other objects in the motion trajectory image even clearer and easier to identify in the main processing unit.

[0014] In one embodiment, the main processing unit includes an artificial neural network, particularly a classification neural network.

[0015] For example, a neural network can be implemented as a so-called convolutional neural network (CNN). Neural networks can also be referred to as AI systems. In an exemplary implementation, the YOLO (You Only Look Once) network, known to have multiple convolutional layers, can be used or reused. For example, in this implementation, one of the input channels of a typical RGB image data source for a YOLO network is used to feed the motion trajectory image. Furthermore, a depth map generated by a camera (i.e., a depth image) can be simultaneously fed to a second input channel of the YOLO network, and / or a grayscale or black-and-white image can be fed to a third input channel, where the grayscale or black-and-white image specifically corresponds to the depth image but does not contain any depth information, instead including, for example, intensity values. In this way, the three input channels typically used for the three color components (red, green, and blue) can be (re)used.

[0016] As further developed, the neural network was trained using multiple annotated first training motion trajectory images during the training phase.

[0017] For example, during the training phase, the neural network is pre-trained, i.e., trained using synthetic or real training data before its actual use during operation. According to the invention, motion trajectory images (which are annotated or labeled, and each image shows a person or object, and are referred to as training motion trajectory images) are used as training data. Thus, the neural network is trained to perform person classification and object recognition as functional elements of the main processing unit. In an exemplary implementation based on the YOLO network, a pre-trained YOLO network can be used, which, according to the invention, is further trained during the training phase using training motion trajectory images. Specifically, in the pre-trained YOLO network, the earlier layers closer to the input neurons have already been well-trained on basic image data. During the training phase specified herein, the later layers closer to the output neurons are substantially tuned through training.

[0018] In a second embodiment, the apparatus further includes a second data input and a second preprocessing unit, the second data input being used to supply a second sequence of individual images. The second preprocessing unit is configured to generate a second motion trajectory image from a settable number of consecutive individual images in each case. The main processing unit is further configured to evaluate the first motion trajectory image and the second motion trajectory image, and thereby provide an output signal.

[0019] Therefore, in the preprocessing unit of the apparatus according to the invention, namely the first preprocessing unit and the second preprocessing unit, two motion trajectory images are created in parallel in time based on the first sequence and the second sequence of individual images. In this respect, the second sequence and the first sequence refer to the same scene, but differ in some aspects, as will be explained in detail below. The motion trajectory images generated in this way are both evaluated in the main processing unit. Therefore, the classification quality performed by the main processing unit is further improved. For example, the classification becomes more stable, and the robustness of the main processing unit's decision is improved.

[0020] In this regard, the consecutive individual images of the first or second sequence preferably refer to only directly consecutive individual images. Alternatively, individual images can be supplied as a sequence of individual images at irregular intervals, every one or every two or several frames, to the first or second data input.

[0021] In an exemplary implementation, a second settable number of consecutive individual images is adapted to a first settable number of consecutive individual images. Furthermore, the starting point of the second sequence of individual images can be adapted to the starting point of the first sequence of individual images. For example, the same scene is recorded twice, almost simultaneously, in different ways, such as using two different image sensors or cameras. The resulting corresponding image series (referred to herein as the first and second sequences of individual images) are supplied to the apparatus according to the invention via a first data input and a second data input. Therefore, preferably, the first motion trajectory image and the second motion trajectory image are based on approximately the same number of individual images recorded approximately at the same time.

[0022] As further developed, the first sequence of individual images includes two-dimensional individual images, particularly grayscale images. The second sequence of individual images includes three-dimensional individual images, particularly depth maps.

[0023] In the second embodiment described above, the motion trajectory images differ accordingly because the first motion trajectory image is generated in the first preprocessing unit based on a two-dimensional individual image (e.g., a grayscale image), while the second motion trajectory image is generated in the second preprocessing unit using a three-dimensional individual image (specifically, a depth map). In this sense, both the first and second sequences of individual images are based on the same scenario, namely, a scenario for personnel identification or counting.

[0024] In a further development of the second embodiment, the main processing unit includes an artificial neural network, particularly a classification neural network.

[0025] A classification neural network can be used to evaluate a first motion trajectory image and a second motion trajectory image. Similar to the description above, the neural network can be implemented as, for example, a convolutional neural network (CNN). The YOLO network can be based on this.

[0026] According to a further development of the second embodiment, the neural network is trained using multiple annotated first training motion trajectory images and multiple annotated second training motion trajectory images during the training phase.

[0027] A classification neural network, pre-trained with appropriate first and second labels on motion trajectory images, can be used to evaluate the first and second motion trajectory images. In subsequent routine operation of the device, this neural network classifies objects based on features discovered or identified by the network during the training phase, thereby distinguishing people from other objects. Therefore, the number of people or individuals identified in the motion trajectory images is provided as an output signal.

[0028] According to further development, the first motion trajectory image or the second motion trajectory image is the superposition of individual images of a set number of consecutive individual images from the first or second sequence of individual images.

[0029] The first motion trajectory image is a superposition, accumulation, or aggregation of individual images from a set number of consecutive individual images from a first sequence. The second motion trajectory image is a superposition of individual images from a set number of consecutive individual images from a second sequence. If two sequences of individual images are used in an exemplary implementation (such as the second embodiment described above), the first motion trajectory image is, for example, a two-dimensional grayscale image, while the second motion trajectory image is a three-dimensional image or a depth map. Figure 6 An example of a two-dimensional motion trajectory image is shown, while Figure 7 An exemplary 3D motion trajectory image is shown. In the example, in each case, both images are calculated from five consecutive individual images. The motion trajectory images shown are each based on an example sequence of individual images, which in this case were recorded at a reduced frame rate of less than 30 frames per second (fps). Therefore, the corresponding motion trajectories appear more fragmented than they would at a standard frame rate (approximately 30 fps). Figure 6 and Figure 7 Each figure depicts a person holding an object in a floor plan, and in each case, the person has moved from the bottom left corner to the top right corner.

[0030] In further development, the first motion trajectory image or the second motion trajectory image is formed by summing and / or averaging pixel by pixel of a set number of consecutive individual images of the first sequence or the second sequence.

[0031] Each motion trajectory image is formed by performing one or more of the aforementioned measurements on individual images of a set number of consecutive individual images in the corresponding sequence. For example, image features of different individual images in the image sequence are transferred to a common image, i.e., the motion trajectory image. Optionally, background removal can be performed on each individual image first. In this option, so-called frame differencing can be used to subtract consecutive individual images in order to emphasize the moving object in this way.

[0032] By using motion trajectory images as input data to the main processing unit, temporal relationships that would otherwise only be discernible in a sequence of multiple individual images can be examined within a single image (i.e., a motion trajectory image). Motion trajectory images can also be advantageously used to anonymize grayscale images, as subtle features of the individuals who can be identified can be effectively blurred. However, in this respect, the relevant motion and shape features of the individuals are preserved.

[0033] A further advantage of using motion trajectory images is that they inherently eliminate personal features that are still visible in the individual images of the corresponding sequence, such as... Figure 5 As shown. In motion trajectory images, these optical features are anonymized, such as... Figure 6 and Figure 7 It is readily apparent in the images. In this respect, the degree of anonymization depends particularly on the number of individual images used for motion trajectory images. Due to this anonymization, the classification and identification of people in motion trajectory images according to the present invention also comply with the strict data protection regulations established in Europe regarding data storage and use.

[0034] According to a third embodiment, in addition to evaluating the first motion trajectory image, the main processing unit is configured to evaluate at least one individual image from a first sequence of individual images, and thereby provide an output signal. In this regard, the first sequence of individual images includes three-dimensional individual images, particularly a depth map.

[0035] In this embodiment, in addition to the motion trajectory image, at least one individual image from the sequence on which the motion trajectory image is based is also used as input information for the main processing unit. Therefore, the additional optical features and anonymization in the motion trajectory image are combined with the lower latency of the individual images and the fact that objects are generally easier to separate. Since 3D data offers better anonymity, in this embodiment, depth map images are preferred over 2D or grayscale images as the additional individual images to be evaluated.

[0036] In a further development of the second embodiment, in addition to evaluating the first motion trajectory image and the second motion trajectory image, the main processing unit is also configured to evaluate at least one individual image from the first sequence or the second sequence of individual images, and thereby provide an output signal.

[0037] In this embodiment, the main processing unit uses a first motion trajectory image, a second motion trajectory image, and one or more individual images respectively derived from a first sequence or a second sequence. As stated above, three-dimensional individual images are preferred in this regard. Therefore, the provisions regarding anonymization from the corresponding data protection regulations are implicitly or automatically implemented. Furthermore, this approach can further enhance personnel identification capabilities.

[0038] In a further development of the first embodiment, the main processing unit includes a further artificial neural network, particularly a further classification neural network, and an output unit. During the training phase, the further neural network is trained using multiple annotated first training individual images to evaluate at least one individual image in a first sequence of individual images. The output unit is configured to provide an output signal based on the evaluation results of the neural network and the evaluation results of another neural network. Alternatively, during the training phase, in addition to the first motion trajectory image, multiple annotated first training individual images are used to train the neural network of the main processing unit to evaluate at least one individual image in the first sequence of individual images.

[0039] Therefore, the main processing unit implemented using artificial intelligence methods can be designed such that only one neural network is used to evaluate the first motion trajectory image and one or more individual images. The neural network is trained accordingly with the first labeled training motion trajectory image and the first labeled training individual images, in a conventional manner. In an alternative implementation, a separate neural network (here referred to as the further neural network) is provided for parallel evaluation of one or more individual images within the meaning of the conceptual diversity detected. In this implementation, the main processing unit thus comprises two artificial neural networks. The output unit generates an output signal based on the evaluation results provided by the corresponding output neurons of the respective neural networks through appropriate combinations (e.g., logical OR or logical AND operations). The output unit may optionally consider the quality of the corresponding evaluation results.

[0040] In the evaluation, individual images are used in addition to motion trajectory images, which advantageously achieves redundancy, particularly beneficial for the reliability of personnel identification in security-related applications. Using two independent neural networks, in addition to redundancy, also achieves the aforementioned diversity. The subsequent neural network is then pre-trained using only labeled training individual images. In this respect, diversity is achieved both through different image features in the individual images or motion trajectory images, and through different training processes and potentially different architectures of the neural networks used in each case.

[0041] In a further development of the second embodiment, the main processing unit includes a further artificial neural network (particularly a further classification neural network) and an output unit. During the training phase, the further neural network is trained using multiple annotated first training individual images and / or multiple annotated second training individual images to evaluate at least one individual image from a first sequence of individual images and / or at least one individual image from a second sequence of individual images. The output unit is configured to provide an output signal based on the evaluation results of the neural network and the further neural network. Alternatively, during the training phase, in addition to the first and second motion trajectory images, multiple annotated first training individual images and / or multiple annotated second training individual images are used to train the neural network of the main processing unit to evaluate one or more individual images from the first sequence of individual images and / or the second sequence of motion trajectory images.

[0042] The evaluation of two motion trajectory images (available as grayscale images and depth map images) can also be supplemented by evaluating one or more individual images, where it is technically possible to achieve this using either grayscale images or depth map images. To comply with European data protection guidelines, depth map images are preferred as individual images. In an implementation, on one hand, a single neural network can be pre-trained using labeled first training motion trajectory images, labeled second training motion trajectory images, and labeled first and / or second training individual images. In a second possible implementation, also in the sense of conceptual diversity, two different neural networks are provided, one trained with the first and second labeled motion trajectory images, and the other trained for person recognition using labeled individual images (preferably depth map images).

[0043] In further development, the first and / or second annotated training motion trajectory images used in the corresponding training phases are for targets of different categories, and are annotated according to these categories.

[0044] By using annotations or tags, tasks can be assigned the exact or correct output category.

[0045] Specifically, the different categories include at least the human category, the vehicle category, and the humanoid robot category.

[0046] In this regard, vehicle categories may include, for example, autonomous vehicles (such as Automated Guided Vehicles (AGVs)) or machines.

[0047] Specifically, the corresponding training phase was conducted using supervised learning.

[0048] Preferably, the training data (i.e., training motion trajectory images and / or training individual images) is designed for the corresponding application to be implemented.

[0049] In further development, training motion trajectory images for human categories include at least one motion feature with human characteristics, such as pendulum motion and / or swinging motion of a person in motion, motion of at least one limb of a person in motion, and plastic deformation of the body of a person in motion.

[0050] Motion features include, for example, geometric motion patterns. It is well known that human motion, particularly locomotion, differs from that of humanoid robots in that it is more irregular. By using a large number of different training motion trajectory images, the variability of corresponding motion patterns within the same category and across different categories is increased, thereby further improving the ability to identify people.

[0051] Because of the training phase according to the invention and using the described training data, the main processing unit (particularly the neural network or network used in each case herein) is trained to identify persons contained in motion trajectory images. Furthermore, the direction of motion can be evaluated, for example, by recording the motion route or trajectory. This is particularly advantageous when the device according to the invention is used to technically process persons moving between two different spatial regions.

[0052] In one embodiment, the people counting system includes means according to the invention for identifying at least one person in image data according to any of the above embodiments, and a camera for generating individual images. The camera's data output is connected to the means's data input. The people counting system is configured to functionally and safely detect the number of people crossing a boundary based on the means's output signal.

[0053] For example, a people counting system is used in a access counter to record the number of people passing through a passageway, particularly the number of people in the corresponding direction. Therefore, such access counters can be used in transitional areas between safe and hazardous areas in a factory to determine the number of people in the hazardous area in a functionally safe manner. In accordance with applicable safety standards, such as IEC 61508 or ISO 13849, access counters must be designed to be safe so that the metering readings they determine can be used in a safety-related manner to minimize risk.

[0054] Functional safety refers to the portion of system safety that relies on the correct operation of safety-related systems and other risk mitigation measures. The challenge of known counting systems lies in distinguishing people from other objects, such as luggage carried by people (e.g., flip charts), and in grouping people into different groups. Known systems are optimized for the specific ability to distinguish people from other objects, but not for the specific reliability enhancements in a functional safety sense. In contrast, the personnel counting system according to the present invention is enhanced in terms of functional safety and, due to the use of motion trajectory images, can reliably identify people.

[0055] A further embodiment of the present invention relates to a computer-implemented method for identifying at least one person in image data. The method includes the following steps:

[0056] -The first sequence of individual images supplied

[0057] - Generate a first motion trajectory image from a settable number of consecutive individual images in a first sequence.

[0058] -Evaluate the first motion trajectory image, and

[0059] - Provides an output signal indicating the number of people and / or individuals identified at least in the first motion trajectory image during the evaluation.

[0060] In this method, additional steps are used to generate motion trajectory images and their subsequent evaluation to improve the ability to identify people in the image data, so that the identified people or the number of people provided by the output signal meet functional safety requirements.

[0061] In one embodiment, the computer-implemented method is used in the apparatus described above according to the invention.

[0062] On the other hand, the foregoing statements regarding the apparatus according to the invention also apply accordingly to the method; particularly regarding the advantages and embodiments. Unless otherwise expressly stated, the mentioned embodiments may be combined with each other. Attached Figure Description

[0063] The invention will be explained in more detail by way of examples and with reference to the accompanying drawings. Components with the same function or effect have the same reference numerals. As shown in the figures:

[0064] Figure 1 This is an example of a first embodiment of the proposed device;

[0065] Figure 2 This is an example of a second embodiment of the proposed device;

[0066] Figure 3 Example of a third embodiment of the proposed device; and

[0067] Figure 4 This is an example of a fourth embodiment of the proposed device;

[0068] Figure 5 Examples of individual images;

[0069] Figure 6 Here is an example of a motion trajectory image;

[0070] Figure 7 For further exemplary motion trajectory images; and

[0071] Figure 8 For further exemplary motion trajectory images. Detailed Implementation

[0072] Figure 1 An example of a first embodiment of the proposed device is shown. The device 100 for identifying at least one person in image data has an input 10 for supplying a first sequence of individual images E1, E2, ..., En. For example, these individual images E1 to En are generated by a camera (not shown here), particularly a 2D or 3D camera. A preprocessing unit 20 generates a motion trajectory image 25 from a predetermined number of individual images. This motion trajectory image 25 can be said to correspond to a superposition of multiple consecutively recorded individual images. A main processing unit 30 evaluates the motion trajectory image 25 to determine whether the motion trajectory image 25 shows a person, or how many people are depicted in the motion trajectory image 25. Accordingly, an output signal 31 is provided, indicating the person or the number of people identified in the motion trajectory image 25.

[0073] The overlay of multiple individual images results in a stronger emphasis on distinguishing features of people and other objects (such as the pendulum motion characteristic of people walking), thereby enabling higher-quality classification in the main processing unit 30. Furthermore, this overlay leads to a degree of anonymization of people (who may be even more clearly identifiable in the individual images). Therefore, the processing of the supplied image data is carried out in accordance with prescribed data protection guidelines.

[0074] For example, a neural network 32 is used in the main processing unit 30. It is pre-trained using synthetic or real motion trajectory images 25 to evaluate those leading to correct results. As previously described, during the training phase, the neural network is provided with examples of different categories by supplying annotated training motion trajectory images. In this phase, the neural network independently develops features that are identifiable only within the neural network structure and classifies operations based on these features.

[0075] Optionally, the individual image Ex can also be used for evaluation in the main processing unit 30. In this sense, the individual image Ex is one of the individual images on which the motion trajectory image 25, just generated in the preprocessing unit 20, is based. In this embodiment, the output signal 31 is generated due to the evaluation of the individual image Ex and the motion trajectory image 25. If the main processing unit 30 is implemented using a single neural network, then in this case, the neural network 32 is pre-trained using the labeled training motion trajectory image 25 and the labeled training individual image Ex.

[0076] Figure 2 Further embodiment examples of the proposed device 100 are shown. This example corresponds to... Figure 1 The example also includes a second input 11, through which a second sequence of individual images F1, F2, ..., Fn is supplied to the device 100. The second sequence of individual images, or several of these individual images, is used in the second preprocessing unit 21 to generate a second motion trajectory image 26. The second motion trajectory image 26 is supplied to the main processing unit 30 in parallel with the first motion trajectory image 25. It evaluates both motion trajectory images 25 and 26, thereby providing an output signal 31 containing information about whether and how many people were detected in the motion trajectory images 25 and 26.

[0077] The first sequence of individual images E1 to En contains, for example, 2D images or grayscale images, while the second sequence of individual images F1 to Fn contains 3D images or depth maps.

[0078] If the main processing unit 30 is implemented using a neural network 33, the neural network 33 is pre-trained using a labeled first training motion trajectory image 25 and a labeled second training motion trajectory image 26.

[0079] In the exemplary implementation, neural network 33 is implemented based on the YOLO network. Of the three nearly identical input channels, one channel is used for the first motion trajectory image 25, the second channel for the second motion trajectory image 26, and the third channel for a second sequence of depth maps of individual images F1 to Fn. For example, network 33 provides locations marked with so-called bounding boxes where people have been identified. For example, it determines the trajectory of the motion trajectory images.

[0080] In this regard, whether 2D and 3D motion trajectory images are evaluated in parallel or separately depends solely on the architecture of the neural network. In separate evaluation, two independent classification results are determined for the two image channels. The final decision (output as a result signal) can then be made in different ways: in some particularly important cases where people cannot be ignored, for example, logic such as "if the result of at least one of the two decisions has people, then the overall result also has people" can be used within the meaning of the logical OR connection of the decision.

[0081] Optionally, in its corresponding evaluation, the main processing unit 30 may include one or more individual images Ex from a first sequence of individual images, and additionally or alternatively, one or more individual images Fx from a second sequence of individual images. In this case, the single neural network 33 is pre-trained using labeled first training motion trajectory images 25, labeled second training motion trajectory images 26, labeled first sequence training individual images Ex and / or labeled second sequence training individual images Fx.

[0082] Figure 3 A third embodiment example of the device 100 is shown. This embodiment example corresponds to... Figure 1 An example embodiment, differing in that the main processing unit 30 provides two independent neural networks 32 and 34 for evaluation, as well as an output unit 35. For example... Figure 1 As shown, neural network 32 processes the first motion trajectory image 25 and thereby generates a first evaluation result for output unit 35. An additional evaluation of the individual image Ex is processed in a separate neural network 34. It generates a second evaluation result, which is also supplied to output unit 35. Output unit 35 generates an output signal 31 based on the two evaluation results. Neural networks 34 and 32 are both pre-trained accordingly. That is, neural network 32 is trained using the labeled first training motion trajectory image 25, while neural network 34 is pre-trained using the labeled training individual image Ex.

[0083] This implementation advantageously provides a high degree of diversity through different image features and different training processes, potentially leading to different architectures for the two neural networks, 32 and 34. The classification quality is thus further improved.

[0084] Figure 4 A fourth embodiment example of the proposed device 100 is shown. This example corresponds to... Figure 2 Examples of embodiments, except for the different implementation of the main processing unit 30. In Figure 4In the main processing unit 30, two independent neural networks 33 and 36 are used. The first motion trajectory image 25 and the second motion trajectory image 26 are evaluated in neural network 33. Individual images Ex of the first sequence and / or individual images Fx of the second sequence are processed in neural network 36. The corresponding evaluation results of neural networks 33 and 36 are supplied to the output unit 35, which then determines the output signal 31. In this embodiment example, the diversity of concepts further increases the reliability of the classification.

[0085] Figure 5 An exemplary individual image is shown, which can be part of a first or second sequence of individual images. The image shown is an image from the camera's grayscale channels. It can be seen that the identity of the person in the image can be determined entirely based on the image.

[0086] Figure 6 An example of a motion trajectory image calculated from a 2D grayscale individual image is shown.

[0087] Figure 7 An exemplary motion trajectory image is shown, which is composed of accumulated 3D individual images.

[0088] Figure 8 A further example of motion trajectory images as 3D point clouds is shown. It can be seen that a person's specific body shape leaves distinctive features in the illustrated motion trajectory. This motion trajectory differs significantly from the trajectory of a vehicle or a rigid object.

[0089] By adding the corresponding tags Figures 6 to 8 Each image in the dataset can be used as a training motion trajectory image. By analyzing... Figure 5 By appropriately labeling the individual images in the dataset, exemplary training individual images can be generated.

[0090] List of reference numerals

[0091] E1, E2, ..., En individual images

[0092] Individual images of F1, F2, ..., Fn

[0093] Input 10, 11

[0094] Preprocessing units 20 and 21

[0095] Images of motion trajectories 25 and 26

[0096] 30 Main Processing Units

[0097] 31 Output signal

[0098] 32, 33, 34, 36 Neural Networks

[0099] 35 Output Unit

Claims

1. An apparatus (100) for identifying at least one person in image data, comprising: The first data input (10) is used to supply a first sequence of individual images (E1, ..., En). A first preprocessing unit (20) is configured to generate a first motion trajectory image (25) from a first settable number of consecutive individual images of the first sequence in each case. The main processing unit (30) is configured to evaluate the first motion trajectory image (25) and thereby provide an output signal (31) indicating the number of people and / or persons identified in the first motion trajectory image during the evaluation.

2. The apparatus (100) according to claim 1, characterized in that, The main processing unit (30) includes an artificial neural network (32).

3. The apparatus (100) according to claim 2, characterized in that, During the training phase, the neural network (32) is trained using multiple annotated first training motion trajectory images.

4. The apparatus (100) according to claim 1, characterized in that, Also includes: The second data input (11) is used to supply a second sequence of individual images. The second preprocessing unit is configured to generate a second motion trajectory image (26) from a second settable number of consecutive individual images (F1, ..., Fn) of the second sequence in each case, and The main processing unit (30) is also configured to evaluate the first motion trajectory image (25) and the second motion trajectory image (26) and thereby provide the output signal (31).

5. The apparatus (100) according to claim 4, characterized in that, The first sequence of individual images includes two-dimensional individual images, and the second sequence of individual images includes three-dimensional individual images.

6. The apparatus (100) according to claim 4, characterized in that, The main processing unit (30) includes an artificial neural network (33).

7. The apparatus (100) according to claim 6, characterized in that, During the training phase, the neural network (33) is trained using multiple annotated first training motion trajectory images and multiple annotated second training motion trajectory images.

8. The apparatus (100) according to claim 1, characterized in that, The first motion trajectory image (25) or the second motion trajectory image (26) is a superposition of individual images of a set number of consecutive individual images from a first sequence or a second sequence of individual images.

9. The apparatus (100) according to claim 1, characterized in that, In addition to evaluating the first motion trajectory image (25), the main processing unit (30) is also configured to evaluate at least one individual image (Ex) from the first sequence of individual images and thereby provide the output signal (31). The first sequence of individual images includes three-dimensional individual images.

10. The apparatus (100) according to claim 3, characterized in that, The main processing unit (30) includes a further artificial neural network (34) and an output unit, wherein, during the training phase, the further neural network is trained using a plurality of annotated first training individual images to evaluate at least one individual image (Ex) from a first sequence of said individual images, and wherein the output unit (35) is configured to provide the output signal (31) based on the evaluation results of the neural network (32) and the evaluation results of the further neural network (34). or During the training phase, the neural network (32) of the main processing unit (30) is additionally trained using multiple annotated first training individual images to evaluate at least one individual image (Ex) from a first sequence of said individual images.

11. The apparatus (100) according to claim 7, characterized in that, The main processing unit (30) includes a further artificial neural network (36) and an output unit (35), wherein, during the training phase, the further neural network (36) is trained using a plurality of annotated first training individual images and / or a plurality of annotated second training individual images to evaluate at least one individual image (Ex) from a first sequence of the individual images and / or at least one individual image (Fx) from a second sequence of the individual images, and wherein the output unit (35) is configured to provide the output signal (31) based on the evaluation results of the neural network (33) and the evaluation results of the further neural network (36). or During the training phase, the neural network (33) of the main processing unit is additionally trained using multiple annotated first training individual images and / or multiple annotated second training individual images to evaluate at least one individual image (Ex) from a first sequence of the individual images and / or at least one individual image (Fx) from a second sequence of the individual images.

12. The apparatus (100) according to claim 3, characterized in that, The first and second training motion trajectory images used in the corresponding training phases contain targets of different categories and are labeled according to these categories. The different categories include at least the human category, the vehicle category, and the humanoid robot category. The corresponding training phase is conducted using supervised learning.

13. The apparatus (100) according to claim 12, characterized in that, The training motion trajectory images for the human category include at least one motion feature with human characteristics.

14. A personnel counting system, comprising: A device (100) for identifying at least one person in image data, the device comprising: The first data input is used to supply the first sequence of individual images. A first preprocessing unit is configured to generate a first motion trajectory image from a first settable number of consecutive individual images of the first sequence in each case. A main processing unit configured to evaluate the first motion trajectory image and thereby provide an output signal indicating at least the number of people and / or individuals identified in the first motion trajectory image during the evaluation, and the people counting system further includes: A camera for generating images of the individual, wherein the data output of the camera is connected to the data input (10) of the device. The personnel counting system is configured to functionally and safely detect the number of people crossing the boundary based on the output signal (31) of the device (100).

15. A computer-implemented method for identifying at least one person in image data, comprising: The first sequence of individual images (E1, ..., En) is supplied. A first motion trajectory image (25) is generated from a first settable number of consecutive individual images of the first sequence. Evaluate the first motion trajectory image (25), and An output signal (31) is provided, which indicates the number of people and / or personnel identified at least in the first motion trajectory image (25) during the evaluation.

16. The apparatus according to claim 2, characterized in that, The artificial neural network is a classification neural network.

17. The apparatus according to claim 4, characterized in that, The second settable number of consecutive individual images is adapted to the first settable number of consecutive individual images, and the time start point of the second sequence of the individual images is adapted to the time start point of the second sequence of the individual images.

18. The apparatus according to claim 5, characterized in that, The two-dimensional individual image is a grayscale image.

19. The apparatus according to claim 5, characterized in that, The three-dimensional individual image is a depth map.

20. The apparatus according to claim 6, characterized in that, The artificial neural network is a classification neural network.

21. The apparatus according to claim 9, characterized in that, The three-dimensional individual image is a depth map.

22. The apparatus according to claim 10, characterized in that, The further artificial neural network is a further classification neural network.

23. The apparatus according to claim 11, characterized in that, The further artificial neural network is a further classification neural network.

24. The apparatus according to claim 13, characterized in that, The personnel characteristic is at least one of the following: pendulum motion of the personnel in motion, swaying motion of the personnel in motion, motion of at least one limb of the personnel in motion, and plastic deformation of the body of the personnel in motion.

Images