A method for generating input data for at least one functional device of a motor vehicle,a computer program product,a non-transitory computer-readable storage

Abstract

A method for generating input data 26 for at least one functional device 28 of a vehicle 10 by a capturing device of the vehicle, comprising the steps of: capturing an image of surroundings 18 of the

Description

[0001] The present invention relates to the field of automobiles. More specifically, the present invention relates to method for generating input data for at least one functional device of a motor vehicle by a capturing device of the motor vehicle. Furthermore, the present invention relates to a computer program product, a corresponding non-transitory computer-readable storage medium, a corresponding capturing device, as well as a corresponding motor vehicle. BACKGROUND INFORMATION

[0002] Image data is input to a lot of function, for example pedestrian detection in a so-called data system. Recent approaches are learning-based and require a big amount of labelled data to perform the task. This data cannot easily be collected because privacy information such as a license plate or face are visible and therefore allow for identify the individual that was captured in the image.

[0003] Typical personal information is blurred or masked before transferring the data. When training the model, those features, however, might be required, for example, the gaze of a pedestrian contains valuable information about their intent of crossing a road. SUMMARY OF THE INVENTION

[0004] It is an object of the present invention to provide a method, a corresponding computer program product, a corresponding non-transitory computer-readable storage medium, a corresponding capturing device as well as a corresponding motor vehicle, by which input data for a functional device of the motor vehicle can be provided in an improved manner.

[0005] This object is solved by a method, a corresponding computer program product, a corresponding non-transitory computer-readable storage medium, a corresponding capturing device, as well as a corresponding motor vehicle according to the independent claims. Advantageous embodiments are presented in the dependent claims.

[0006] One aspect of the invention relates to a method for generating input data for at least one functional device of a motor vehicle by a capturing device of the motor vehicle. An image of surroundings of the motor vehicle is captured by a sensor device of the capturing device, wherein in the captured surroundings at least one privacy information, which needs to be anonymized, is present. The at least one privacy information is identified by an electronic computing device of the capturing device. A realistic substitute of the privacy information is generated by an artificial intelligence of the electronic computing device, and the input data is generated by replacing the privacy information with the realistic substitute in the image by the electronic computing device.

[0007] Therefore, an improved generating for input data can be realized.

[0008] In particular, the present method enables the capturing device that a privacy information is anonymized in such a way that not only the privacy information is blurred or masked, but the privacy information is substituted and therefore, for example, valuable information about the privacy information remains intact.

[0009] Therefore, the artificial intelligence, which may be configured as a generative artificial intelligence, such as a stable diffusion process uses a huge amount of unlabeled image data to build a model of how the world looks like. After training such a model, it can synthesize arbitrary images, potentially even conditioned on existing images.

[0010] The capturing device, which is described herein, uses such a pre-generative artificial intelligence model. In a first step, regions with privacy information are detected, for example, a face detector is used to identify regions in an image that contains faces. In a second step, conditional generation of a new content for that region is provided. Therefore, the generative model used was trained on the specific use case, for example, generating faces. Conditioning on content of that input image is performed in that region, for example, to plant into image and to resemble abstract features, for example, a gaze in the face example. Constraints are added that it should be different to the regional content, in particular, to avoid mimicking.

[0011] The data may only contain information of the anonymized regions that were generated by the model. It is crucial that the generative model is trained on a diverse data set. For the face example, if the generative model was only trained on male faces, the final data set may contain only male faces, and any model trained on that data set might have issues processing female pedestrians.

[0012] According to another embodiment, an object detection algorithm is used for identifying the at least one privacy information.

[0013] In another embodiment, the artificial intelligence is a generative model algorithm.

[0014] In another embodiment, the artificial intelligence was trained such, that different data sets for different privacy information were used.

[0015] In particular, the method is a computer-implemented method. Therefore, another aspect of the invention relates to a computer program product comprising program code means for performing a method according to the preceding aspect.

[0016] A still further aspect of the invention relates to a non-transitory computer-readable storage medium comprising the at least computer program product according to the preceding aspect.

[0017] Furthermore, one aspect of the invention relates to a capturing device for generating input data for at least one functional device of a motor vehicle, comprising at least one sensor device and one electronic computing device, wherein the capturing device is configured for performing a method according to the preceding aspect. In particular, the method is performed by the capturing device.

[0018] Furthermore, the present invention relates to a motor vehicle comprising at least one functional device and one capturing device according to the preceding aspect.

[0019] Advantageous embodiments of the method are to be regarded as advantageous embodiments of the computer program product, the non-transitory computer-readable storage medium, the capturing device, as well as the motor vehicle. The capturing device as well as the motor vehicle therefore comprise means for performing the method.

[0020] An artificial neural network can be understood as a software code or a compilation of several software code components, wherein the software code may comprise several software modules for different functions, for example one or more encoder modules and one or more decoder modules.

[0021] An artificial neural network can be understood as a non-linear model or algorithm that maps an input to an output, wherein the input is given by an input feature vector or an input sequence and the output may be an output category for a classification task or a predicted sequence.

[0022] For example, the artificial neural network may be provided in a computer-readable way, for example, stored on a storage medium of the vehicle, in particular of the at least one computing unit.

[0023] The neural network comprises several modules including the encoder module and the at least one decoder module. These modules may be understood as software modules or respective parts of the neural network. A software module may be understood as software code functionally connected and combined to a unit. A software module may comprise or implement several processing steps and / or data structures.

[0024] The modules may, in particular, represent neural networks or sub-networks themselves. If not stated otherwise, a module of the neural network may be understood as a trainable and, in particular, trained module of the neural network. For example, the neural network and thus all of its trainable modules may be trained in an end-to-end fashion before the method is carried out. However, in other implementations, different modules may be trained or pre-trained individually. In other words, the method according to the invention corresponds to a deployment phase of the neural network.

[0025] In the context of the present disclosure, an object detection algorithm may be understood as a computer algorithm, which is able to identify and localize one or more objects within a provided input dataset, for example input image, by specifying respective bounding boxes or regions of interest, ROI, and, in particular, assigning a respective object class to each of the bounding boxes, wherein the object classes may be selected from a predefined set of object classes. Therein, assigning an object class to a bounding box may be understood such that a corresponding confidence value or probability for the object identified within the bounding box being of the corresponding object class is provided. For example, the algorithm may provide such a confidence value or probability for each of the object classes for a given bounding box. Assigning the object class may for example include selecting or providing the object class with the largest confidence value or probability. Alternatively, the algorithm can specify only the bounding boxes without assigning a corresponding object class.

[0026] A computing unit / electronic computing device may in particular be understood as a data processing device, which comprises processing circuitry. The computing unit can therefore in particular process data to perform computing operations. This may also include operations to perform indexed accesses to a data structure, for example a look-up table, LUT.

[0027] In particular, the computing unit may include one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example, one or more application-specific integrated circuits, ASIC, one or more field-programmable gate arrays, FPGA, and / or one or more systems on a chip, SoC. The computing unit may also include one or more processors, for example one or more microprocessors, one or more central processing units, CPU, one or more graphics processing units, GPU, and / or one or more signal processors, in particular one or more digital signal processors, DSP. The computing unit may also include a physical or a virtual cluster of computers or other of said units.

[0028] In various embodiments, the computing unit includes one or more hardware and / or software interfaces and / or one or more memory units.

[0029] A memory unit may be implemented as a volatile data memory, for example a dynamic random access memory, DRAM, or a static random access memory, SRAM, or as a non-volatile data memory, for example a read-only memory, ROM, a programmable read-only memory, PROM, an erasable programmable read-only memory, EPROM, an electrically erasable programmable read-only memory, EEPROM, a flash memory or flash EEPROM, a ferroelectric random access memory, FRAM, a magnetoresistive random access memory, MRAM, or a phase-change random access memory, PCRAM.

[0030] Further advantages, features, and details of the invention derive from the following description of preferred embodiments as well as from the drawings. The features and feature combinations previously mentioned in the description as well as the features and feature combinations mentioned in the following description of the figures and / or shown in the figures alone can be employed not only in the respectively indicated combination but also in any other combination or taken alone without leaving the scope of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The novel features and characteristic of the disclosure are set forth in the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and together with the description, serve to explain the disclosed principles. The same numbers are used throughout the figures to reference like features and components. Some embodiments of system and / or methods in accordance with embodiments of the present subject matter are now described below, by way of example only, and with reference to the accompanying figures.

[0032] The drawings show in:

[0033] Fig. 1 a schematic side view according to an embodiment of a motor vehicle comprising an embodiment of a capturing device; and

[0034] Fig. 2 a schematic flow chart according to an embodiment of the method.

[0035] In the figures the same elements or elements having the same function are indicated by the same reference signs. DETAILED DESCRIPTION

[0036] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0037] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawing and will be described in detail below. It should be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0038] The terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion so that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a system or apparatus preceded by “comprises” or “comprise” does not or do not, without more constraints, preclude the existence of other elements or additional elements in the system or method.

[0039] In the following detailed description of the embodiment of the disclosure, reference is made to the accompanying drawing that forms part hereof, and in which is shown by way of illustration a specific embodiment in which the disclosure may be practiced. This embodiment is described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present disclosure. The following description is, therefore, not to be taken in a limiting sense.

[0040] Fig. 1 shows a schematic side view according to an embodiment of a motor vehicle 10 comprising an embodiment of a capturing device 12. The capturing device 12 comprises at least one sensor element 14, for example a camera. Furthermore, the capturing device 12 comprises at least one electronic capturing device 16. The capturing device 12 is configured for capturing surroundings 18 of the motor vehicle 10.

[0041] Furthermore, Fig. 1 shows a pedestrian 20 in the surroundings 18. According to the shown embodiment, the pedestrian 20 may be substituted by a substitute 22. Therefore, the electronic capturing device 16 may comprise an artificial intelligence 24.

[0042] Fig. 2 shows a schematic flow chart according to an embodiment of the method. In particular, Fig. 2 shows a method for generating input data 26 for at least one functional device 28 of the motor vehicle 10. An image 30 of the surroundings 18 is captured by the sensor device 14, wherein in the captured surroundings 18 at least one privacy information, which needs to be anonymized, is present. For example, in the shown embodiment of Fig. 1, the privacy information may be a face of the pedestrian 20. This step is shown with a first step S1. In a second step S2, the at least one privacy information is identified by the electronic capturing device 16. The realistic substitute 22 is generated by the artificial intelligence 24 in a third S3. In a fourth step S4, the input data 26 is generated by replacing the privacy information with the realistic substitute 22 in the image 30 by the electronic capturing device 16.

[0043] In particular, an object detection algorithm is used for identifying the at least one privacy information. Furthermore, as the artificial intelligence 24 a generative model is used. Furthermore, the artificial intelligence 24 was trained such that different data sets for different privacy information were used.

[0044] In particular, the shown figures provide a generative artificial intelligence solution such as a stable diffusion process for a huge amount of unlabeled image data to build a model of how the world looks like. After training such a model, it can synthesize arbitrary images, potentially even conditioned on existing images. The capturing device 12 is characterized by using such a pre-trained generative artificial intelligence model. The artificial intelligence 24 is configured for detecting regions with privacy information. For example, a face detector is used to identify regions in the image 30 that contains faces. In a second part of the invention the conditional generation of the new content for that region is provided. Therefore, the generative model is used that was trained on the specific use case, for example, generating faces. Condition and content of input image in that region is provided, in particular, to plant into image and to resemble abstract features, for example, a gaze in the face example. The constraint is added that it should be different to the regional content, in particular, to avoid mimicking.

[0045] Therefore, privacy laws can be obeyed. Furthermore, the information allows for large data collecting to get diverse, big and real-world data sets. No loss of relevant abstract features, for example a gaze of pedestrians, is to be registered. Furthermore, the condition generative model is provided on the input image to create realistic fill-in that blends into image context. Constraints are added that forced the generative model to create content that is different to the input content to avoid similarity. signs motor vehicle capturing device sensor device electronic capturing device surrounding pedestrian substitute artificial intelligence input data functional device image steps of the method

Claims

1. A method for generating input data (26) for at least one functional device (28) of a motor vehicle (10) by a capturing device (12) of the motor vehicle (10), comprising the steps of:- capturing an image (30) of surroundings (18) of the motor vehicle (10) by a sensor device (14) of the capturing device (12), wherein in the captured surroundings (18) at least one privacy information, which needs to be anonymized, is present;- identifying the at least one privacy information by an electronic computing device (16) of the capturing device (12);- generating a realistic substitute (22) of the privacy information by an artificial intelligence (24) of the electronic computing device (16); and- generating the input data (26) by replacing the privacy information with the realistic substitute (22) in the image by the electronic computing device (16).

2. The method according to claim 1, characterized in thatan object detection algorithm is used for identifying the at least one privacy information.

3. The method according to claim 1 or 2, characterized in thatas the artificial intelligence (24) a generative model algorithm is used.

4. The method according to any one of claims 1 to 3, characterized in thatthe artificial intelligence (24) was trained such, that different datasets for different privacy information were used.

5. A computer program product comprising program code means for performing a method according to any one of claims 1 to 4.

6. A non-transitory computer-readable storage medium comprising the computer program product according to claim 5.

7. A capturing device (12) for generating input data (26) for at least one functional device (28) of a motor vehicle (10), comprising at least one sensor device (14) and one electronic computing device (16), wherein the capturing device (12) is configured for performing a method according to any one of claims 1 to 5.

8. A motor vehicle (10) comprising at least one functional device (28) and one capturing device (12) according to claim 7.

Images