Method for the relational storage of object properties

Abstract

The invention relates to a computer-implemented method for the relational storage of object properties 116 of an object 102, 104, the method comprising a step of obtaining 118 sensor data 172, 174, 176, 178 in which the object 102, 104 is captured, wherein the sensor data 172, 174, 176, 178 are generated by at least one sensor 106, 168, 170 arranged on a vehicle 100. Furthermore, the method 116 comprises a step of detecting and classifying 120 the object 102, 104 by means of the sensor data 172, 174, 176, 178, wherein a first class is assigned to the object 102, 104 and at least one object property is determined, wherein a confidence value 191 is generated. Furthermore, the method 116 comprises a step of requesting 124 an occupant of the vehicle 100 to provide a second class relating to the object 102, 104 if the confidence value 191 is below a first threshold value 192. Furthermore, the method 116 comprises a further step of receiving 126 a first feedback from the occupant. Furthermore, the method 116 comprises a further step of extracting 128 the second class from the first feedback from the occupant. Furthermore, the method 116 comprises a further step of storing 130 a data record 193 having at least the second class, the at least one object property and a derivative of the sensor data 172, 174, 176, 178.

Description

[0001] METHOD FOR RELATIONAL STORAGE OF OBJECT PROPERTIES

[0002] Technical area

[0003] The present invention relates to the field of data processing. In particular, the present invention relates to computer-implemented methods for the relational storage of object properties of an object. Furthermore, the present invention relates to a computer-implemented method for training artificial intelligence, computer-implemented methods for generating a class relating to an object, a computer program product, a computer-readable storage medium, a data carrier signal, devices for classifying an object, and a vehicle.

[0004] Technical background and task

[0005] The detection of objects outside a vehicle is used to enable automated driving. For this purpose, the objects must also be classified in order to calculate appropriate automated driving maneuvers. The technology is also used to transmit information to vehicle occupants. For example, systems are known that detect and classify vehicles located in front of the vehicle, which are then displayed using a digital graphic model based on the vehicle's class. A truck is then represented as such and not as a generic vehicle.

[0006] Detection and classification are typically performed using artificial intelligence, such as neural networks (NNs). The accuracy with which a NN can detect and classify an object depends on the training data set. If a specific object is missing from this data set, it can only be reliably detected and classified with low probability. If a vehicle occupant, such as the driver, is presented with a misinterpreted vehicle on an in-vehicle display, this can confuse the driver and, in the worst case, provoke an accident.

[0007] In the context of automated driving, a misinterpretation can lead to an over- or underreaction. For example, a safety-relevant object may not be recognized as such, and no appropriate evasive maneuver may be initiated. Or a safety-irrelevant object may be classified as safety-relevant, and the vehicle may perform an unnecessary maneuver that may surprise other road users.

[0008] It is therefore the object of the present invention to provide computer-implemented methods for the relational storage of object properties of an object, which eliminate at least one of the aforementioned disadvantages. Furthermore, it is the object of the invention to provide a computer-implemented method for training an artificial intelligence, computer-implemented methods for generating a class relating to an object, a computer program product, a computer-readable storage medium, a data carrier signal, devices for classifying an object, and a vehicle.

[0009] Disclosure of the invention

[0010] The object is achieved according to the invention by the features of the main claims. Advantageous embodiments can be found in the subclaims.

[0011] According to a first aspect of the invention, a computer-implemented method for the relational storage of object properties of an object comprises a step in which sensor data in which the object is detected is obtained. The sensor data is generated by at least one sensor arranged on a vehicle. The sensor data can be present, for example, as image data from a camera or radar data from a radar sensor. A vehicle can be, for example, a motor vehicle, a truck, a train, a helicopter, an aircraft, or the like. Object properties are specific information in the sensor data, typically related to a region in the sensor data. This can be a physical region such as coordinates, or another region, such as a frequency. Examples of object properties are points, lines, or corners—i.e., geometric properties.A relationship between such information, for example a density of geometric properties or their specific arrangement, is also understood as object properties.

[0012] In a further step of the method, the object is detected and classified using the sensor data, whereby a first class is assigned to the object and at least one object property is determined. A confidence value is generated in this step.

[0013] This step is a well-known task in data and image processing. Those skilled in the art are aware of various methods available to solve this task. Typically, such a task is solved by a (convolutional) neural network ((C)NN). Essentially, a neural network is trained to recognize patterns and features within an image or other data that indicate a specific class. Common methods generate a confidence value that indicates how likely the class has been correctly assigned to the object. The higher the confidence value, the more reliably the object is assigned to the correct class. Lower values, on the other hand, indicate that the object has not been assigned to the correct class. This can happen, for example, if the (C)NN has not been trained on the class and the class is therefore unknown.The first class can therefore also index an unknown class.

[0014] In a further step of the process, a vehicle occupant is asked to provide a second class relating to the object if the confidence value lies below a first threshold. The first threshold represents a limit, but assignment to a class is considered certain, for example 0.95 or 95%. If the confidence value is below this, the object may have been assigned the wrong class. In this case, a vehicle occupant, for example the driver, is asked to provide the correct class, which corresponds to the second class. The request can be made via the vehicle's audio system, for example, by asking the driver "What kind of object is that in front of us?". A display can also be used to address such a request to the occupant. For example, objects can be shown to the driver as symbols / icons, and the driver taps the appropriate symbol / icon.

[0015] In a further step of the process, initial feedback from the occupant is received.

[0016] This can be done, for example, via a microphone as audio feedback or by pressure via a touch display.

[0017] In a further step of the procedure, the second class is extracted from the occupant's first feedback.

[0018] If the feedback is provided as audio feedback, speech recognition may need to analyze the audio feedback to extract the second class. For example, if the passenger responds "This is a carriage," only the word "carriage" needs to be extracted as the second class.

[0019] In a further step of the method, a data set is stored which comprises at least the second class, at least one object property and a derivative of the sensor data.

[0020] The dataset can also only contain a correlation between the second class, the at least one object property, and the derivative of the sensor data. This means that the actual data is not stored together in one data structure. This allows the different components of the dataset to be stored in a distributed manner.

[0021] A derivative of the sensor data can be the sensor data itself or a modification of it, for example, compressed sensor data or only the part of the sensor data that contains object properties. This can potentially save storage space.

[0022] In an advantageous embodiment, an identifier of the occupant and / or the vehicle is stored in the data set. This allows conclusions to be drawn about the user of the method. Although the identifier is unique, it can be anonymous. For example, this information can be used to exclude occupants or vehicles from the method that intentionally attempt to establish false classes. The occupant can be identified, for example, via voice recognition, their connected mobile device such as a smartphone, or facial recognition using interior cameras, which allows assignment to an existing identifier or the creation of a new one. An identifier for a vehicle can be realized, for example, via a vehicle serial number.

[0023] In an advantageous embodiment, an artificial intelligence is trained with at least the derivative of the sensor data or the object properties as input and with the second class as output.

[0024] Machine learning methods, such as a neural network (NN) or a convolutional neural network (CNN), are particularly suitable for this. These can receive sensor data, such as image data, as input. At the same time, the correct class, namely the second class, is known and can be provided to the NN or CNN as output. In a training phase, the weights of the connections between the artificial neurons of the (C)NN are updated using an error function. In an advantageous embodiment, the method comprises further steps if the confidence value is above a second threshold:

[0025] - Asking the occupant of the vehicle to confirm the first class of the object,

[0026] - Receiving a second feedback from the occupant,

[0027] - Extracting an indicator from the second feedback of the occupant, where the indicator indicates a truth value whether the object belongs to the first class, and

[0028] - Save the indicator.

[0029] For example, the request can be made via the vehicle's audio system, asking the driver, "Is there a carriage in front of us?" The driver's response can be provided via voice feedback or a touch display. Here, too, it may be necessary to analyze the feedback to extract the indicator. For example, if the passenger answers "Yes, that's a carriage," this can be converted into a binary value of a Boolean variable.

[0030] According to this implementation, the inmate is only asked to provide feedback if the object was detected with a confidence above a second threshold. The second threshold is typically higher than the first threshold, for example, 0.99 or 99%. However, if the inmate's response also suggests a misidentified class, the inmate may be attempting to introduce additional or incorrect classes into the process. The indicator can therefore be used to identify such users (trolls) of the process.

[0031] The indicator is preferably stored in correlation with the occupant. Storage in correlation with the underlying data set, in particular the sensor data and the object properties, is also advantageous. With the help of this correlation or information, artificial intelligence can be trained more specifically. According to a second aspect of the invention, a computer-implemented method for training artificial intelligence comprises a step in which a method as described above for the relational storage of object properties of an object is carried out for a plurality of occupants. In this process, a plurality of indicators is generated and stored.

[0032] In a further step of the process, a number of indicators is calculated for each occupant or vehicle. Using the identifier, indicators can be assigned to the occupants or vehicles. In particular, the number indicates how often a specific occupant or vehicle has provided feedback indicating that an object was not correctly classified, despite a high confidence value for the classification.

[0033] In a further step of the process, the artificial intelligence is trained only with data sets whose identifiers have a number of indicators below a third threshold. The third threshold defines a limit up to which an occupant or vehicle is still considered trustworthy. For example, above a third threshold of "5," any further feedback from the affected occupant is not used to train the artificial intelligence.

[0034] According to a third aspect of the invention, a computer-implemented method for generating a class relating to an object comprises a step in which a method as described above for relationally storing object properties of an object for a plurality of occupants is performed. A plurality of data records are thereby generated and stored.

[0035] In a further step of the process, the data records are divided into groups, with data records that have similar object properties and / or similar second classes being grouped into a specific group.

[0036] For example, data records with identical second classes, i.e., the class suggested by the occupants, are grouped together. Data records with similar object properties, such as similar geometric patterns such as points or lines, can also be added to the group. Similarity between such patterns can be calculated, for example, using geometric deviation. Similarity between second classes can be calculated, for example, using translation if the majority of occupants come from different countries and therefore suggest different words for the second class, but actually mean the same thing. The same applies to occupants who use synonymous words for the same object class, such as "carriage" and "horse-drawn carriage."

[0037] In a further step of the process, a count is calculated for each group. In other words, it calculates how many occupants or vehicles have already consistently classified certain object properties.

[0038] In a further step of the process, a third class is created for those groups whose number exceeds a fourth threshold. In other words, a new class should not be created from a single response.

[0039] Rather, swarm intelligence should be implemented, meaning that a new class is only generated when multiple responses have consistently classified a previously unknown object. For example, the fourth threshold value for this could be "10." Thus, 10 occupants must consistently classify an unknown object before this class is assigned to the object.

[0040] In a further step of the process, the third class is assigned to the data sets in the corresponding groups. This creates a correlation between the occupants' feedback, i.e., the class of the previously unknown object, and the data sets underlying the feedback.

[0041] In an advantageous embodiment, an artificial intelligence is trained with at least the derivative of the sensor data or the object properties as input and the third class as output. Since the occupants now have a reliable assessment of what type of object was detected, the artificial intelligence can be trained more reliably.

[0042] According to a fourth aspect of the invention, a computer program product comprises instructions which, when executed by a computer, cause the computer to execute a method as described above. The computer program product can be written in a programming language, for example, Python or C++.

[0043] According to a fifth aspect of the invention, a computer-readable storage medium comprises instructions that, when the program is executed by a computer, cause the computer to execute a method as described above. The computer-readable storage medium can be implemented, for example, as an SSD (solid-state disk) or as a flash memory. The computer-readable storage medium can also store other data, for example, sensor data and / or data that is (temporarily) stored during the execution of the method.

[0044] According to a sixth aspect of the invention, a data carrier signal transmits the computer program product as described above. The data carrier signal can be transmitted via a cable, for example, via a CAN bus (controller area network) or Ethernet cable, or wirelessly via Wi-Fi, Bluetooth, or the like.

[0045] According to a seventh aspect of the invention, a device for classifying an object has an evaluation unit which is configured such that it can carry out a method as described above. The evaluation unit can, for example, have a processor. The evaluation unit can be a single component or distributed across multiple components. For example, parts of the method can be executed on a server and other parts on a local processor in a vehicle. Furthermore, the device has at least one memory unit which is communicatively connected to the evaluation unit. The computer program product as described above can, for example, be stored on the memory unit. It is also possible for data to be stored in databases. The memory unit can be a single component or distributed across multiple components.For example, parts of the data can be stored locally in the vehicle, while others can be stored on a server.

[0046] Furthermore, the device has at least one sensor which is communicatively connected to the evaluation unit, wherein the sensor generates sensor data in which the object is detected, wherein the at least one sensor is arranged on a vehicle.

[0047] In an advantageous embodiment, at least one sensor is a lidar, a radar, or a camera. These sensors are readily available and inexpensive and are usually already installed in many vehicles for other functions.

[0048] According to an eighth aspect of the invention, a vehicle comprises a device as described above.

[0049] Summary of the characters

[0050] The invention is explained in more detail below using exemplary embodiments and figures. The figures show:

[0051] Figure 1: A driving scene to describe the problem underlying the invention;

[0052] Figure 2: A first embodiment of a method for relational storage of object properties;

[0053] Figure 3: A second embodiment of a method for relational storage of object properties; Figure 4: An embodiment of a method for training artificial intelligence;

[0054] Figure 5: An embodiment of a method for creating a class relating to an object; and

[0055] Figure 6: A device for classifying an object.

[0056] Detailed description of the characters

[0057] Figure 1 shows a driving scene to describe the problem underlying the invention.

[0058] Figure 1 shows the driving scene from the perspective of a driver of a vehicle 100. A first road user 102 and a second road user 104 are driving on the road in front of the vehicle 100. The first road user 102 is a carriage, the second road user 104 is a motor vehicle.

[0059] The vehicle 100 has a camera 106, in whose field of view 108 objects 102, 104 on the road in front of the vehicle 100 can be detected. The vehicle 100 also has a display device 110. An algorithm has detected and classified the road users 102, 104 from image data 172 of the camera 106 using an object recognition and classification algorithm. Furthermore, a digital model of the road users 102, 104 was created with a first digital image 112 and a second digital image 114, which are displayed on the display device 110.

[0060] However, the first road user 102 was incorrectly classified. Instead of a carriage, the first digital image 112 displayed is a truck because the classification algorithm was not trained accordingly. This can confuse the driver or cause an automated vehicle to perform a risky maneuver. Figure 2 shows a first embodiment of a method for relationally storing object properties 116 of an object 102, 104.

[0061] In a first obtaining step 118, sensor data 172,174,176,178 are obtained in which the object 102,104 is detected.

[0062] In a detection and classification step 120, the object 102, 104 is detected, or localized, and classified in the sensor data 172, 174, 176, 178. This step can be performed using conventional image processing methods. Typically, a neural network (NN) is trained with corresponding data and a predefined class.

[0063] In this step, object properties such as geometric patterns and their distribution are determined for the object 102,104. Furthermore, the object 102,104 is assigned a class, for example, "pedestrian," "bicycle," "tree," "motor vehicle," "truck," and the like. A confidence value is generated, which is a measure of how certain the classification into the class is. The confidence value is typically specified as a floating-point number such as 0.97 or as a percentage value such as 97%.

[0064] In a first decision step 122, it is checked whether the confidence value is below a first threshold 192. If this is not the case (f-branch), a reliable classification is present, and the process 116 simply continues. However, if the confidence value is less than the first threshold 192, a reliable classification is not present. The object 102,104 is considered unknown.

[0065] In this case (t-branch), a first request step 124 is performed. In this step, an occupant of the vehicle 100 is requested to provide a second class relating to the object 102, 104. The request can be made, for example, via an audio signal, in which the occupant is asked, "What type of object is that in front of the vehicle?" The second class is the correct class to which the object 102, 104 belongs. The occupant's response is received in a first receiving step 126. This can again be done via an audio signal. The occupant can, for example, respond, "That's a carriage."

[0066] In a first extraction step 128, the second class is extracted from the occupant's feedback. For example, the word "coach" is isolated from the audio signal and used as the second class.

[0067] In a first storage step, a data set 193 is stored, which includes at least the second class, at least one object property, and a derivative of the sensor data 172, 174, 176, 178. A derivative can be the sensor data 172, 174, 176, 178 itself, but it can also be a compressed version or a relevant part of the sensor data 172, 174, 176, 178.

[0068] Figure 3 shows a second embodiment of a method for relational storage of object properties 116.

[0069] Most of the steps in Figure 3 (see dashed line) have already been explained in Figure 2 and will therefore not be discussed again here. However, Figure 3 also shows further steps that will now be explained.

[0070] If the first decision step 122 determines that the confidence value is above the first threshold 192, it can be assumed that the object 102, 104 has been classified with sufficient certainty. For example, the first threshold can be 0.95 or 95%.

[0071] Subsequently, in a second decision step 134, it is checked whether the confidence value is above a second threshold 194. For example, this is 0.99 or 99%. If this is the case (t-branch), the object 102, 104 is very reliably classified into the first class. The subsequent steps 136, 138, 140, and 142 are intended to ensure that the method 116 is not intentionally misused by generating incorrect classifications. To this end, the occupant is first requested to confirm the first class in a second request step 136. This can again be implemented, for example, via an audio signal in which the occupant is asked, "Is there a carriage in front of us?"

[0072] Subsequently, a second response from the occupant is received in a second receiving step 138. The response could be, for example, "Yes, that's a carriage" or "No, that's a bicycle."

[0073] Therefore, in a second extraction step 140, the second feedback from the occupant is evaluated and assessed to determine whether it is consent or not. This is represented by an indicator, for example, a Boolean variable. The indicator thus indicates a truth value indicating whether the object 102, 104 belongs to the first class. The indicator can, for example, have binary states (true / false) or exhibit a value along a spectrum. The indicator therefore stores whether the occupant provides trustworthy answers, because it is to be expected that a positive answer will be given given the high level of confidence.

[0074] The indicator is then stored in a second storage step 142. The storage occurs in such a way that the indicator is linked to the identifier of the occupant or the car. In other words, information is generated that links the occupant's response to their identity.

[0075] Subsequently, the training step 132 can be designed more specifically by only including data sets 193 of inmates in the training phase of the artificial intelligence if the inmate is classified as trustworthy via the indicator.

[0076] Figure 4 shows an embodiment of a method for training an artificial intelligence 144. In a first execution step 146, the method from Figure 3 is carried out continuously (illustrated by the circular arrow) for a plurality of occupants or vehicles 100. In the process, a corresponding plurality of indicators is also generated and stored.

[0077] In a first calculation step 148, a count is calculated for the multitude of indicators. The count indicates how often an inmate has given a trustworthy answer, since the indicator is only generated for objects 102, 104 that have been classified with very high certainty. The count is therefore a measure of the inmate's trustworthiness.

[0078] Subsequently, the number is compared with a third threshold 196 in a third decision step 150. If the number is lower than the third threshold 196 (f-branch), all data records 193 originating from the occupant or vehicle 100 can be used to train the artificial network in training step 132. If the number is higher than the third threshold 196, the data records 193 associated with the occupant or vehicle 100 are excluded from training step 132. The third threshold 196 is thus a measure of the extent to which an occupant or vehicle 100 should be considered trustworthy.

[0079] Figure 5: An embodiment of a method for generating a class 152 relating to an object.

[0080] In a second implementation step 154, the method from Figure 2 is carried out continuously (illustrated by the circular arrow) for a plurality of occupants or vehicles 100. In the process, a corresponding plurality of data records 193 are also generated and stored.

[0081] In a classification step 156, the data records 193 are divided into groups.

[0082] The members of a group are characterized by a similarity in their object properties and / or their second classes. For example, a certain arrangement (e.g., distances) of points, lines, or vertices can establish a similarity.

[0083] In a second calculation step 158, a count is then calculated for each group. The count is thus a measure of how many different occupants or vehicles 100 have transmitted a matching classification for an object 102, 104.

[0084] Subsequently, in a fourth decision step 160, it is checked whether the number exceeds a fourth threshold 198. The fourth threshold 198 is a measure of whether sufficient similar feedback is available regarding a class of an object 102, 104. If this is the case (t-branch), a generation step 162 is executed.

[0085] In the first generation step 162, a third class is created for the data records in the groups. The third class may, in particular, correspond to the second class, ie, the class transmitted by the occupants is adopted.

[0086] In an assignment step 164, the third class is assigned to the data records 193. Thus, the data records 193 are now provided with a reliable class.

[0087] The artificial intelligence can then be trained using the data sets 193 provided with the third class. The sensor data 172, 174, 176, 178 (or a derivative) present in the data sets 193, or the object properties, or a combination thereof, serve as input for the artificial intelligence, with the class present in the data sets 193 serving as the default for the output. In training step 132, the internal properties of the artificial intelligence, for example, weights between artificial neurons, can be adjusted.

[0088] Figure 6 shows a device 166 for classifying an object 102, 104. The device 166 comprises a camera 106, a lidar sensor 168, and a radar sensor 170. The camera 106 generates image data 172, the lidar sensor 168 generates lidar data 174, and the radar sensor 170 generates radar data 176. The sensor data 172, 174, 176 generated by these sensors 106, 168, 170 are highly suitable for detecting objects 102, 104 because they contain patterns characteristic of the object.

[0089] Before the sensor data 172, 174, 176 are further processed, they are merged into fused sensor data 178 in the example of Figure 6. These are then forwarded to a central computer (HPC; English: high-performance computer) 180. The HPC 180 has an evaluation unit 182, which in the example of Figure 6 is implemented as a processor 182. In addition, the HPC 180 has a non-volatile memory 184, which is communicatively connected to the processor 182 with the participation of a volatile memory 200.

[0090] A computer program product 186 is stored on the non-volatile memory 184, which contains instructions that, when executed by the processor 182, cause the processor 182 to execute the methods 116, 144, 152 from Figures 3 to 5. Furthermore, a first database 188 is stored on the non-volatile memory 184, in which, among other things, the fused sensor data 178 can be stored. Furthermore, a first neural network (NN) 190 is stored on the non-volatile memory 184, which has been trained to detect and classify objects 102, 104. Furthermore, a first threshold value 192 and a second threshold value 194 are stored on the non-volatile memory 184, which are necessary for executing the methods 116, 144, 152 from Figures 3 to 5.

[0091] The device 166 further comprises an external memory 202, which is implemented as a cloud storage 202. This memory comprises a second neural network 204 and a second database 206. Furthermore, a third threshold value 196 and a fourth threshold value 198 are stored on the cloud storage 202, which are necessary for executing the methods 116, 144, 152 from Figures 3 to 5. The device 166 is communicatively connected to an audio system 208 of a vehicle 100 in a bidirectional manner.

[0092] The computer program product 186 is now executed on the processor 182. The fused sensor data 178 is evaluated by the first NN 188. The object 102, 104 is detected and classified in the fused sensor data 178. Object properties such as geometric patterns are recognized in the fused sensor data 178. The object properties are characteristic of a first class, which is assigned to the object 102, 104. The first NN 188 implicitly learned these characteristics in a previous training phase. The first NN 188 also generates a confidence value 191, which indicates how reliable the classification into the first class was.

[0093] A check is now carried out to determine whether the confidence value 191 is below the first threshold value 192, for example, 0.95 or 95%. If this is the case, no reliable classification is available, and an occupant of the vehicle 100 is prompted via the audio system 208 to provide a second class by asking, "What type of object is that in front of the vehicle?" The response is received, and the second class is extracted from it. If the occupant answers, for example, "That's a carriage," "carriage" is extracted as the second class. The second class is then stored as a data record 193 in the first database 186 together with the object properties, the fused sensor data 178, and an identifier. The identifier is a specific value assigned to each occupant or vehicle 100. It ensures that data records 193 can be assigned to a specific source. This is preferably done anonymously.

[0094] However, if the confidence value 191 is above the second threshold 194, for example 0.99 or 99%, a very certain classification is present and the occupant is prompted to provide confirmation by asking "Is this a truck?" If the occupant answers, for example, "Yes, this is a truck" or "No, this is not a truck" or "This is an airplane," an indicator 195 is extracted from this feedback. In the example of Figure 6, the indicator 195 is generated as a binary Boolean variable, i.e., with the values ​​"true" or "false." The indicator 195 is then saved together with the underlying data set 193. It can be added to the data set 193, or a reference to the data set 193 can be appended to the indicator 195.

[0095] The data records 193 and indicators 195 in the first database 186 are added wirelessly to the second database 206 in the cloud 202. A multitude of feedbacks from a multitude of occupants or vehicles 100 are collected in the cloud 202.

[0096] In the cloud 202, a check is now carried out to determine how many data sets 195 of an occupant or vehicle 100 have an indicator 195 indicating negative feedback. Negative feedback indicates that the occupant or vehicle 100 is not providing reliable answers, since the classification of the object 102, 104 was very likely correct. If the number of these data sets 195 is above the third threshold, for example, "10," these data sets 195 are not used to train the second NN 204.

[0097] Furthermore, grouping is performed in the cloud 202. Data records 193 are assigned to a group if they have similar second classes and similar object properties, ie, if the plurality of occupants or vehicles 100 have made a matching classification.

[0098] If the number of data records in a group exceeds a fourth threshold, for example “20”, it can be assumed that the object properties in the data records 193 actually correspond to the second class of the data records 193.

[0099] Therefore, a third class is created for the group, which in the example shown in Figure 6 corresponds to the second. However, the third class can also differ from the second, for example, because different languages ​​were used for the second class. The third class is assigned to the corresponding members in the group.

[0100] The second NN 204 is now trained with the data sets 193, as long as they were not excluded from training due to an excessive number of indicators. After the training phase, the second NN 204 has implicitly learned to recognize the new third class and can be transmitted back to the vehicle 100. In other words, the first NN 188 is replaced by the second NN 204, or the internal properties such as weights are transmitted to the first NN 188 for updating. The vehicle 100 can now recognize the new, third class and, based on this, create better planning of driving maneuvers or generate more precise representations of digital objects 112, 114 for a display device 110 of the vehicle 100.

[0101] List of reference symbols

[0102] 100 vehicles

[0103] 102 First road user; carriage

[0104] 104 Second road user; car

[0105] 106 Camera

[0106] 108 field of view

[0107] 110 Display device

[0108] 112 First digital image

[0109] 114 Second digital image

[0110] 116 Methods for the relational storage of object properties

[0111] 118 First conservation step

[0112] 120 detection and classification steps

[0113] 122 First decision step

[0114] 124 First prompt step

[0115] 126 First reception step

[0116] 128 First extraction step

[0117] 130 First storage step

[0118] 132 training steps

[0119] 134 Second decision step

[0120] 136 Second request step

[0121] 138 Second reception step

[0122] 140 Second extraction step

[0123] 142 Second storage step

[0124] 144 Methods for training artificial intelligence

[0125] 146 First implementation step

[0126] 148 First calculation step

[0127] 150 Third decision step

[0128] 152 Methods for creating a class relating to an object

[0129] 154 Second implementation step

[0130] 156 Classification step

[0131] 158 Second calculation step

[0132] 160 Fourth decision step

[0133] 162 Generation step

[0134] 164 Assignment step

[0135] 166 Device

[0136] 168 Lidar sensor Radar sensor

[0137] Image data

[0138] Lidar data

[0139] Radar data

[0140] Fused sensor data

[0141] central computer; HPC

[0142] Evaluation unit; Processor Non-volatile memory Computer program product First database

[0143] First Neural Network

[0144] Confidence value

[0145] First threshold

[0146] data set

[0147] Second threshold

[0148] indicator

[0149] Third threshold

[0150] Fourth threshold

[0151] Volatile memory

[0152] External storage; Cloud Second neural network Second database

[0153] audio system

Claims

Patent claims 1 . A computer-implemented method for relationally storing object properties (116) of an object (102, 104), comprising the steps of: a) obtaining (118) sensor data (172, 174, 176, 178) in which the object (102, 104) is detected, wherein the sensor data (172, 174, 176, 178) are obtained from at least one sensor arranged on a vehicle (100). (106, 168, 170), b) detection and classification (120) of the object (102, 104) by means of the sensor data (172, 174, 176, 178), wherein a first class is assigned to the object (102, 104) and at least one object property is determined, thereby generating a confidence value (191), c) requesting (124) an occupant of the vehicle (100) to provide a second class relating to the object (102, 104) if the confidence value (191) is below a first threshold value (192), d) receiving (126) a first feedback from the occupant, e) extracting (128) the second class from the first feedback from the occupant, and f) storing (130) a data set (193) which contains at least the second class, the at least one object property and a derivative of the sensor data (172,174,176,178).

2. Computer-implemented method according to claim 1, characterized in that an identifier of the occupant and / or the vehicle (100) is stored in the data set (193).

3. Computer-implemented method according to claim 1 or 2, characterized in that an artificial intelligence (190,204) is trained at least with the derivative of the sensor data (172,174,176,178) or the object properties as input and with the second class as output.

4. Computer-implemented method according to one of claims 2 or 3, characterized in that the method (116) comprises the following further steps if the confidence value (191) is above a second threshold value (194): a) requesting (136) the occupant of the vehicle (100) to confirm the first class of the object (102, 104), b) receiving (138) a second feedback from the occupant, c) extracting (140) an indicator (195) from the second feedback from the occupant, wherein the indicator (195) indicates a truth value as to whether the object (102,104) belongs to the first class, and d) storing (142) the indicator (195).

5. A computer-implemented method for training an artificial intelligence (144), comprising the steps of: a) carrying out (146) the method (116) according to claim 4 for a plurality of occupants, thereby generating a plurality of indicators (195) which are stored, b) calculating (148) a number of the indicators (195) for each occupant or for each vehicle (100), c) training (132) the artificial intelligence (190, 204), wherein only those data sets (193) are used for the training for whose identifier (195) the number of indicators (195) below a third threshold (196) lies.

6. A computer-implemented method for generating a class (152) relating to an object, comprising the steps of: a) carrying out (154) the method (116) according to claim 1 or 2 for a plurality of occupants, thereby generating a plurality of data records which are stored, b) dividing (156) the data records (193) into groups, wherein data records (193) which have similar object properties and / or similar second classes are grouped in a specific group, c) calculating (158) a number for each group, d) generating (162) a third class for groups whose number exceeds a fourth threshold value (198), and e) assigning (164) the third class to the data records (193) in the corresponding groups.

7. Computer-implemented method according to claim 6, characterized in that an artificial intelligence (190,204), at least with the derivative of the sensor data (172,174,176,178) or the object properties as input and with the third class as output, is trained.

8. A computer program product (186) comprising instructions which, when the program is executed by a computer, cause the computer to carry out a method (116,144,152) according to one of the preceding claims 9. A computer-readable storage medium (184) comprising instructions which, when executed by a computer, cause the computer to perform a method (116, 144, 152) according to any one of claims 1 to 7.

10. A data carrier signal which transmits the computer program product (186) according to claim 8.

11. A device for classifying an object (166), comprising: a) an evaluation unit (182) which is configured such that it can carry out a method (116, 144, 152) according to one of claims 1 to 7, b) at least one memory unit (184, 202) which is communicatively connected to the evaluation unit (182), and c) at least one sensor (106, 168, 170) which is communicatively connected to the evaluation unit (182), wherein the at least one sensor (106, 168, 170) generates sensor data (172, 174, 176, 178) in which the object (102, 104) is detected, wherein the at least one sensor (106, 168, 170) is arranged on a vehicle (100).

12. Device according to claim 11, characterized in that the at least one sensor (106,168,170) is a lidar (168), a radar (170) or a camera (106).

13. A vehicle (100) comprising a device (166) according to claim 11 or 12.

Images