Automatically identifying faulty signature for autonomous driving application

Abstract

To provide a method for automatically identifying faulty signatures for autonomous driving applications, and a non-transitory computer-readable medium.SOLUTION: A method includes: receiving, by a processing circuit, a signature; matching the signature to a group of first signatures that are untagged and are randomly obtained; identifying, based on the matching, first top matching signatures; matching the signature to a group of second signatures that are untagged and are correctly or erroneously indicative of a detection of reference elements; identifying, based on the matching of the signature to the group of second signatures, second top matching signatures; determining an overlap between the first top matching signatures and the second top matching signatures; and determining whether the signature is faulty or faultless based on the overlap.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to the autonomous driving domain, and more particularly to a method and non-transitory computer-readable storage medium for automatically identifying fault signatures for autonomous driving applications. [Background technology]

[0002] Sensing systems are a key building block of all modern advanced driver assistance systems (ADAS) and automotive (AV) solutions. They are responsible for detecting, tracking, and measuring driving-related entities such as road users, lanes, traffic signs, traffic signals, etc. The output of a sensing system is a 3D environment model, which is used as the basis for the vehicle's decisions and path planning.

[0003] All modern sensing systems are based on advanced deep learning techniques.

[0004] Deep learning models may be trained using supervised or unsupervised training, both of which have limitations that can introduce errors.

[0005] Therefore, in the case of autonomous driving applications, there is an increasing need to automatically detect errors resulting from the detection process applied by deep learning models. Summary of the Invention [Problem to be solved by the invention]

[0006] The present disclosure provides a method and a non-transitory computer-readable storage medium for identifying fault signatures. [Means for solving the problem]

[0007] In a first aspect of the present disclosure, a method for automatically identifying fault signatures for an autonomous driving application is provided, the method including receiving a signature by a processing circuit, matching the signature to a set of unlabeled and randomly obtained first signatures, identifying a first top matching signature based on the matching, matching the signature to a set of unlabeled second signatures that correctly or incorrectly indicate detection of a reference element, identifying a second top matching signature based on the matching between the signature and the set of second signatures, determining an overlap between the first top matching signature and the second top matching signature, and determining whether the signature is faulty based on the overlap.

[0008] In another aspect of the present disclosure, a non-transitory computer-readable medium is provided that stores instructions that, when executed by at least one processor, cause the at least one processor to perform operations for automatically identifying fault signatures for an autonomous driving application, including receiving a signature by a processing circuit, matching the signature to a set of unlabeled and randomly obtained first signatures, identifying a first top matching signature based on the matching, matching the signature to a set of unlabeled second signatures that correctly or incorrectly indicate detection of a reference element, identifying a second top matching signature based on matching the signature with the set of second signatures, determining an overlap between the first top matching signature and the second top matching signature, and determining whether the signature is faulty based on the overlap.

[0009] It should be understood that all combinations of the above concepts and additional concepts described in more detail herein are considered part of the subject matter disclosed herein, e.g., all combinations of claimed subject matter appearing at the end of this publication are considered part of the subject matter disclosed herein. [Brief explanation of the drawings]

[0010] The embodiments of the present disclosure can be more fully understood and clarified through the following detailed description taken in conjunction with the drawings, in which: [Figure 1A] FIG. 1 is a block diagram of an in-vehicle and out-vehicle vehicle system according to some embodiments of the present disclosure. [Figure 1B] 1 is a block diagram of an example of an in-vehicle and an out-of-vehicle vehicle system according to an embodiment of the present disclosure. FIG. [Figure 2] 1 is a block diagram of an example of an in-vehicle and an out-of-vehicle vehicle system according to an embodiment of the present disclosure. FIG. [Figure 3] FIG. 1 is a flow chart diagram of identifying fault signatures in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011] A method, a non-transitory computer-readable storage medium, and a system are provided for automatically identifying fault signatures for autonomous driving applications.

[0012] Examples of autonomous driving applications include applications such as ADAS applications, self-driving applications, etc.

[0013] The different figures show examples of units and / or software and / or information items and / or steps and / or components. These examples are provided for simplicity of explanation. At least one of the units and / or software and / or information items and / or steps and / or components is optional or mandatory.

[0014] A computer-implemented method and non-transitory computer-readable medium are provided that uses signatures that are identified as faulty or fault-free, where the identification process is cost-effective, does not require manual tagging, and is reliable even when the signature is not associated with a defined class. Identifying a signature as faulty or fault-free is reliable.

[0015] 1A, 1B, and 2 show an example of a vehicle 100, a network 132, and a remote computerized system 134.

[0016] In FIG. 1A, a vehicle 100 is shown to include a sensing system 110, a communication system 130, one or more memory and / or storage units 120, a control unit 125′, and a network 132 for communicating with a remote computerized system 134.

[0017] One or more memory and / or storage units 120 are shown to store information 191, metadata 192, software 193, and operating system 194. Information 191, metadata 192, software 193, and operating system 194 are necessary to perform one or more methods (e.g., method 200) described herein.

[0018] Processor 126 of FIG. 1A is shown to include multiple processing units 126(1) through 126(J), where J is an integer greater than one.

[0019] 1B and 2, the control unit 125′ is replaced with different components such as an advanced driver assistance system (ADAS) control unit 123, an autonomous driving control unit 122, a vehicle computer 121, and a controller 125. Note that only some of these components may be included in the vehicle.

[0020] 1B and 2 further provide examples of one or more types of information 191 and metadata 192 and / or software 193 stored in one or more memory and / or storage units 120.

[0021] The communication system 130, the one or more memory and / or storage units 120, and the processing system 124 may form a computerized system, which may include one or more other systems and / or units, such as the sensing system 110 (at least the image signal processor 114), the ADAS control unit 123, the autonomous driving control unit 122, the vehicle computer 121, and the controller 125.

[0022] The sensing system 110 includes an optical device 111, a sensing element group 112, readout circuitry 113, and an image signal processor 114. Following the optical device 111 are sensing elements, for example, a sensing element line or a sensing element array, forming the sensing element group 112. Following the sensing element group 112 are readout circuitry 113, which reads out detection signals generated by the sensing element group 112. The image signal processor 114 is configured to perform initial processing of the detection signals, for example, by improving the quality of the detection information, performing noise reduction, etc. The sensing system 110 is configured to output one or more sensory information units (SIUs).

[0023] The communication system 130 is configured to enable communication between one or more memories and / or memory units 120 and / or sensing system 110 and / or any additional units and / or network 132 (for communicating with remote computerized systems).

[0024] The controller 125 is configured to control the operation of the sensing system 110 and / or one or more memories and / or memory units 120 and / or one or more additional units other than the controller.

[0025] The ADAS control unit 123 is configured to control the ADAS operation.

[0026] The autonomous driving control unit 122 is configured to control the autonomous driving of the autonomous vehicle.

[0027] The vehicle computer 121 is configured to control the operation of the vehicle, particularly the engine, transmission, and any other vehicle systems or components.

[0028] Processing system 124 may include processor 126 and one or more other processors and is configured to perform any of the methods described herein.

[0029] The one or more memory and / or storage units 120 are configured to store firmware and / or software, one or more operating systems, data, and metadata necessary to perform any of the methods described herein.

[0030] FIG. 1B illustrates one or more memory and / or storage units 120 as storing: · First set of signature 171. · Second set of signature 172. First top matching signatures 181. This may include the best matching signatures (e.g., the top 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 15, 20, 25, 30, or any other number of best matching signatures). According to an embodiment, a match is determined based on the distance between the (evaluated) signature and a signature in the first signature of the set. Any distance can be calculated. According to an embodiment, a signature includes an index for data lookup, and a match includes an exact match. A second top matching signature 182, which may include the best matching signatures (e.g., the top 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 15, 20, 25, 30, or any other number of best matching signatures). The signature generation software 161 is configured to generate a signature. The signature may be generated based on the SIU, but may also be generated based on other information. When the processing system 124 executes the signature generation software, a detector can be provided. An example of a signature and a signature generator as an example of a detector are shown in U.S. Patent Application Serial No. 17 / 309064, Publication No. 2022 / 0041184, which is incorporated herein by reference. Signature matching software 162 is configured to (a) match the signature with a signature in first set of signatures 171 to provide a first top matching signature 181, and (b) match the signature with a signature in second set of signatures 172 to provide a second top matching signature 182. The matching-based signature status determination software 163 is configured to determine whether a signature is faulty (i.e., not faulty) based on an overlap between the first top matching signature 181 and the second top matching signature 182. The overlap may be a signature that is included in both the first top matching signature and the second top matching signature. Parameter tuning software 164 is configured to tune any parameters related to determining whether a signature is normal or faulty. One or more signatures 165 generated by the detector, including one or more signatures that can be evaluated to determine whether they are good or bad signatures. ·Operation system 166. Detector signature accuracy history information 167 represents the accuracy of the detector (reflected by having a good or faulty signature). This history information can provide an indication of whether the detector is improving accuracy, maintaining accuracy, or degrading signatures over time. Parameter tuning software can use this information. Additional software 168 that can be used to perform any other function in the vehicle and / or any other unit shown in FIG. 1B. · Detector verified signature 175 includes the signature of the detector that was tested as normal or faulty. · A whitelist of assumed normal (fault-free) signatures generated by the detector176. · A blacklist of assumed faulty signatures generated by the detector177. A training dataset 178 for training the machine learning process. The training dataset 178 can be updated by adding signatures that are discovered as normal, and these signatures can be marked as correct. This training dataset 178 can also be updated by adding signatures as faulty (sometimes referred to as faulty). A test dataset 179 for testing the machine learning process. The test dataset 179 can be updated by adding signatures that are discovered as normal, and these signatures can be marked as correct. This test dataset 179 can also be updated by adding signatures that are faulty (sometimes referred to as being faulty).

[0031] The vehicle computer 121 may communicate with an engine control module, a transmission control module, a powertrain control module, and the like.

[0032] The memory and / or storage unit 120 is shown as storing software, any reference to software should be made with the necessary amendments to apply to code and / or firmware and / or instructions and / or commands, etc.

[0033] Any reference to one unit or project should be amended as necessary to apply to multiple units or projects, for example, any reference to a processor should be amended as necessary to apply to multiple processors, and any reference to communications system 130 should be amended as necessary to apply to multiple communications systems.

[0034] According to an embodiment, the one or more memory and / or storage units 120 may include one or more memory units, each of which may include one or more memory bodies.

[0035] According to an embodiment, the one or more memory and / or storage units 120 include volatile memory and / or non-volatile memory. The one or more memory and / or storage units 120 may be random access memory (RAM) and / or read-only memory (ROM).

[0036] According to an embodiment, the non-volatile storage unit is a mass storage device capable of providing non-volatile storage of computer code, computer-readable instructions, data structures, program modules, and other data to a processor or any other unit of the vehicle. For example, the mass storage device may be, but is not limited to, a hard disk, a removable disk, a removable optical disk, a tape cartridge or other magnetic storage device, a flash memory card, a CD-ROM, a digital multifunction disk (DVD) or other optical storage, a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0037] Any content may be stored in any part of a memory unit or in any type of memory unit.

[0038] According to an embodiment, the at least one memory unit stores at least one database, such as any database known in the art, e.g., DB2®, Microsoft® Access, Microsoft® SQL Server, Oracle®, MySQL, PostgreSQL, etc.

[0039] The various units and / or components communicate with each other using any communication element and / or protocol. Different communication elements than the communication system 130 may be provided.

[0040] 1A, 1B and 2 show that the communication system 130 is in communication with various processors and / or units and a network 132.

[0041] Communications system 130 may include a bus. This bus represents one or more of several possible types of bus architectures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor bus or local bus using any of a variety of bus architectures. By way of example, these architectures may include an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, an Accelerated Graphics Port (AGP) bus and a Peripheral Component Interconnect (PCI), a PCI-Express bus, a Personal Computer Memory Card Industry Association (PCMCIA), a Universal Serial Bus (USB), etc. This bus, and all buses specified herein, may also be implemented via wired or wireless network connections and their respective subsystems.

[0042] The network 132 is located outside the vehicle and is used for communication between the vehicle and at least one remote computing system. By way of example, the remote computing system may be a personal computer, laptop computer, portable computer, server, router, network computer, peer-to-peer device, or other public network node. Logical connections between the processor and any remote computing system may be realized via local area networks (LANs) and general wide area networks (WANs). Such network connections may be made through network adapters (which may belong to the communication system 130), which may be implemented in wired and wireless environments. Such network environments are common and typical in large networks such as offices, enterprise-wide computer networks, intranets, and the Internet.

[0043] It should be noted that at least a portion of the content shown as being stored in one or more memory / storage units 120 may be stored external to the vehicle. It should also be noted that the processor may evaluate signatures produced by multiple detectors.

[0044] FIG. 2 shows an example of a vehicle 100, a network 132, a remote computerized system 134, and also shows an external memory / storage unit 136.

[0045] 2 differs from FIG. 1B in that it shows an external memory / storage unit 136 that stores the set of first signatures 171, the set of second signatures 172, a training data set 178, a test data set 179, a whitelist 176, and a blacklist 177. In contrast to FIG. 1B, the training data set 178, the test data set 179, the whitelist 176, and the blacklist 177 are not stored in the one or more memory / storage units 120.

[0046] A further difference between FIG. 2 and FIG. 1B is that one or more memory / storage units 120 are shown as storing verified signatures from multiple (K) detectors (from the verified signature of the first detector 175(1) to the verified signature of the Kth detector 175(K)).

[0047] A further difference between FIG. 2 and FIG. 1B is that one or more memory / storage units 120 are shown as storing one or more signatures from a plurality of (K) detectors (from the signature of the first detector 165(1) to the signature of the Kth detector 165(K)).

[0048] FIG. 3 illustrates an example computer-implemented method 200 for automatically identifying fault signatures for autonomous driving applications.

[0049] According to an embodiment, the method 200 includes receiving 210, by a processing circuit, a signature associated with an identification of an element, which is at least one of an object or a road scene. Thus, the signature can be associated with an ID or an object. Alternatively, the signature can be associated with an identification of a scene.

[0050] The scene may be, for example, at least one of: (a) vehicle position; (b) one or more weather conditions; (c) one or more context parameters; (d) road conditions; and (e) traffic parameters.

[0051] Various examples of road conditions include the roughness of the road, the level of road maintenance, the presence of potholes or other relevant road obstructions, whether the road is smooth, or whether it is covered with snow or other particles.

[0052] Various examples of traffic parameters and one or more context parameters include time (hour, day, period or year, specific time of a specific day, etc.), traffic load, distribution of vehicles on the road, one or more vehicle behaviors (aggressive, peaceful, predictable, unpredictable, etc.), presence of pedestrians near the road, presence of pedestrians near the vehicle, presence of pedestrians away from the vehicle, pedestrian behaviors (aggressive, peaceful, predictable, unpredictable, etc.), risks associated with driving near the vehicle, complexities associated with driving inside the vehicle, presence of at least one of a kindergarten, a school, a group of people (close to the vehicle), etc. The context parameters can be associated with the context of the detected information. The context can be associated with or surrounding the environment that forms the setting for an event, a sentence, or a concept.

[0053] Examples of contexts and context-based processing are provided in US patent application Ser. No. 16 / 035732, which is incorporated herein by reference.

[0054] According to an embodiment, step 210 includes accessing a memory unit or buffer that stores the signature generated by the detector.

[0055] According to an embodiment, step 210 is followed by step 220 and step 240 .

[0056] According to an embodiment, step 220 includes matching the signature with a set of first signatures that are unmarked and can be obtained in any manner, for example randomly, or in any manner that can be done without knowledge of the content represented by the signature and / or without consideration of the content represented by the signature (which may be unknown).

[0057] According to an embodiment, step 220 is followed by step 230 of identifying a first top matching signature based on the matches of step 220 .

[0058] According to an embodiment, step 240 includes matching the signature with a set of second signatures that are unlabeled and correctly or incorrectly indicate the detection of a reference element. For example, the second signatures are determined (correctly or incorrectly) by a detector to identify a reference element. The second signatures that correctly indicate the detection of a reference element may be true positive signatures or true negative signatures. The second signatures that incorrectly indicate the detection of a reference element may be false positive signatures or false negative signatures.

[0059] According to an embodiment, step 240 is followed by step 250 of identifying a second top matching signature based on the matches of step 240 .

[0060] According to an embodiment, steps 230 and 250 are followed by step 260 of determining overlap between the first top matching signature and the second top matching signature.

[0061] According to an embodiment, the overlapping motion feature appears in both the first top matching motion feature and the second top matching motion feature.

[0062] According to an embodiment, overlapping signatures appear in the first top matching signature, and signatures that are close enough (within a defined distance) appear in the second top matching signature.

[0063] According to an embodiment, step 260 is followed by step 270 of determining whether the feature is faulty based on overlap.

[0064] According to an embodiment, step 270 includes step 271 , step 272 and step 273 .

[0065] According to an embodiment, step 271 includes comparing the overlap to a threshold value, which may be dynamically updated and may be, for example, in the range of 1% to 99%, or any sub-range that is a part of the range of 1% to 99% (e.g., 5%, 15%, 20%, 30%, 35%, 45%, 50%, 55%, 60%, 70%, 75%, 80%, 85%, 90%, 95%, etc.), or any value range equal to N1 / N2, where N1 and N2 are positive numbers and N1 is less than N2.

[0066] According to an embodiment, step 272 includes determining that the signature is faulty if the overlap is below a threshold.

[0067] According to an embodiment, step 273 includes determining that the signature is not faulty if the overlap is higher than a threshold.

[0068] According to an embodiment, if the overlap is equal to a threshold value, it is determined that the classification of the signature is not faulty according to the defined rules.

[0069] According to an embodiment, step 270 is followed by step 280 of responding to the results of step 270 .

[0070] According to an embodiment, the response may include at least one of the following: Generate signature status indication - indicates whether the signature is faulty or not. The signature status indicator is stored in the memory unit and / or the memory unit. The signature status indication is transmitted over a communications link or channel. Mark instructions as faulty or not. The sensing is fed back to the signature generator which generates the received signature in step 210. · Trigger or request or instruct to remove a signature from the whitelist if a malfunction is detected. If no failure is detected, trigger or request or instruct to insert the signature into the whitelist. If a fault is detected, trigger or request or instruct to add the signature to the blacklist. · If a fault is detected, trigger or request or signature to be removed from the blacklist. Triggering or requesting or directing the execution of step 290. Trigger the inclusion of a signature in the training dataset used to train the machine learning process. The signature can be added along with its state (faulty or not) or can be added based on its state. Trigger the removal of a signature from the training dataset used to train the machine learning process. Signatures can be removed based on their state (faulty or not). Triggering the inclusion of a signature in a test dataset used to test the machine learning process. Signatures can be added along with or based on their state (faulty or not). Trigger the removal of a signature from the test dataset used to test the machine learning process. Signatures can be removed based on their state (faulty or not). Trigger or request or instruct another detector to start serving another detector to be assigned to generate a signature. Triggering, requesting or instructing a detector to evaluate a signature-generating detector. · Triggering or requesting or instructing to evaluate a sensing unit that generates a processed sensory information unit to provide a signature. · Triggering or requesting or instructing to reconfigure the sensing units that generate the sensed information units that are processed to provide the signature. Triggering or requesting or instructing to determine a scene that should be at least partially represented by a signature. · Trigger or request or instruct the sensing unit to evaluate the compatibility of the sensing scene. Triggering or requesting or instructing a signature generator to evaluate its compatibility to generate a signature associated with the scene. Triggering or requesting or instructing another detector to begin providing the detector that is assigned to generate a signature associated with the scene. · Monitor the results of multiple iterations of steps 210-270 and send feedback to a signature generator that generates a signature based on that monitoring. Mark the signature based on the result of step 270.

[0071] According to an embodiment, the method 200 includes a step 290 of automatically adjusting one or more parameters of the method 200 .

[0072] According to an embodiment, step 290 is automatically adjusted based on a defined period and / or the events and / or results of executing steps 210-280.

[0073] For example, step 290 may be triggered if a change in the accuracy of the signature produced by the detector is discovered.

[0074] For example, if the rate of fault signatures generated by a detector decreases over time, the detector can be adjusted under more reliable assumptions. Step 290 may include increasing the threshold, running method 200 less frequently, calculating updated false positives and updated true negatives (for the signatures evaluated by method 200), and updating the trade-off between detecting false positives and true positives.

[0075] The precision change required to trigger step 290 can be defined in various ways (e.g., having a rule defining a minimum amount of result that favors triggering step 270), but for other examples, the change may be higher than the minimum, e.g., a change of 0.001 may not favor triggering step 290. Hysteresis can be applied to reduce frequent changes in any parameter.

[0076] According to an embodiment, step 292 includes automatically determining the value of the threshold based on a dynamically configured tradeoff between false positive detection and true negative detection.

[0077] According to an embodiment, step 294 includes automatically determining the value of the threshold based on a dynamically determined accuracy metric of the signature generator that generates the signature.

[0078] According to an embodiment, step 296 includes dynamically determining a plurality of first signatures among the first top matching signatures.

[0079] According to an embodiment, step 298 includes dynamically adjusting a signature generator that generates the signature based on the results of step 280 .

[0080] In the foregoing detailed description, many specific details have been set forth. These are necessary to understand the present invention. However, it should be understood by those skilled in the art that the present invention may be practiced without these specific details. Also, in other instances, well-known methods, steps, and components have not been described in detail so as not to obscure the description of the present invention.

[0081] While certain subject matter is particularly claimed and distinctly sought in the concluding portion of this specification, the structure and method of operation of the invention, together with its objects, features, and advantages, can best be understood by reference to the following detailed description when read in conjunction with the drawings.

[0082] It should be understood that for simplicity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some elements may be exaggerated relative to other elements for clarity. Also, where necessary, figure numbers may be repeated among the figures to indicate corresponding or analogous elements.

[0083] Because the illustrated embodiments of the present invention can be implemented largely using electronic components and circuits known to those skilled in the art, further details will not be described in order to understand the basic concepts of the present invention and to avoid confusing or distracting the teachings of the present invention.

[0084] References herein to a method should be modified as necessary to apply to an apparatus or system on which it can be performed and / or to a non-transitory computer-readable medium storing instructions for carrying out the same.

[0085] References herein to a system or apparatus should be modified as appropriate to apply to methods executable thereby and / or to a non-transitory computer-readable medium storing instructions executable thereby.

[0086] References herein to non-transitory computer readable media should be made with the necessary modifications to apply to an apparatus or system capable of executing instructions stored on the non-transitory computer readable media and / or may be made with the necessary modifications to apply to a method of executing the instructions.

[0087] Any combination of any modules or units described in any figure, any part of the specification, and / or any claim may be provided.

[0088] Any one of the sensing unit, the narrow AI agent, and the driving intention determination unit may be implemented in hardware and / or code, instructions, and / or commands stored on a non-transitory computer-readable medium, and may be included within the vehicle, external to the vehicle, in a mobile device, or in a server.

[0089] The vehicle may be a vehicle such as a ground transport vehicle, an air vehicle, or a water vehicle.

[0090] This specification and / or drawings may refer to sensory information units (SIUs). SIUs may be images, media units, etc. References to media units may be modified as necessary to apply to any type of natural signal, such as, but not limited to, naturally occurring signals, signals representing human behavior, signals representing stock market related activity, medical signals, financial sequences, geodetic signals, geophysical, chemical, molecular, textual and digital signals, time series, etc. Sensory information may be of any type and may be sensed by any type of sensor, such as, for example, a visual-optical camera, an audio sensor, a sensor capable of sensing infrared, radar imaging, ultrasound, electro-optical, radiography, LIDAR (light detection and ranging), etc. Sensing may include generating samples (e.g., pixels, audio signals) representing a signal transmitted or otherwise arriving at a sensor. SIUs may be any arrangement of sensory information and may be of any size and / or format, such as, for example, an image, one or more images, an audio packet, a sensory information block, etc.

[0091] References to SIUs must be modified, if necessary, to apply to the processed SIU. A processed SIU can be generated through previously processed SIUs, etc. Processing can include any operations, such as filtering, noise reduction, SIU manipulation, padding, etc.

[0092] Citations for clusters must be modified as necessary to apply to the cluster structure. A concept structure may contain one or more clusters. Each cluster may contain a signature and associated metadata.

[0093] Obtaining content may include receiving the content, generating the content, participating in processing the content, processing only a portion of the content, and / or receiving only another portion of the content. Examples of content may include one or more signatures, SIUs, etc.

[0094] Acquiring the content may be performed with or without performing object detection.

[0095] The present specification and / or drawings may refer to a processor. A processor may be a processing circuit. The processing circuit may be implemented as a central processing unit (CPU) and / or one or more other integrated circuits, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a custom integrated circuit, or the like, or a combination thereof.

[0096] Any combination of any steps of any method illustrated in this specification and / or in the drawings may be provided.

[0097] Any combination of any subject matter of any claim may be provided.

[0098] Any combination of the systems, units, components, processors and sensors shown in this specification and / or drawings may be provided.

[0099] References to objects may also be applied to patterns, so references to object detection must be modified as necessary to apply to pattern detection.

[0100] A situation may simply be a particular location / property combination in time. A scene may be a series of events that follow logically within a causal frame of reference. References to scenes must be modified as necessary to apply to the situation.

[0101] The sensory information unit may be sensed by one or more sensors of one or more types, and the one or more sensors may belong to the same device or system, or may belong to different devices of the system.

[0102] A fault signature is a signature that, once used, may introduce errors associated with object detection. According to an embodiment, a fault signature is an ambiguous signature that, when used for object detection, results in inconsistent detection of an object. According to an embodiment, a fault signature, when used for object detection, results in at least one of the following: (i) a false negative detection, or (ii) a false positive detection. References to ambiguous signatures must be modified as needed to apply to any other fault signatures. References to false positive signatures must be modified as needed to apply to any other fault signatures. References to false negative signatures must be modified as needed to apply to any other fault signatures.

[0103] In the foregoing specification, the invention has been described with reference to specific examples of embodiments thereof. It will be apparent, however, that various changes and modifications can be made therein without departing from the broader spirit and scope of the invention as set forth in the appended claims.

[0104] Additionally, the terms "front," "rear," "top," "bottom," "upper," "lower," etc. (if any) used in the specification and claims are used for descriptive purposes only and do not necessarily describe permanent relative positions. It should be understood that terms so used are interchangeable where appropriate, and thus, embodiments of the invention described herein may be configured to operate in, for example, different orientations than those illustrated herein.

[0105] Also, the terms "assert," "set," and "negate" (or "deassert" or "clear") are used herein to cause a signal, status bit, or similar device to assume its logically true or logically false state. If the logically true state is a logic level 1, then the logically false state is a logic level 0. If the logically true state is a logic level 0, then the logically false state is a logic level 1.

[0106] Those skilled in the art should recognize that the boundaries between logic blocks are merely illustrative, and that alternative embodiments may combine logic blocks or circuit elements, or impose alternative functional decompositions on the various logic blocks or circuit elements. Thus, it should be understood that the systems described herein are merely exemplary, and that in practice many other systems may be implemented that achieve the same functionality.

[0107] Any arrangement of parts to achieve the same functionality is effectively "associated" to thereby achieve the desired functionality. Thus, any two parts that combine to achieve a particular function herein are considered to be "associated" with each other to thereby achieve the desired functionality, regardless of system or intermediate parts. Similarly, any two parts so associated can be considered to be "operably connected" or "operably coupled" to each other to achieve the desired functionality.

[0108] Those skilled in the art will also recognize that boundaries between operations described above are merely illustrative. Multiple operations may be combined into a single operation, a single operation may be distributed across additional operations, and operations may be performed in at least partial overlapping fashion. Furthermore, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be altered in various other embodiments.

[0109] Similarly, for example, in one embodiment, the illustrated examples may be implemented as circuits on a single integrated circuit or within the same device, or the examples may be implemented as any number of separate integrated circuits or separate devices connected together in any suitable manner.

[0110] However, other modifications, variations, and substitutions are possible. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

[0111] Any drawing numbers placed in parentheses in the claims should not be construed as limiting the claims. The term "comprising" does not exclude the presence of components or steps other than those recited in the claim. Also, as used herein, the terms "a" and "one" define one or more. Furthermore, the use of the introductory phrases "at least one" and "one or more" in a claim, when another claim element is introduced by the indefinite article "a" or "one," should not be construed as limiting a particular claim containing that claim element to an invention containing only one of that claim element. The same applies when the same claim contains the introductory phrases "a" or "at least one" and the indefinite article "a" or "one." The same applies to the use of definite articles. Unless otherwise specified, terms such as "first" and "second" are used to arbitrarily distinguish between the elements they describe. Thus, these terms do not necessarily indicate a chronological or other priority of such elements. The fact that certain measures are recited in mutually different claims does not indicate that these measures cannot be used to advantage.

[0112] While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes, and equivalents may occur to those skilled in the art. It is therefore intended that the appended claims be interpreted as covering all such modifications and changes that fall within the true spirit of the invention.

[0113] It should be understood that various features of embodiments of the present disclosure that are, for clarity, described in the context of a single embodiment, may also be provided in combination in a single embodiment, and conversely, various features of embodiments of the present disclosure that are, for brevity, described in the context of a single embodiment, may also be provided individually or in any suitable subcombination.

[0114] Those skilled in the art will appreciate that the embodiments of the present invention are not limited to those specifically set forth above. Rather, the scope of the disclosed embodiments is limited by the appended claims and their equivalents. Further examples are given below. [Example]

[0115] A method for automatically identifying fault signatures for an autonomous driving application includes receiving a signature by a processing circuit; matching the signature to a set of unlabeled and randomly obtained first signatures; identifying a first top matching signature based on the matching; matching the signature to a set of unlabeled second signatures that correctly or incorrectly indicate detection of a reference element; identifying a second top matching signature based on the matching between the signature and the set of second signatures; determining an overlap between the first top matching signature and the second top matching signature; and determining whether the signature is faulty based on the overlap. [Example]

[0116] 2. The method of example 1, further comprising comparing the overlap to a threshold. [Example]

[0117] The method of any one of Examples 1 to 2, further comprising determining that the signature is faulty if the overlap is below the threshold. [Example]

[0118] 4. The method of any one of Examples 1 to 3, further comprising determining that the signature is not faulty if the overlap is greater than the threshold. [Example]

[0119] The method of any one of Examples 1 to 4, further comprising automatically determining the value of the threshold based on a dynamically configured trade-off between false positive detection and true negative detection. [Example]

[0120] The method of any one of Examples 1 to 5, further comprising automatically determining the value of the threshold based on a dynamically determined accuracy metric of a signature generator that generates the signature. [Example]

[0121] 7. The method of any one of Examples 1 to 6, further comprising dynamically determining a plurality of the first signatures in the first top matching signatures. [Example]

[0122] The method according to any one of the first to seventh embodiments, further comprising dynamically adjusting a signature generator that generates the signature based on the result of determining whether the signature is faulty. [Example]

[0123] The method of any one of Examples 1 to 8 further includes monitoring results of multiple iterations of receiving the signature, matching the signature to the first signature, identifying the first top matching signature, matching the signature to the second signature, identifying the second top matching signature, determining the overlap, and determining whether the signature is faulty, and sending feedback to a signature generator that generates the signature based on the monitoring. [Example]

[0124] The method according to any one of the first to ninth embodiments, further comprising marking the signature based on a determination result of whether the signature is faulty. [Example]

[0125] A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations for automatically identifying fault signatures for an autonomous driving application, the operations including receiving a signature by a processing circuit, matching the signature to a set of unmarked and randomly acquired first signatures, identifying a first top matching signature based on the matching, matching the signature to a set of unmarked and randomly acquired second signatures that correctly or incorrectly indicate detection of a reference element, identifying a second top matching signature based on matching the signature with the set of second signatures, determining an overlap between the first top matching signature and the second top matching signature, and determining whether the signature is faulty based on the overlap. [Example]

[0126] 12. The non-transitory computer-readable medium of Example 11, wherein the operation further comprises comparing the overlap to a threshold. [Example]

[0127] In the non-transitory computer-readable medium of Example 11 or 12, the operations further include determining that the signature is faulty if the overlap is below the threshold. [Example]

[0128] In any one of Examples 11 to 13, the operation further includes determining that the signature is not faulty if the overlap is higher than the threshold. [Example]

[0129] In the non-transitory computer-readable medium of any one of Examples 11 to 14, the operation further includes automatically determining the value of the threshold based on a dynamically set trade-off between false positive detection and true negative detection. [Example]

[0130] In the non-transitory computer-readable medium of any one of Examples 11 to 15, the operation further includes automatically determining the value of the threshold based on a dynamically determined accuracy metric of a signature generator that generates the signature. [Example]

[0131] 17. The non-transitory computer-readable medium of any one of Examples 11 to 16, wherein the operation further includes dynamically determining a plurality of the first signatures in the first top matching signatures. [Example]

[0132] In the non-transitory computer-readable medium described in any one of Examples 11 to 17, the operation further includes dynamically adjusting a signature generator that generates the signature based on the result of determining whether the signature is faulty. [Example]

[0133] In the non-transitory computer-readable medium of any one of Examples 11 to 18, the operations further include monitoring results of multiple iterations of receiving the signature, matching the signature to the first signature, identifying the first top matching signature, matching the signature to the second signature, identifying the second top matching signature, determining the overlap, and determining whether the signature is faulty, and sending feedback to a signature generator that generates the signature based on the monitoring. [Example]

[0134] In the non-transitory computer-readable medium of any one of Examples 11 to 19, the operation further includes marking the signature based on a determination result of whether the signature is faulty.

Claims

1. 1. A method for automatically identifying fault signatures for an autonomous driving application, comprising: receiving a signature by a processing circuit; matching the signature with a set of unmarked and randomly obtained first signatures; identifying a first top matching signature based on the matching; matching said signature with a set of second signatures that are unmarked and that correctly or incorrectly indicate the detection of a reference element; identifying a second top matching signature based on matching the signature with the set of second signatures; determining an overlap between the first top matching signature and the second top matching signature; determining whether the signature is faulty based on the overlap; A method characterized by:

2. further comprising comparing the overlap to a threshold.

2. The method of claim 1 .

3. further comprising determining that the signature is faulty if the overlap is below the threshold.

3. The method of claim 2.

4. determining that the signature is not faulty if the overlap is greater than the threshold.

3. The method of claim 2.

5. automatically determining the value of the threshold based on a dynamically configured trade-off between false positive detection and true negative detection.

3. The method of claim 2.

6. automatically determining the value of the threshold based on a dynamically determined accuracy metric of a signature generator that generates the signature.

3. The method of claim 2.

7. and dynamically determining a plurality of the first signatures among the first top matching signatures.

2. The method of claim 1 .

8. and dynamically adjusting a signature generator that generates the signature based on the determination of whether the signature is faulty, or marking the signature based on the determination of whether the signature is faulty.

2. The method of claim 1 .

9. monitoring results of multiple iterations of receiving the signature, matching the signature to the first signature, identifying the first top matching signature, matching the signature to the second signature, identifying the second top matching signature, determining the overlap, and determining whether the signature is faulty, and sending feedback to a signature generator that generates the signature based on the monitoring.

2. The method of claim 1 .

10. A non-transitory computer-readable medium, comprising:

10. Instructions stored therein that, when executed by at least one processor, cause the at least one processor to perform the method of any one of claims 1 to 9.

1. A non-transitory computer-readable medium comprising:

Images