Real-time management of detected problems

Abstract

To provide a method for real-time management of detected issues and a non-transitory computer-readable medium.SOLUTION: The method includes the steps in which: a classification unit with a neural network produces classification decisions for sensed information obtained in a vehicle environment; one or more computing devices with an automatic labeling function generate automated ground truth labeling for the sensed information; the one or more computing devices detect issues with the classification decisions based on performance metrics related to the automated ground truth labeling; and, in response to the detection, a vehicle-related computer device resolves the issues detected during driving in the vehicle environment by using at least the classification decisions or signatures generated in connection with the detected issues.SELECTED DRAWING: Figure 6

Description

Background Art

[0001] In a vehicle, a neural network is used for various purposes including classifying items sensed by vehicle-related sensors.

[0002] Even after being trained extensively, a neural network may output an incorrect classification decision.

[0003] An incorrect classification decision can be corrected by retraining or by correcting the incorrect classification decision in another way. Retraining takes time and requires vehicle manufacturers to perform a large number of costly software updates.

[0004] There is an increasing need to provide a more efficient method for resolving incorrect classification decisions.

Summary of the Invention

[0005] In this application, a method, a system, and a non - transient computer - readable medium are disclosed.

Brief Description of the Drawings

[0006] From the following detailed description in conjunction with the drawings, embodiments of the present disclosure can be more fully understood and grasped. In the figures, [Figure 1] Examples of a vehicle and a unit. [Figure 2] Examples of a system. [Figure 3] Examples of a unit. <​​​​​​​​​​​​​​​This is an example of a method. [Modes for carrying out the invention]

[0007] Different drawings illustrate examples of units and / or software and / or information items and / or steps and / or components. These examples are provided for the sake of concise interpretation. At least one of the units and / or software and / or information items and / or steps and / or components is optional or mandatory.

[0008] This invention provides a method, system, and computer-readable medium for maintaining the state of a neural network by automatically and systematically detecting and correcting erroneous classification decisions, without requiring the neural network to be retrained. The state of the neural network is defined as the weights associated with the neurons in the network and the connectivity between neurons.

[0009] Figure 1 (Part A) shows an example of an operation-related unit 82, an analysis unit 84, and a repair unit 86, which communicate with each other to solve problems related to classification decisions generated by the neural network of the operation-related unit 82.

[0010] The Driving Unit (DRU) 82 includes a first number (N2) of processing circuits 82(1) to 82(N2) and a DRU memory / storage unit 82a, the DRU memory / storage unit being configured to store software (or any other form of instructions and / or code) and / or information and / or metadata necessary for performing driving-related functions (e.g., object detection, scene detection, classification, etc.).

[0011] The analysis unit 84 has an automatic labeling function and includes a second number (N4) of processing circuits 84(1) to 84(N4) and an analysis memory / storage unit 85a, the analysis memory / storage unit being configured to store software (or any other form of instructions and / or code) and / or information and / or metadata necessary for performing the analysis.

[0012] The repair unit 86 is configured to repair various problems without altering the state of the neural network and includes a third number (N6) of processing circuits 86(1) to 86(N6) and a repair memory / storage unit 86a, the repair memory / storage unit being configured to store software (or any other form of instructions and / or code) and / or information and / or metadata necessary for repairing the problem.

[0013] Figure 1 (part B) shows that the analysis unit 84o is located outside the vehicle 100 but communicates with the vehicle. The vehicle 101 includes a DRU 82 and a repair unit 86. The vehicle may further include an advanced driver assistance system (ADAS) control unit 81 and an autonomous driving (AD) control unit 82.

[0014] Figure 1 (part C) shows that the analysis unit 84i, DRU 82, and repair unit 86 are all located inside the vehicle 101.

[0015] According to the embodiment, the analysis unit 84 (part B) in Figure 1 is configured to be able to allocate more computing and memory resources (e.g., twice or ten times or more) to analyze the sensing information unit compared to the operation-related unit. This allows the analysis unit to be more reliable than the operation-related unit.

[0016] For tasks such as classification, for example, the analysis unit may include far more processing circuits and / or far stronger processing circuits than the driving unit. Additionally or alternatively, the analysis unit may operate non-real-time or at least not at the acquisition rate of the sensing information unit by being allocated more time (e.g., seconds, days, minutes, hours, or days) to process the sensing information unit.

[0017] According to an embodiment, the in-vehicle analysis unit may include fewer resources than the out-of-vehicle analysis unit and / or may process only selected sensing information units (instead of processing each received sensing information unit) and / or may perform a simpler process than the process performed by the out-of-vehicle analysis unit. According to an embodiment, the analysis unit (particularly the in-vehicle analysis unit) may be configured to select which sensing information unit to process and / or how to process the sensing information unit based on one or more factors (e.g., relative speed between the vehicle and other road users or the road itself, dangers related to the vehicle environment, etc.) (e.g., select a process from various options, and these options show different trade-offs between the associated accuracy and / or latency).

[0018] FIG. 1 (Part D) shows a vehicle 102 including a driving-related unit 82, an analysis unit 84, a repair unit 86, and a vehicle computer 421.

[0019] The ADAS control unit 81 is configured to control ADAS operations.

[0020] The autonomous driving control unit 82 is configured to control the autonomous driving of the autonomous vehicle.

[0021] The vehicle computer 421 is configured to control the operation of the vehicle and particularly to control the engine, transmission, and any other vehicle system or component.

[0022] The vehicle computer 421 can communicate with an engine control module, a transmission control module, a powertrain control module, and the like.

[0023] FIG. 2 shows an example of a computerized system 400, which includes a communication system 430, one or more memories and / or storage units 420, and a processing system 424 including a processor 426. The computerized system may be a server, a laptop computer, a desktop computer, or any other computer, and may include or communicate with a sensing unit and / or a controller.

[0024] According to an embodiment, the computerized system 400 communicates with a network 432 and one or more other remote computerized systems 434 that communicate with the network 432. FIG. 1 (Part B) shows an example of a remote computerized system, where the analysis unit is located outside the vehicle.

[0025] According to an embodiment, the communication system 430 is configured to enable communication between any one of the one or more memories and / or storage units 420 and / or the sensing system 410 and / or an additional unit and / or the network 432 (which communicates with the remote computerized system).

[0026] The memory and / or storage unit 420 is shown as storage software. Any reference to software should, where appropriate and as modified, be suitable for code and / or firmware and / or instructions and / or commands, etc.

[0027] The processor 426 includes multiple processing units 426(1) to 426(J), where J is an integer greater than 1. Any reference to a single unit or item should, when modified as appropriate, be suitable for multiple units or items. For example, any reference to a processor should, when modified as appropriate, be suitable for multiple processors, and any reference to a communication system 430 should, when modified as appropriate, be suitable for multiple communication systems.

[0028] According to the embodiment, one or more memory and / or storage units 420 include one or more memory units, and each memory unit may include one or more storage bodies.

[0029] According to the embodiment, one or more memory and / or storage units 420 include volatile memory and / or non-volatile memory. One or more memory and / or storage units 420 may also be random access memory (RAM) and / or read-only memory (ROM).

[0030] According to the embodiment, the non-volatile memory unit is a high-capacity storage device that can provide non-volatile storage of computer code, computer-readable instructions, data structures, program modules, and other data for a vehicle processor or any other unit. For example, but not limited to, high-capacity storage devices may be hard disks, portable magnetic disks, portable optical disks, cassette magnetic tapes or other magnetic storage devices, flash memory cards, CD-ROMs, digital versatile disks (DVDs) or other optical storage devices, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and the like.

[0031] Any content may be stored in any part or in any type of memory and / or storage unit.

[0032] According to the embodiment, at least one memory unit stores at least one database, for example, any database known in the art, such as DB2®, Microsoft® Access, Microsoft® SQL Server, Oracle®, MySQL, PostgreSQL, etc.

[0033] The memory and / or storage unit 420 is configured to store firmware and / or software, one or more operating systems, data and metadata necessary for performing any of the methods described in this application.

[0034] The memory and / or storage unit 420 is shown as storage software. Any references to the software should be adapted to code and / or firmware and / or instructions and / or commands, etc., as they may be modified as appropriate.

[0035] Various units and / or components communicate with each other using arbitrary communication elements and / or protocols. An example of a communication system is shown in 430. Other communication elements may be provided.

[0036] The communication system 430 can communicate with the bus 436. The bus represents one or more of several possible types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any one of various bus architectures. For example, such architectures may include the Industry Standard Architecture (ISA) bus, Microchannel Architecture (MCA) bus, Expansion ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, Accelerated Graphics Port (AGP) bus and Peripheral Component Interconnect (PCI), PCI-Express bus, Personal Computer Memory Card International Association (PCMCIA), Universal Serial Bus (USB), etc. The bus and all buses specified herein may be implemented by each subsystem of the wired or wireless network connectivity subsystem.

[0037] Network 432, located outside the vehicle, is used for communication between the vehicle and at least one remote computing system. For example, the remote computing system may be a personal computer, laptop computer, portable computer, server, router, network computer, peer-to-peer device, or other common network node. The logical connection between the processor and any one of the remote computing systems may be implemented via a local area network (LAN) and a general-purpose wide area network (WAN). Such network connections may be implemented via network adapters (which may belong to communication system 430) that can be implemented in wired and wireless environments. Such network environments are common in larger networks such as offices, enterprise-scale computer networks, intranets, and the internet.

[0038] As a step to note, at least some of the content, which is indicated to be stored in one or more memory / storage units 420, may be stored outside the vehicle. As a further step to note, the processor may evaluate the signatures generated by multiple detectors.

[0039] U.S. Patent Application No. 18 / 527,701 provides an example of generating a signature and / or cropped image, which is incorporated herein by reference.

[0040] According to the embodiment, the memory and / or storage unit 220 stores at least one of the operating system 494, information 491, metadata 492, and software 493.

[0041] Using software, the processing system is configured to perform one or more of the methods 100, 200, 500, or 600.

[0042] The vehicle 400 further includes a sensing system 410 and a control unit 425.

[0043] The control unit 425 can cooperate with an advanced driver assistance system (ADAS) control unit, an autonomous driving control unit 422, and / or control or communicate with other vehicle components (including a vehicle computer).

[0044] The sensing system 410 may include an optical device, a group of sensing elements, a readout circuit, and an image signal processor. Following the optical device is a group of sensing elements, for example, a row of sensing elements or an array of sensing elements forming the group of sensing elements. Following the group of sensing elements is a readout circuit, which reads the detection signal generated by the group of sensing elements. The image signal processor is configured to perform initial processing of the detection signal, for example, by improving the quality of the detection information and performing noise reduction. The sensing system 410 is configured to output one or more sensing information units (SIUs).

[0045] The control unit 425 is configured to control the operation of the sensing system 410 and / or one or more memory and / or storage units 420 and / or one or more additional units (other than the controller).

[0046] According to the embodiment, one or more memory and / or storage units 420 include one or more memory units, and each memory unit may include one or more storage bodies.

[0047] According to the embodiment, one or more memory and / or storage units 420 include volatile memory and / or non-volatile memory. One or more memory and / or storage units 420 may also be random access memory (RAM) and / or read-only memory (ROM).

[0048] According to the embodiment, the non-volatile memory unit is a high-capacity storage device that can provide non-volatile storage of computer code, computer-readable instructions, data structures, program modules and other data for the vehicle's processor or any other unit. For example, but not limited to, the high-capacity storage device may be a hard disk, a portable magnetic disk, a portable optical disk, a cassette magnetic tape or other magnetic storage device, a flash memory card, a CD-ROM, a digital versatile disk (DVD) or other optical storage device, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and the like.

[0049] For example, but not limited to, computer-readable media may include “computer storage media” and “communication media.” “Computer storage media” includes volatile and non-volatile, movable and immovable media for storing information (e.g., computer-readable instructions, data structures, program modules, or other data) implemented by any method or technique. Illustrative computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies; CD-ROM, digital versatile disk (DVD), or other optical storage devices; magnetic tape cassettes, magnetic tapes, magnetic disk storage devices, or other magnetic storage devices; or any other media that can be used to store desired information and are computer-accessible.

[0050] Any content may be stored in any part of memory and / or storage units or in any type of memory and / or storage units.

[0051] According to the embodiment, at least one memory unit stores at least one database, for example, any database known in the art, such as DB2®, Microsoft® Access, Microsoft® SQL Server, Oracle®, MySQL, PostgreSQL, etc.

[0052] Various units and / or components communicate with each other using arbitrary communication elements and / or protocols. An example of a communication system is shown in 430. Other communication elements may be provided.

[0053] Figure 3 shows an example of an operation-related unit 74, an analysis unit 76, and a repair unit 78 that communicate with each other.

[0054] The driving-related unit 74 is configured to receive a sensing information unit 70 that may be unlabeled or partially labeled (for example, in a sensing information unit, 10%, 20%, 30%, or 40% of the items may be labeled) and to supply the sensing information unit 70 to a first subunit 72 which includes a neural network (NN) 73. The NN generates an erroneous NN signature 43. The NN signature is then supplied to a post-NN classification unit 74, which is configured to generate a classification decision 44 based on a comparison of the NN signature 74 with reference clusters associated with defined road elements, for example, objects related to the vehicle environment in an environment-related scene.

[0055] The classification decision 44 and the sensing information unit are supplied to an analysis unit 76 having an automatic labeling function. The analysis unit 76 is configured to output automated ground true labeling 45, which indicates one or more road elements captured by the sensing information unit. According to the embodiment, the analysis unit 76 is further configured to generate a KPI report 46. According to the embodiment, the analysis unit 76 is further configured to detect problems with the classification decision based on performance indicators related to the automated ground true labeling. According to the embodiment, problem detection is performed by another unit, not by the analysis unit 76.

[0056] According to the embodiment, in response to the detection, the system requests the remediation unit 78 to resolve the detected problem using at least a signature generated in relation to the classification decision or the detected problem. The signature may be the NN signature 43, or a new signature 49 generated by the remediation unit 78. For example, when the NN 73 has not been retrained, the NN signature related to the problem may be marked as erroneous and added, for example, to the erroneous signature blacklist 41. When an erroneous NN signature is received, the post-NN classification unit 74 discovers the step to which the NN signature belongs in the blacklist 41 and refrains from transmitting a classification decision based on the erroneous NN signature. According to the embodiment, the new signature 49 may be added to the correct signature whitelist 42 and subsequently linked to the erroneous NN signature. As a result, once the post-NN classification unit 74 receives an incorrect NN signature, the classification decision will be based on a new signature 49, which may, for example, be compared to a reference cluster.

[0057] Figure 4 shows an example of metadata and / or software stored in at least one of the memory / storage units, DRU memory / storage unit 82a, analysis memory / storage unit 84a, and repair memory / storage unit 86a.

[0058] Examples include operation-related software 30 and operation-related metadata 40 to facilitate the operation of DRUs (72, 82), analysis software 32 and analysis metadata 42 to facilitate the operation of analysis units (74, 84), and repair software 34 and repair metadata 44 to facilitate the operation of repair units (76, 86).

[0059] An example of the driving software 30 includes cropping software 30-1 for generating cropped images, neural network software 30-2 (used for implementing NN 93), and classification software 30-3 (used by the post-NN classification unit 74).

[0060] An example of operational metadata 40 includes NN weights 40-1, reference clusters 40-2, a whitelist 41, and a blacklist 42.

[0061] Examples of analysis software 32 include analysis software 32-1, automatic labeling software 32-2, lane detection software 32-3, object detection software 32-4, scene detection software 32-5, high-level decision software 32-6, KPI report generator software 32-7, and problem detection software 32-8.

[0062] An example of analysis metadata 42 includes decision rules 43-1, problem detection parameters 43-2, and performance metrics 43-3.

[0063] An example of the repair software 34 includes signal generation software 34-1 for generating a new signature and error signature labeling software 34-2 for labeling an incorrect NN signature as an error signature.

[0064] Figure 5 shows an example of the analysis unit 10a.

[0065] The analysis unit 10 includes an object detection unit 10, a lane detection unit 20, a scene classification unit 50, and a high-level decision unit 79, which includes a KPI generator that determines, for example, which KPIs / errors to send to the repair unit.

[0066] A crop2vec is configured to generate a cropped image and a vector representing the content of the cropped image. A crop2vec may include one or more neural networks and a converter, which is used to convert the characterized neural networks into vectors, such as embedded or embedded signatures. An example of a crop2vec is shown in U.S. Patent Application No. 18 / 527,701, which is incorporated herein by reference.

[0067] The object detection (OD) portion includes the following: A. The OD crop generator 11 receives a high-resolution image and generates small crops, one of which focuses on near and medium distances, and the other on medium and far distances. The crops are defined statically based on object density analysis. B. The first OD stage 12 is a high-level computer object detector model, such as CoDETR, which is configured to operate on two types of crops to produce bounding boxes (bboxes) for all ranges of objects. C. CoDETR is introduced in the following document: "DETRs with Collaborative Hybrid Assignments Training," Zhuofan Zong, Guanglu Song, and Yu Lin, arXiv:2211.12860. The D.OD bounding box (bbox) integrator 13 is configured to handle overlaps of objects detected in two crops and to mark bboxes that should be discarded (bboxes that have a high degree of overlap between crops but have relatively low confidence). E. OD Crop Generator - The second stage 14 is configured to dynamically crop a box from the image based on the bbox predicted in the first stage. F. Four-wheel (4W) occlusion and separator 15-1 is configured to predict how much an object will be occluded by predicting several useful attributes (3D separator, left tag, right tag, occlusion score) used for mapping to 3D position and orientation. It may be configured to run the model (e.g., a DinoV2 model with a specific head) on a four-wheel type crop. Dinov2 is introduced in the following article: "Learning Robust Visual Features without Supervision (DINOv2)", Oquab er et al., arXiv:2304.07193. G. 4W crop2vec: Proximal 15-2 is configured to run a model (e.g., a DinoV2 model with a specific head and training program) on a four-wheeled box higher than a certain height (corresponding to a nearly nearby object), with the aim of distinguishing between four-wheeled subtypes (cars, freight cars, trucks, route buses, trains) and identifying and labeling false alarms and noise predictions. H. 4W crop2vec: Distal 15-3 is configured to run a model (e.g., a DinoV2 model with a specific head and training program) on a four-wheeled box lower than a certain height (corresponding to a nearly distant object), with the aim of distinguishing between four-wheeled subtypes (cars / freight cars / trucks / buses) and identifying and labeling false alarms / noise predictions. I. Pedestrian (Ped)crop2vec: Proximal 15-4 is configured to run a model (e.g., a DinoV2 model with a specific head and training program) on a pedestrian box higher than a certain height (corresponding to a nearly nearby object), with the aim of distinguishing pedestrian subtypes (pedestrian / lidar) and identifying and marking false alarm / noise predictions. J. Ped crop2vec: Distal 15-6 is configured to run a model (e.g., a DinoV2 model with a specific head and training program) on a pedestrian box lower than a certain height (corresponding to a nearly distant object), with the aim of distinguishing pedestrian subtypes (pedestrian / lidar) and identifying and labeling false alarm / noise predictions. The K.OD marker 16 is configured to collect all predictions from various models and to make a final decision on whether a bbox should be discarded, ignored during the verification period, or verified as a true object.

[0068] For example, any number and / or types of modules relating to objects other than pedestrians and vehicles, such as modules 15-1 to 15-5, may be provided.

[0069] The lane detection section is configured to filter lane segments using crop2vec. The lane segment determiner adds the ignored lanes to the remaining lanes. The lane integrator integrates only the lanes that have not been ignored. The position is configured to perform the assignment of the global lane indicator.

[0070] The lane detection (LD) section includes the following: A. LD Crop Generator - The first stage 21 receives a high-resolution image and generates multiple (e.g., 16) smaller crops, configured to focus on various parts of the image, such as different areas on the road, the front lane and side lane, or lanes on a curved road. The crops are statically defined based on density analysis. B. The first LD stage 22 may be a high-level computing object detector model such as CLRerNet, and is configured to operate on all crops to generate predicted lane segments of lanes and road boundaries (RB) in all ranges and viewing angles. CLRerNet is introduced in the following article: "Improving Confidence of Lane Detection with LaneIoU (CLRerNet)", Hiroto Honda and Yusuke Uchida, arXiv:2305.08366. C. LD Crop Generator - The second stage 23 dynamically crops a rotating bounding box from the image based on the predicted lane from the first stage. D. Lane crop2vec 24-1 is configured to run a model (e.g., a DinoV2 model with a specific head unit and training program) on all rotating boxes that package predicted lane segments, with the aim of distinguishing lane types (dashed lines, continuous dots, dots, white, yellow, etc.) and marking and removing false alarms / noise predictions. E. RB (Road Boundary) crop2vec 24-2 is configured to run a model (e.g., a DinoV2 model with a specific head unit and training program) on all rotating boxes that package predicted road boundary (RB) segment stages, with the aim of distinguishing lane types (e.g., elevated lanes, level ground, with obstacles) and marking and removing false alarm / noise predictions. F. The lane segment determiner 25 is configured to apply confidence thresholds to the remaining lanes and road boundaries. The thresholds determine which lane segments should be ignored within the evaluation period and which lane segments should be verified and communicated to the lane integrator. G. The lane integrator 27 is configured to operate only in lanes that have not been ignored by the decision-maker, and it identifies and integrates different lane segments belonging to the same entire lane or road boundary line by applying a graph-based clustering algorithm. H. Indicator assignment 28 is configured to assign global line indicators to lanes or R / B lines relative to the vehicle's position, i.e., the left and right lanes relative to position 0, with the main lane marked as L0\R0 and adjacent lanes as L1\R1, and so on. Road boundaries are marked similarly. The algorithm uses the relative positioning and orientation of lanes with respect to the camera view, but future versions will use a stronger learning algorithm.

[0071] For example, any number and / or types of modules relating to road elements other than road boundaries and lanes, such as modules 24-1 to 24-2, may be provided.

[0072] Scene classification section 50 includes the following: A. The road type classifier 51 is configured to classify the image based on the detected road type (highway, urban area, tunnel, etc.). B. The weather classifier 52 is configured to classify the image based on the detected weather (sunny, cloudy, rainy, snowy, etc.). C. The light classifier 53 is configured to classify images based on the detected lighting conditions (daytime, nighttime, dawn, twilight, dusk, etc.). D. The scene marker 54 is configured to smooth the prediction over time to ensure that discontinuities exist in the prediction, and to support marking images in situations where classification is not clearly defined (for example, some unclear transition areas between urban areas and highways) by comparing them with various thresholds.

[0073] Any number and / or types of scene classifiers, such as urban or rural environment classifiers, may be provided.

[0074] KPIs may be defined in any manner, for example, by the analysis unit itself, the vehicle user, the vehicle manufacturer, any provider, or any one of the following software or hardware components: the vehicle, vehicle components, analysis unit components, repair unit components, etc.

[0075] Non-exclusive examples of KPIs include object detection KPIs, lane detection KPIs, scene classification KPIs, repair-related KPIs, detected problem KPIs, classification decision KPIs, signature-related KPIs, classification KPIs, and neural network-related KPIs.

[0076] For example, object detection KPIs may include the number of boxes in the test set, the scores of boxes whose scores are higher than the test quality measure, the intersection of the union of measurement similarities between finite sample tests (IoU), true positive TP (may be determined based on the IoU value), false positive FP (may be determined based on the IoU value), false negative FN (may be determined based on the IoU value), precision (e.g., TP / (TP+FP)), recall (e.g., TP / (TP+FN)), and false positives per image (e.g., FP / total number of images).

[0077] The KPIs may be based on time of day, vehicle distance traveled, vehicle speed, vehicle acceleration, vehicle status, duration of sensed events, spatial relationship between vehicle and lane (e.g., distance from lane start point, distance to one or more lane boundaries, angle between vehicle movement and lane direction), spatial relationship between vehicle and road boundary (e.g., distance to road boundary start point, distance to one or more road boundaries, angle between vehicle movement and road boundary direction), lane parameters (e.g., lane type, i.e., whether passing over lane boundaries is permitted or prohibited, the geometric shape of the lane, lane color), number of images, number of signatures, distance range to vehicle (e.g., short, medium, or long distance), lane, environment, weather conditions, lighting conditions, road user type, or any combination of the above or any other parameters.

[0078] KPI values ​​may be filtered or smoothed, and may have any other features to prevent delays or changes in KPI values ​​that occur too quickly. Additionally or alternatively, lane characteristics—for example, the width of a straight road segment is kept essentially the same along the lane segment, and the lane shape is kept essentially the same until the lane shape changes, such as a curve or intersection—may be considered.

[0079] The KPI may respond to an event (e.g., vehicle movement (e.g., a vehicle event triggered by error classification)) and / or trigger vehicle movement (e.g., request or instruct the ADAS control unit and / or AD control unit to perform vehicle movement).

[0080] Figure 6 shows an example of 100 methods used for real-time management of detected problems.

[0081] According to the examples, method 100 includes at least some of steps 102, 104, 106, 108, 100, and 112.

[0082] According to the embodiment, step 102 includes generating a classification decision for sensory information acquired in the vehicle environment by a classification unit having a neural network.

[0083] According to the embodiment, step 104 includes generating automated ground true labeling for sensing information using one or more computing devices having an automatic labeling function.

[0084] According to the embodiment, step 106 includes detecting problems with classification decisions based on performance metrics related to automated ground truth labeling using one or more computing devices.

[0085] According to the examples, the problem relates to inaccuracies in classification decisions. A problem may exist if, when comparing classification decisions with automated ground truth labeling, the type differs from the type identified in automated ground truth labeling. The problem may be determined based on one or more additional factors (e.g., whether the error affects the movement of the vehicle).

[0086] According to the embodiment, step 108 includes, in response to the detection, a computer device associated with the vehicle solving the problem detected while driving the vehicle in the vehicle environment, using a signature generated in relation to at least a classification decision or the detected problem.

[0087] According to the embodiment, the neural network is in the same state during the steps of making a classification decision, detecting a problem, and solving the detected problem.

[0088] According to the embodiment, step 108 involves verifying whether the solution puts the system in a better position, for example, whether the solution (e.g., adding a new signature) introduces new problems, such as adding more errors to the classification process.

[0089] According to the embodiment, a signature is generated in relation to the classification decision, and step 108 includes labeling the signature as being related to a classification error.

[0090] According to the embodiment, step 108 includes determining that the detected problem should be ignored.

[0091] According to the embodiment, step 108 is triggered if the detected problem causes unnecessary movement of the vehicle.

[0092] According to an embodiment, method 100 further includes step 110 of generating a key performance indicator (KPI) report by one or more computing devices.

[0093] According to the embodiment, method 100 further includes step 112 of analyzing problems detected within a certain period and marking these detected problems for further downstream analysis.

[0094] According to the example, resolving a detected problem involves resolving other detected problems that are similar to the problem detected in the classification using signatures (for example, they can be mapped to the same reference cluster). Therefore, objects similar to the object that triggered the remediation also benefit from the remediation because they will be correctly classified after the remediation.

[0095] According to the examples, solving the detected problem involves solving other detected problems that are similar to the problem detected in the classification using the signature.

[0096] According to the examples, resolving the detected problem involves using signatures to resolve other classification decisions that are similar to the classification decision in terms of classification.

[0097] According to the example, the neural network of the classification unit and another neural network of one or more computing devices with an auto-labeling function are trained using the same or different training datasets.

[0098] According to one embodiment, step 108 includes generating signatures for the detected problem using another neural network, the other neural network being trained using the same training dataset as the neural network of the classification unit.

[0099] Figure 7 shows an example of method 200 used for real-time management of detected problems.

[0100] According to the examples, method 200 includes at least some of steps 204, 207, 208, 200, and 212.

[0101] According to the embodiment, step 204 includes generating automated ground true labeling for sensing information using one or more computing devices having an automatic labeling function.

[0102] According to the embodiment, step 207 includes detecting, by one or more computing devices, problems with classification detection performed on sensing information using a neural network in a predetermined state, based on performance metrics related to automated ground truth labeling.

[0103] According to the examples, the problem relates to inaccuracies in classification decisions. A problem may exist if, when comparing classification decisions with automated ground truth labeling, the type differs from the type identified in automated ground truth labeling. The problem may be determined based on one or more additional factors (e.g., whether the error affects the movement of the vehicle).

[0104] According to the embodiment, step 208 includes, in response to the detection, a computer device associated with the vehicle solving the problem detected while driving the vehicle in the vehicle environment, using at least a classification decision or a signature generated in relation to the detected problem.

[0105] According to the embodiment, the neural network is in the same state during the steps of making a classification decision, detecting a problem, and solving the detected problem.

[0106] According to the embodiment, a signature is generated in relation to the classification decision, and step 208 includes marking the signature as being related to a classification error.

[0107] According to the embodiment, step 208 includes determining that the detected problem should be ignored.

[0108] According to the embodiment, step 208 is triggered if the detected problem causes unnecessary movement of the vehicle.

[0109] According to an embodiment, method 200 further includes step 210 of generating a key performance indicator (KPI) report by one or more computing devices.

[0110] According to the embodiment, method 200 further includes step 212 of analyzing problems detected within a certain period and marking the detected problems for further downstream analysis.

[0111] According to the example, resolving a detected problem involves resolving other detected problems that are similar to the problem detected in the classification using signatures (for example, they can be mapped to the same reference cluster). Therefore, objects similar to the object that triggered the remediation also benefit from the remediation because they are correctly classified after the remediation.

[0112] According to the examples, solving the detected problem involves solving other detected problems that are similar to the problem detected in the classification using the signature.

[0113] According to the examples, resolving the detected problem involves using signatures to resolve other classification decisions that are similar to the classification decision in terms of classification.

[0114] According to the example, the neural network of the classification unit and another neural network of one or more computing devices with an auto-labeling function are trained using the same or different training datasets.

[0115] According to the embodiment, step 208 includes the step of generating a signature for the detected problem using another neural network, the other neural network being trained using the same training dataset as the neural network of the classification unit.

[0116] According to the embodiment, a method that can be performed at least partially by the in-vehicle analysis unit may involve fewer resources than the external analysis unit, and / or process only selected sensing information units (instead of processing each received sensing information unit), and / or perform simpler processing than that performed by the external analysis unit.

[0117] According to the embodiment, the analysis unit (particularly the in-vehicle analysis unit) may be configured to select which sensing information units to process (therefore providing a determined set of sensing information units), and / or how to process the sensing information units based on one or more elements, the one or more elements including, but not limited to, the following: A. The relative speed between the vehicle and other road users or the road itself (higher relative speeds may require more frequent analysis). B. Hazards related to the vehicle environment (the greater the hazard, the more frequently it may need to be analyzed). Hazards may be determined by any method, for example, monitoring the physiological responses of one or more people in the vehicle; monitoring the current driving mode of the vehicle and comparing it to a driving mode marked as hazardous; calculating the deviation between the current driving mode and the safe driving policy; and calculating the deviation between the steady-state acceleration and the average acceleration over a certain period (day, hour, month) based on the use of brakes and / or changes in deceleration acceleration. C. This is the difference between the current speed and the allowable speed. If the difference is higher or the percentage of the difference is higher, it may be necessary to analyze it more frequently. D. The vehicle driver's familiarity with the vehicle's current environment (this familiarity may be determined based on the number of times the driver has driven in that environment). This is because a higher level of familiarity allows for a lower frequency of analysis. E. This is the deviation between the current driving mode and the driver's average driving mode. F. This is the relationship between the analysis duration and the acquisition rate of sensing units. For example, if the acquisition rate of Y sensing units per second is X seconds and the analysis duration is X seconds, it is specified that one analysis is performed for every X*Y sensing units. X and Y are positive numbers, and the range of these positive numbers may be 0.1 to 100, or 2 to 50, or any other range. G. The complexity of the environment. For example, compared to a desolate rural road, a densely populated urban environment has many intersections and / or is typically inhabited by pedestrians and / or motorcyclists, and therefore needs to be analyzed more frequently. Complexity may also be based on the density of expected driving-related events (events that have a potential or actual impact on vehicle driving) and / or the history of events that have occurred in different environments.

[0118] According to the embodiment, any rule and / or model (including, but not limited to, a machine learning model) may be applied to any one or any combination of elements (A) to (G) and / or any other elements to determine at least one of the methods for applying the analysis frequency and / or analysis process. According to the embodiment, any mapping between the value of any element and the amount to which the analysis frequency is changed, such as a linear mapping, a nonlinear mapping, a step mapping, etc., may be provided. The value of the analysis frequency may be selected from a set of limited number of frequency values ​​(2, 3, 4, 5, 6, 7, 8 or more).

[0119] According to the embodiment, the method for processing one or more sensing information units may also relate to determining the processing from different processing options that indicate different processing accuracy and / or different processing options that indicate different delays and / or different processing options that indicate different balances between accuracy and time.

[0120] According to the embodiment, the different processing options may differ from each other in at least one respect, such as resolution, accuracy, size of input information, size of output information, complexity of the analysis process, amount of computing resources required or obtainable to complete the analysis, and amount of memory resources required or obtainable to complete the analysis.

[0121] Examples of controlled processes (optionally performed before the initial preliminary processing of (multiple) sensing information units) are as follows: A. Process only one or more regions of the sensing information unit, for example, only the center of the sensing information unit. The center may be 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 45%, 50%, 55%, or 60% of the sensing information unit. B. Process only a finite number of bounding boxes (1, 2, 3, etc.) that need to be processed. C. Only process objects that represent a finite percentage of the detected information (for example, objects that reach 1 / 10, 1 / 9, 1 / 8, 1 / 5, 1 / 4, 1 / 3, or 1 / 2 of the object). D. Ignore one or more sensing units for airborne and / or distant objects. E. Process one, two, or three (or any number) of the most relevant supplementary terms. F. Apply only a portion of the analysis path. For example, use only one or two units, or selected units from object detection, lane detection and scene classification, etc. Or use only some of the units from the analysis path. For example, use only one or two of modules 15-1 to 15-5, or use only one of modules 24-1 and 24-2.

[0122] Figure 8 shows an example of method 500 used for real-time management of detected problems.

[0123] According to the examples, method 500 includes at least some of steps 502, 505, 506, 508, 500, and 512.

[0124] According to the embodiment, step 502 includes generating a classification decision for sensory information acquired in the vehicle environment by a classification unit having a neural network.

[0125] According to the embodiment, step 505 includes generating automated ground true labeling for a determined set of sensing information units by one or more computing devices having an automated labeling function operating in real time during the operation of the vehicle.

[0126] According to the embodiment, step 506 includes detecting problems with classification decisions based on performance metrics related to automated ground truth labeling using one or more computing devices.

[0127] According to the examples, the problem relates to inaccuracies in classification decisions. A problem may exist if, when comparing classification decisions with automated ground truth labeling, the type differs from the type identified in automated ground truth labeling. The problem may be determined based on one or more additional factors (e.g., whether the error affects the movement of the vehicle).

[0128] According to the embodiment, step 508 includes, in response to the detection, a computer device associated with the vehicle solving the problem detected while driving the vehicle in the vehicle environment, using a signature generated in relation to at least a classification decision or the detected problem.

[0129] According to the embodiment, the neural network is in the same state during the steps of making a classification decision, detecting a problem, and solving the detected problem.

[0130] According to the embodiment, a signature is generated in relation to the classification decision, and step 508 includes marking the signature as being related to a classification error.

[0131] According to the embodiment, step 508 includes determining that the detected problem should be ignored.

[0132] According to the embodiment, step 508 is triggered if the detected problem causes unnecessary movement of the vehicle.

[0133] According to an embodiment, method 500 further includes step 510 of generating a key performance indicator (KPI) report by one or more computing devices.

[0134] According to the embodiment, method 500 further includes step 512 of analyzing problems detected within a certain period and marking these detected problems for further downstream analysis.

[0135] According to the example, resolving a detected problem involves resolving other detected problems that are similar to the problem detected in the classification using signatures (for example, they can be mapped to the same reference cluster). Therefore, objects similar to the object that triggered the remediation also benefit from the remediation because they are correctly classified after the remediation.

[0136] According to the examples, solving the detected problem involves solving other detected problems that are similar to the problem detected in the classification using the signature.

[0137] According to the examples, resolving the detected problem involves using the signature to resolve other classification decisions that are similar to the classification decision in the classification.

[0138] According to the example, the neural network of the classification unit and another neural network of one or more computing devices with an auto-labeling function are trained using the same or different training datasets.

[0139] According to one embodiment, step 508 includes generating signatures for the detected problem using another neural network, the other neural network being trained using the same training dataset as the neural network of the classification unit.

[0140] Figure 9 shows an example of method 600 used for real-time management of detected problems.

[0141] According to the embodiment, method 600 includes at least some of steps 605, 607, 606, 608, 600, and 612.

[0142] According to the embodiment, step 605 includes generating automated ground true labeling for a set of sensing information units determined by one or more computing devices having an automated labeling function operating in real time during the operation of the vehicle.

[0143] According to the embodiment, step 607 includes detecting, by one or more computing devices, a problem with classification detection performed on sensing information using a neural network in a predetermined state, based on performance metrics related to automated ground truth labeling.

[0144] According to the examples, the problem relates to inaccuracies in classification decisions. A problem may exist if, when comparing classification decisions with automated ground truth labeling, the type differs from the type identified in automated ground truth labeling. The problem may be determined based on one or more additional factors (e.g., whether the error affects the movement of the vehicle).

[0145] According to the embodiment, step 608 includes, in response to the detection, a computer device associated with the vehicle solving the problem detected while driving the vehicle in the vehicle environment, using a signature generated in relation to at least a classification decision or the detected problem.

[0146] According to the embodiment, the neural network is in the same state during the steps of making a classification decision, detecting a problem, and solving the detected problem.

[0147] According to the embodiment, a signature is generated in relation to the classification decision, and step 608 includes marking the signature as being related to a classification error.

[0148] According to the embodiment, step 608 includes determining that the detected problem should be ignored.

[0149] According to the embodiment, step 608 is triggered if the detected problem causes unnecessary movement of the vehicle.

[0150] According to an embodiment, method 600 further includes step 610 of generating a key performance indicator (KPI) report by one or more computing devices.

[0151] According to the embodiment, method 600 further includes step 612 of analyzing problems detected within a certain period and marking these detected problems for further downstream analysis.

[0152] According to the example, resolving a detected problem involves resolving other detected problems that are similar to the problem detected in the classification using signatures (for example, problems that can be mapped to the same reference cluster). Therefore, objects similar to the object that triggered the remediation also benefit from the remediation because they are correctly classified after the remediation.

[0153] According to the examples, solving the detected problem involves solving other detected problems that are similar to the problem detected in the classification using the signature.

[0154] According to the embodiment, the step of resolving the detected problem includes using the signature to resolve other classification decisions that are similar to the classification decision in terms of classification.

[0155] According to the example, the neural network of the classification unit and another neural network of one or more computing devices with an auto-labeling function are trained using the same or different training datasets.

[0156] According to one embodiment, step 608 includes generating signatures for the detected problem using another neural network, the other neural network being trained using the same training dataset as the neural network of the classification unit.

[0157] Any combination of any step of any method shown in this application is provided.

[0158] The detailed description above includes many specific details to provide a deeper understanding of the present invention. However, as those skilled in the art will understand, the present invention can be carried out even without these specific details. In other cases, well-known methods, programs, and components are not described in detail so as not to obscure the present invention.

[0159] In the conclusion section of the specification, the themes of the present invention are specifically pointed out and explicitly claimed for protection. However, when reading in conjunction with the drawings, the organization and operation method and their objectives, features and advantages of the present invention can be best understood by referring to the following specific embodiments.

[0160] To ensure clarity and conciseness, the elements depicted are not necessarily drawn to scale. For example, for clarity, the size of some elements may be enlarged relative to others. Furthermore, where appropriate, symbols may be repeated in the diagram to indicate corresponding or similar elements.

[0161] Since most of the embodiments of the present invention can be implemented using electronic components and circuits known to those skilled in the art, we will not provide any further details beyond what is deemed necessary, in order to ensure that the basic idea of ​​the present invention is understood and recognized, and that the teachings of the present invention are not obscured or dispersed.

[0162] Any references to the methods in this specification, if modified as appropriate, should be suitable for a device or system capable of performing the methods and / or for a non-temporary computer-readable medium storing instructions for performing the methods.

[0163] Any reference to a system or device in the specification should, if modified as appropriate, be suitable for a method that can be performed by the system, and / or, if modified as appropriate, be suitable for a non-temporary computer-readable medium that stores instructions that can be executed by the system.

[0164] Any reference in this specification to non-temporary computer-readable media should, as modified as appropriate, be suitable for a device or system capable of executing instructions stored on non-temporary computer-readable media, and / or, as modified as appropriate, be suitable for a method of executing such instructions.

[0165] Any combination of any one of the drawings, any part of the specification, and / or any module or unit listed in any claim may be provided.

[0166] Any one of the conversion module, active learning module, or clustering module, or any other module described herein, may be implemented in code, instructions, and / or commands stored in hardware and / or non-temporary computer-readable media, and may be contained within a vehicle, outside a vehicle, within a mobile device, within a server, etc.

[0167] The vehicle may be any type of vehicle, such as a ground transport vehicle, an air transport vehicle, or a water transport vehicle.

[0168] The specification and / or drawings may relate to images. Images are examples of sensing information. Any reference to images can be adapted as appropriate to any type of natural signal, including, but not limited to, naturally generated signals, signals indicating human behavior, signals indicating operations related to stock markets, medical signals, financial sequences, geometric signals, geophysical, chemical, molecular, text and digital signals, time sequences, etc. Any reference to media units can be adapted as appropriate to sensing information. Sensing information may be of any kind and may be perceived by any type of sensor (e.g., visible light cameras, audio sensors, perceptible infrared, radar imaging, ultrasound, electro-optics, radiography, laser radar (light detection and ranging)). Sensing may include generating samples (e.g., pixels, audio signals) that indicate signals to be transported or reach the sensor.

[0169] The specification and / or drawings may relate to a processor. The processor may also be a processing circuit. The processing circuit may be implemented as a central processing unit (CPU) and / or one or more other integrated circuits, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), custom integrated circuits, or a combination of such integrated circuits.

[0170] Any combination of any steps of any method shown in the specification and / or figures may be provided.

[0171] Any combination of any theme of any claim may be provided.

[0172] Any combination of systems, units, components, processors, and sensors shown in the specification and / or figures may be provided.

[0173] Any reference to an object can be adapted to a mode. Therefore, any reference to object detection, if modified appropriately, can be adapted to mode detection.

[0174] A situation can be a single location / characteristic combination at a given point in time. A scene is a series of events that logically occur within a causal reference framework. Any references to a scene, if modified as appropriate, should be relevant to the situation.

[0175] The sensing information unit may be sensed by one or more sensors of one or more types. One or more sensors may belong to the same device or system, or they may belong to different devices in the system.

Claims

1. A method used for real-time management of detected problems, A classification unit having a neural network makes a classification decision based on sensory information acquired in the vehicle environment. A step of generating automated ground true labeling for the sensing information using one or more computing devices having an automatic labeling function, wherein the automated ground true labeling indicates one or more road elements captured in the sensing information and is a label used as a criterion for evaluating the accuracy of the classification decision, The steps include detecting a problem with the classification decision based on performance metrics related to the automated ground truth labeling using one or more computing devices, The steps include, in response to the detection, resolving the issue detected by a computer device associated with the vehicle while driving the vehicle in the vehicle's environment, using at least the signature generated in relation to the classification decision or the detected problem, In the steps of making the classification decision, detecting the problem, and solving the detected problem, the neural network is in the same state, and the state of the neural network is the weights associated with the neurons of the neural network and the connectivity between neurons. The step of resolving the detected problem includes labeling the signature as being related to a classification error. method.

2. A method used for real-time management of detected problems, A step of automatically generating automated ground true labeling for sensory information acquired in the vehicle environment using one or more computing devices having an automatic labeling function, wherein the automated ground true labeling indicates one or more road elements captured in the sensory information and is a label used as a criterion for evaluating the accuracy of the classification decision, The steps include: detecting a problem related to classification detection performed on the sensing information using a neural network based on performance metrics related to automated ground truth labeling using one or more computing devices; The steps include, in response to the detection, resolving the problem detected by a computer device associated with the vehicle while the vehicle was being driven in the vehicle's environment, using at least a classification determination or a signature generated in relation to the detected problem, In the steps of detecting the aforementioned problem and solving the detected problem, the neural network is in the same state, and the state of the neural network is the weights associated with the neurons of the neural network and the connectivity between neurons. The step of resolving the detected problem includes labeling the signature as being related to a classification error. method.

3. The aforementioned signature is generated in connection with the classification decision. The method according to claim 1 or 2.

4. The further step includes determining that the detected problem should be ignored, The method according to claim 1 or 2.

5. The detected problem includes resolving the detected problem if it causes unnecessary movement of the vehicle. The method according to claim 1 or 2.

6. The step further includes generating a key performance indicator (KPI) report using one or more of the aforementioned computing devices, The method according to claim 1 or 2.

7. The process further includes analyzing problems detected within a certain period and marking the detected problems for further downstream analysis. The method according to claim 1 or 2.

8. The step of solving the detected problem is: Using the aforementioned signature, solve other detected problems that belong to the same cluster as the problem detected in the classification, and / or Using the aforementioned signature, resolve other classification decisions that belong to the same cluster as the aforementioned classification decision. including, The method according to claim 1 or 2.

9. The neural network of the classification unit and another neural network of one or more computing devices with automatic labeling capabilities are trained using the same training dataset. The method according to claim 1.

10. The process further includes the step of generating a signature for the detected problem using another neural network, wherein the other neural network is trained using the same training dataset as the neural network of the classification unit. The method according to claim 1.

11. A non-temporary computer-readable medium used for real-time management of detected problems, The non-temporary computer-readable medium stores instructions for performing the method according to claim 1. do, A non-temporary computer-readable medium.

12. A non-temporary computer-readable medium used for real-time management of detected problems, The non-temporary computer-readable medium stores instructions for performing the method according to claim 2. death, In the above method, in the steps of making the classification decision, detecting the problem, and solving the detected problem, the neural network is in the same state. A non-temporary computer-readable medium.

Images