Autonomous Vehicle Data Recorder

Abstract

Systems, methods, and apparatus for collecting sensor data generated in an autonomous vehicle. Sensors in the vehicle generate a sensor data stream during operation of the vehicle on a road. An advanced driver assistance system (ADAS) of the vehicle uses the sensor data stream to operate the vehicle and generate a trigger signal in response to a failure in object detection, recognition, identification, or classification and / or in response to the detection / prediction of an accident. A circular buffer buffers at least a portion of the sensor data stream. In response to the trigger signal, a selected segment of the sensor data stream is stored in a non-volatile memory. The selected segment may be partially before the trigger signal and partially after the trigger signal; and the selected segment may be longer than a segment that can be fully buffered in the circular buffer at the time of the trigger signal.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. patent application serial number 16 / 263,403, filed on January 31, 2019, entitled “Autonomous Vehicle Data Recorders,” the entire disclosure of which is hereby incorporated herein by reference.

[0003] This application is related to U.S. patent application serial number 16 / 263,359, filed on January 31, 2019, and entitled “Data Recorders for Autonomous Vehicles”; U.S. patent application serial number 15 / 923,820, filed on March 16, 2018, and entitled “Black Box Data Recorder for Autonomous Driving Vehicle”; U.S. patent application serial number 15 / 938,504, filed on March 28, 2018, and entitled “Black Box Data Recorder with Artificial Intelligence Processor in Autonomous Driving Vehicle”; and U.S. patent application serial number 15 / 938,504, filed on June 18, 2018, and entitled “Downloading System Memory Data in Response to Event Detection”. No. 16 / 010,646, entitled “Distributed Architecture for Enhancing Artificial Neural Network,” filed on December 29, 2017, and U.S. patent application serial number 15 / 858,143, entitled “Distributed Architecture for Enhancing Artificial Neural Network,” filed on December 29, 2017, and U.S. patent application serial number 15 / 858,505, entitled “Self-Learning in Distributed Architecture for Enhancing Artificial Neural Network,” filed on December 29, 2017, the entire contents of which are incorporated herein by reference. Technical Field

[0004] At least some embodiments disclosed herein relate to recording sensor data of an autonomous vehicle for subsequent analysis, such as accident review and / or updating data, maps, and artificial neural network models used by advanced driver assistance systems (ADAS). Background Art

[0005] Autonomous vehicles typically contain a number of sensors to help control the operation of the autonomous vehicle. In the event of an accident, collision, or near-collision involving an autonomous vehicle, reviewing sensor data recorded before and / or during the accident may be helpful in potentially helping to determine the cause of the accident and / or whether a design flaw and / or vehicle malfunction existed.

[0006] If power is lost during an accident, vehicle sensor data stored in volatile memory may be lost. Volatile memory requires power to maintain the data stored therein. Examples of volatile memory include dynamic random access memory (DRAM) and static random access memory (SRAM).

[0007] Some memory integrated circuits are nonvolatile and retain stored data even when power is not supplied for long periods of time (e.g., days, months, years). Examples of nonvolatile memory include flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electronically erasable programmable read-only memory (EEPROM) memory, among others.

[0008] The useful service cycles of some non-volatile memories are limited by the cycles of programming and erasing to store new data. A program-erase (P / E) budget represents a predetermined number of program and erase cycles that can be reliably performed to replace data in an erasable medium. After a predetermined number of erase cycles, the program-erase (P / E) budget of this erasable medium is exhausted; and therefore, statistically speaking, the medium may become unreliable and therefore be considered to be at the end of its useful life. For example, the P / E budget of a single-level cell (SLC) flash memory may be between approximately 100,000 cycles; the P / E budget of a multi-level cell (MLC) flash memory may range from 10,000 to 30,000 cycles; and the P / E budget of a triple-level cell (TLC) or quad-level cell (QLC) flash memory may be between 3,000 and 5,000 cycles. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar elements.

[0010] Figure 1A system is shown in which a vehicle is configured with a data logger to collect sensor data for improving its advanced driver assistance system (ADAS) and / or for accident review.

[0011] Figure 2 An autonomous vehicle with a data logger is shown according to one embodiment.

[0012] Figure 3 A data logger according to one embodiment is shown.

[0013] Figure 4 A method of operating a data logger according to one embodiment is shown.

[0014] Figure 5 Another method of operating a data logger according to one embodiment is shown.

[0015] Figure 6 Another data logger according to one embodiment is shown.

[0016] Figure 7 Yet another data logger according to an embodiment is shown.

[0017] Figure 8-10 A method of operating a data logger according to some embodiments is shown. DETAILED DESCRIPTION

[0018] At least some embodiments disclosed herein provide systems, methods, and apparatuses to collect and store sensor data generated in an autonomous vehicle or another vehicle with an advanced driver assistance system (ADAS) under different conditions and for different purposes.

[0019] In one example, after an accident (e.g., a collision or near-collision) or near-accident event, sensor data may be reviewed to determine the cause of the event. This sensor data may be referred to herein as accident sensor data. Accident sensor data may be analyzed to, for example, identify the cause of the accident or near-accident event and / or unsafe aspects or design of the autonomous vehicle. The analysis may result in improved control designs and / or configurations for safe operation of autonomous vehicles and / or similar vehicles.

[0020] In another example, after the ADAS (e.g., autonomous driving system) of the vehicle detects a fault / error in processing sensor data, or detects a mismatch in identifying an object from the sensor data, or fails to identify / classify an object from the sensor data, or fails to recognize, detect, identify, or classify an object with a minimum confidence level, the sensor data can be used to improve the advanced driver assistance system (ADAS). Such sensor data that is not related to an accident or near-accident event may be referred to as training sensor data in this article. For example, improvements may be made by using sensor data for machine learning and / or updating a data model used in the ADAS to classify or identify an object. Such a data model may include a high-definition (HD) map of the road system on which the autonomous vehicle is traveling and / or an artificial neural network for object detection, classification, recognition, and / or identification.

[0021] High-definition (HD) maps are often used for motion planning in autonomous vehicles. This map can have high accuracy at the centimeter level. HD maps can provide geometric and semantic information about the environment (e.g., road system) in which the vehicle operates. The geometric information provided in HD maps can contain raw sensor data collected by light detection and ranging (lidar), radar, cameras, sonar, GPS, etc. The semantic information can identify objects such as lane boundaries, intersections, parking spaces, stop signs, traffic lights, and other information (e.g., vehicle speed, lane change restrictions, etc.).

[0022] Some techniques have been developed to further use the HD maps to generate representations / inputs for object detection, classification, identification and / or recognition using artificial neural networks (ANNs).

[0023] Objects identified in the HD map may be compared to objects identified via sensor data generated by digital cameras, LiDAR (Light Detection and Ranging), RADAR (Radio Detection and Ranging), ultrasonic sonar (Sound Navigation and Ranging), etc. When there is a mismatch in object detection, classification, recognition, or identification, the sensor data may be used for analysis to update the HD map and / or train an artificial neural network (ANN) for object detection, classification, recognition, or identification.

[0024] In some embodiments disclosed herein, the data recorder of the autonomous vehicle is configured with two separate circular buffers for buffering accident sensor data for subsequent accident review and for buffering training sensor data for improving ADAS (e.g., HD maps and / or ANN models). The separate circular buffers can be configured to meet different requirements for buffering sensor data for different purposes (e.g., accident review and ADAS improvement). In other embodiments disclosed herein, a single circular buffer is controlled by a controller to meet the different requirements of the two separate circular buffers.

[0025] For example, the data recorder may be configured with an accident data circular buffer to buffer accident sensor data for subsequent accident review, and a training data circular buffer to buffer training sensor data for improving ADAS.

[0026] For example, a data logger may receive a data stream containing real-time video data from various cameras mounted on a vehicle and various output signals from other sensors (e.g., radar, lidar, ultrasonic sonar, etc.). The sensor data may be raw data from the sensor or compressed image / video data.

[0027] Each circular buffer is configured to buffer data cyclically, where incoming data is filled into the buffer until the buffer is full, and then more new data is buffered to replace the oldest data in the circular buffer. The circular buffer can be implemented using a memory with very high endurance (e.g., allowing hundreds of petabytes of data to be written and rewritten in the circular buffer). For example, the circular buffer can be implemented via SRAM or DRAM (which is volatile and requires power to retain its contents). Alternatively, the circular buffer can be implemented using non-volatile memory with very high endurance (e.g., cross-point memory).

[0028] Cross-point memory uses transistor-free memory elements, each of which has a memory cell and a selector, stacked together as a column. The columns of memory elements are connected via two vertical layers of wires, one above the columns of memory elements and the other below the columns of memory elements. Each memory element can be individually selected at the intersection of a wire on each of the two layers. Cross-point memory devices are fast and non-volatile, and can be used as a unified pool of memory for processing and storage.

[0029] Some non-volatile memories with very high endurance have data retention for a short period of time (eg, minutes or hours).

[0030] The autonomous vehicle may be configured to generate a signal indicating the occurrence of an accident or the prediction of a potential accident. This signal may be referred to herein as an accident signal.

[0031] An accident signal may be generated by an ADAS (e.g., an autonomous driving system) based on sensor data. For example, activation of an automatic emergency brake may result in the generation of an accident signal for a data recorder. For example, activation of an anti-lock braking system may result in the generation of an accident signal for a data recorder. For example, acceleration exceeding a threshold may result in the generation of an accident signal for a data recorder. For example, an accident signal may be generated for a data recorder when a collision avoidance system of the vehicle detects a threat of impact or determines that the probability of an impact is above a threshold. For example, an accident signal may be generated for a data recorder when an airbag sensor determines that a collision is occurring during the initial impact of a car accident. For example, an accident signal may be generated for a data recorder when an ADAS detects a dangerous scenario from sensor data (e.g., another car or object is closer to the autonomous vehicle than a threshold).

[0032] The vehicle's data logger is configured with a non-volatile memory. The non-volatile memory of the data logger may be implemented using a high-density and high-data-retention flash memory that can retain data for months and / or years without power. For example, a triple-level cell (TLC) or quad-level cell (QLC) NAND flash memory may be used as the non-volatile memory of the data logger.

[0033] In response to the accident signal, the data recorder copies the contents of the accident data circular buffer to non-volatile memory so that the accident sensor data can be saved after the accident, which may prevent the data recorder from being powered for a long period of time.

[0034] Preferably, the non-volatile memory of the data recorder is configured to include multiple slots for accident sensor data. Each slot is a dedicated storage space for storing sensor data copied from the accident data circular buffer. In response to each accident signal, the contents of the accident data circular buffer are copied to an available slot in the slots in the non-volatile memory. Initially, all slots are available for storing sensor data copied from the accident data circular buffer. After all slots are used in response to past accident signals, the slot storing sensor data associated with the oldest accident signal can be used to store new current sensor data copied from the accident data circular buffer in response to the current accident signal.

[0035] The data recorder is configured to ensure that the copying of sensor data from the accident data circular buffer to the non-volatile memory is completed even if the accident causes the power supply of the data recorder to be interrupted. For example, the data recorder may have a backup power supply that is sufficient to power the operation to complete the copying of sensor data from the accident data circular buffer to the non-volatile memory even if the external power supply of the data recorder is removed immediately after the accident signal. The backup power supply may be implemented via a capacitor that is charged to store power for operation while the data recorder is normally powered via the external power supply (e.g., during operation of recording data in the accident data circular buffer before the accident). Alternatively or in combination, the backup power supply may include a battery mounted within the housing of the data recorder.

[0036] The accident data circular buffer is configured to buffer sensor data for a period of time (e.g., 30 seconds) before the accident signal. In response to the accident signal, the sensor data buffered during this period of time is copied to a slot in the non-volatile memory of the data recorder; and before the corresponding data is copied to the slot, the existing data in the accident data circular buffer at the time of the accident signal will not be overwritten by the new data from the sensor.

[0037] In one embodiment, in response to an accident signal, the accident data circular buffer temporarily stops buffering incoming new sensor data to save its contents and resumes buffering data after at least the oldest portion of the contents in the accident data circular buffer has been copied to a slot in the non-volatile memory of the data recorder. Optionally, the accident data circular buffer may resume buffering of incoming new sensor data after completing storage of existing data of the accident signal in the non-volatile memory. Optionally, the accident data circular buffer may resume buffering of incoming new sensor data after a predetermined portion of the contents of the accident data circular buffer has been copied to the slot in the non-volatile memory and before the entire contents of the accident data circular buffer are copied to the slot, wherein the buffering of the incoming new sensor data does not overwrite the accident sensor data to be copied to the non-volatile memory of the data recorder.

[0038] In another embodiment, in response to an accident signal, the accident data circular buffer is configured to read the oldest portion of the contents in the accident data circular buffer to record in the non-volatile memory, and then record the incoming new sensor data in place to overwrite the oldest portion in the accident data circular buffer. Thus, the data recorder vacates the contents from the accident data circular buffer to the slots in the non-volatile memory while continuing to record the incoming sensor data in the vacated portion of the accident data circular buffer; and the data recorder can continue to buffer the incoming sensor data without failing to buffer any incoming sensor signals while copying the contents associated with the accident signal (e.g., generated within a 30 second period prior to the accident signal) to the slots in the non-volatile memory.

[0039] Optionally, the data logger and / or the autonomous vehicle can be configured to upload the data content from the slot in the non-volatile memory to a remote server via a communication network / connection. For example, the data transfer can be configured to be performed in one of a number of options. One option is to wirelessly transmit the data to the server via a cellular communication network. Another option is to wirelessly transmit the data to the server via a wireless local area network (e.g., Wi-Fi). Yet another option is to transmit via a wired connection through a diagnostic port of the autonomous vehicle. Yet another option is to remove the data logger from the autonomous vehicle to connect to the communication port of the data logger by wire.

[0040] The accident sensor data recorded in the slot of the non-volatile memory of the data recorder may be used to conduct an investigation of an accident or near-accident event.

[0041] The autonomous vehicle may generate a signal indicating a request to save sensor data for training and / or improving the ADAS. This signal may be referred to herein as a training signal.

[0042] For example, a training signal may be generated by an ADAS (e.g., an autonomous driving system) in response to a mismatch between a road object (e.g., a lane boundary, an obstacle, a traffic sign, etc.) identified in an HD map and a corresponding object detected or recognized from sensor data. Sensor data associated with the mismatch may be used to train the ADAS to recognize the object and / or update the corresponding road segment of the HD map.

[0043] The training data circular buffer is configured to buffer the training sensor data in parallel with the accident data circular buffer buffering the accident sensor data. The training data circular buffer is configured to have a capacity to store data collected over a longer period of time (e.g., 3 minutes) compared to the accident data circular buffer (e.g., 30 seconds).

[0044] In response to the training signal, the training sensor data is copied from the training data circular buffer to the non-volatile memory of the data recorder. The training sensor data copied in response to the training signal may include a portion generated before the training signal and a remaining portion generated after the training signal. In some cases, the time period of the portion generated before the training signal is equal to or longer than the length of the stream segment buffered in the accident data circular buffer (e.g., 30 seconds or more). In other cases, all the training sensor data copied in response to the training signal is generated before the training signal. Optionally, a time period selection indication is provided with the training signal so that the data recorder can selectively copy the portion generated before the training signal and the remaining portion generated after the training signal. Alternatively, the ADAS can select the time period by adjusting the timing of the training signal so that the time window predetermined relative to the training signal is the time period of the sensor data stream segment to be saved as the training sensor data. In some cases, when the training signal is received in the data recorder, the training data circular buffer continues to buffer the incoming sensor data signal until the remaining portion generated after the training signal is buffered in the training data circular buffer to be copied to the non-volatile memory of the data recorder. Optionally, the training data circular buffer may temporarily stop buffering further incoming sensor data after the remaining portion generated after the training signal is buffered in the training data circular buffer and before completing copying the training sensor data associated with the training signal to the non-volatile memory of the training data recorder. The copying of the training sensor data may be configured to begin in response to the training signal and before completing buffering of all training sensor data associated with the training signal. Alternatively, the copying of the training sensor data may be configured to begin at or after completing buffering of all training sensor data associated with the training signal.

[0045] Preferably, the non-volatile memory of the data recorder is configured with a dedicated area which is reserved for storing training sensor data. The training data area in the non-volatile memory is separate from a slot for sensor data configured for accident review.

[0046] Optionally, the training data area may have a plurality of slots for storing training sensor data associated with a plurality of training signals.The data logger may be configured to retain the training sensor data associated with the most recent training signal.

[0047] Optionally, the data recorder is further configured to receive a priority indication of the training signal. For example, the ADAS can evaluate the degree of mismatch between the objects identified in the HD map and the corresponding objects detected / identified from the sensor data. A high degree of mismatch will be assigned a high priority. Similarly, a low confidence level when detecting, classifying, recognizing or identifying an object can be assigned a high priority. Therefore, when the data recorder does not have enough slots to store training data that has not yet been transmitted to the server, the data recorder can be configured to retain the training sensor data based on the priority indication.

[0048] The content copied to the training data area of ​​the data logger's non-volatile memory can be automatically transmitted to a remote server via some of the communication options discussed above for transmitting accident data (e.g., wireless cellular communication networks (e.g., 5G cellular communications) and / or wireless local area networks (e.g., Wi-Fi)). Depending on the network bandwidth and coverage, the data transmission may take a period of several minutes to several hours / days. The data logger's non-volatile memory can retain the data in the absence of power until the data transmission is completed.

[0049] Figure 1 A system is shown in which a vehicle is configured with a data logger to collect sensor data for improving its advanced driver assistance system (ADAS) and / or for accident review.

[0050] Figure 1 The system includes a vehicle (111) having a data recorder (101). The data recorder (101) can be configured according to any of the embodiments disclosed herein. The vehicle (111) has an advanced driver assistance system (ADAS) (105) and one or more sensors (103) that provide sensor data input to the ADAS (105).

[0051] For example, the sensor (103) may include a digital camera, a lidar, a radar, an ultrasonic sonar, a brake sensor, a speed sensor, an acceleration sensor, an airbag sensor, a GPS (Global Positioning System) receiver, etc.

[0052] The output of the time-varying sensor (103) is provided as a sensor data stream to the ADAS (105) to support its operation and to the data recorder (101) for buffering and / or recording. The ADAS (105) may generate an accident signal and / or a training signal to initiate the transfer of sensor data from the circular buffer to the non-volatile memory of the data recorder (101), as discussed above.

[0053] For example, in response to detection of an accident or near-accident event, the ADAS (105) may generate an accident signal to cause the data recorder (101) to store accident sensor data (e.g., 127) for a duration B (e.g., 30 seconds). The accident sensor data (127) is typically generated by a sensor for a duration B (e.g., 30 seconds) and buffered in a circular buffer of the data recorder (101) prior to the accident signal generated by the ADAS (105).

[0054] For example, in response to detecting the possibility of training or improving the ADAS (105) other than an accident or near-accident event, the ADAS (105) may generate a training signal to cause the data recorder (101) to store training sensor data (e.g., 121) for a duration A (e.g., 3 minutes). The training sensor data (121) is generated by the sensor for the duration A (e.g., 3 minutes) and buffered in a circular buffer of the data recorder (101). The duration A (e.g., 3 minutes) may include a portion before the training signal generated by the ADAS (105) and optionally include a portion after the training signal.

[0055] The vehicle (111) is configured with a wireless communication device to transmit accident sensor data (127) and / or training sensor data (121) to a remote server (119) via a wireless signal (113) and a communication network (117). The remote server (119) is typically configured at a location remote from the road (102) served by the vehicle (111). An example of a communication network (117) is a cellular telephone network having one or more base stations (e.g., 115) to receive wireless signals (e.g., 113). Another example of a communication network (117) is the Internet, where a wireless local area network signal (e.g., 113) transmitted by the vehicle (113) is received in an access point (e.g., 115) to further communicate with the server (119). In some embodiments, the vehicle (111) transmits the sensor data (e.g., 121 or 127) to the server (119) using a communication link (107) to a satellite (109) or a communication balloon.

[0056] The server (119) may be configured to include an accident module (129). The accident sensor data (127) may be reviewed and / or analyzed in the accident module (129) to identify causes of the accident and / or possible design or configuration changes to improve autonomous driving and / or ADAS.

[0057] The server (119) may be configured to include a training and updating model (123). The training sensor data (121) may be used to train and update the module (123) to improve an artificial neural network (ANN) (125) of an ADAS (105) and / or update a high definition (HD) map (126) of a road (102) used by the vehicle (111).

[0058] In some embodiments, the ADAS (105) may optionally include a machine learning module configured to improve its ANN using training sensor data (121) stored in the non-volatile memory of the data logger (101), with or without assistance from a server (119).

[0059] Figure 2 An autonomous vehicle (111) with a data logger (101) is shown according to one embodiment. For example, Figure 1 The vehicle (111) in the system can use Figure 2 The method is implemented by an autonomous vehicle (111).

[0060] Figure 2 The vehicle (111) is configured with an advanced driver assistance system (ADAS) (105). The ADAS (105) of the vehicle (111) may have a high definition (HD) map (126) for motion planning and an artificial neural network (ANN) (125) for object detection, recognition, identification and / or classification. Optionally, the HD map (126) may also be used for object detection, recognition, identification and / or classification.

[0061] When there is a mismatch between objects identified in the HD map (126) and the identification of objects via the ANN (125), the ADAS (105) may generate a training signal for the data recorder (101).

[0062] When the ADAS (105) detects an accident or a near-accident event, the ADAS (105) may generate an accident signal for the data recorder (101).

[0063] The vehicle (111) typically includes an infotainment system (149), a communication device (139), one or more sensors (103), and a computer system (131) connected to some controls of the vehicle (111), such as a steering control (141) for the direction of the vehicle (111), a brake control (143) for stopping the vehicle (111), an acceleration control (145) for the speed of the vehicle (111), etc. In some embodiments, Figure 1 The vehicles (111) in the system have similar configurations and / or similar components.

[0064] The computer system (131) of the vehicle (111) includes one or more processors (133), a data recorder (101), and a memory (135) storing firmware (or software) (147) including computer instructions and a data model of the ADAS (105). The data model of the ADAS (105) may include an HD map (126) and an ANN (125).

[0065] The one or more sensors (103) may include a visible light camera, an infrared camera, a lidar, a radar or sonar system, a peripheral sensor, a global positioning system (GPS) receiver, a satellite positioning system receiver, a brake sensor and / or an air bag sensor. The sensor may provide a real-time sensor data stream to the computer system (131). The sensor data generated by the vehicle's sensors (103) may include images of objects captured using a camera that uses light visible to the human eye for imaging or a camera that uses infrared light or a sonar, radar or LIDAR system for imaging. Image data obtained from at least one sensor of the vehicle is part of the collected sensor data for recording in the data recorder.

[0066] The output of the ADAS (105) may be used to control (e.g., (141), (143), (145)) the acceleration of the vehicle (111), the speed of the vehicle (111), and / or the direction of the vehicle (111) during autonomous driving.

[0067] In some embodiments, after the server (119) updates the ANN (125) and / or the HD map (126), the server (119) may update the ADAS of the vehicle (111) over the air by downloading the ANN (125) and / or the HD map (126) to the computer system (131) of the vehicle (111) via a communication device (139) and / or a wireless signal (113).

[0068] Figure 3 A data logger according to one embodiment is shown. For example, Figure 1 systems and / or Figure 2 The data logger (101) in the vehicle (111) can be used Figure 3 The method is implemented by a data logger (101).

[0069] Figure 3The data recorder (101) may include a housing (137) enclosing its components. Optionally, the housing (137) of the data recorder (101) seals the components of the data recorder (101) and protects the data recorder (101) and / or data stored in the non-volatile memory (165) from being corrupted during an accident; and thus, the data stored in the non-volatile memory (165) of the data recorder (101) can be retrieved after an impact / crash, fire, and / or flood (which is typical in accidents involving vehicles).

[0070] Figure 3 The data recorder (101) includes one or more communication interfaces or devices (153), an accident data circular buffer (161), a training data circular buffer (163), a controller (159) and a non-volatile memory (165).

[0071] The communication interface or device (153) can be used to receive a sensor data stream (151) from a sensor (103) of a vehicle (111) in which the data recorder (101) is installed. The sensor data stream (151) is provided in parallel to an accident data circular buffer (161) and a training data circular buffer (163).

[0072] Additionally, the communication interface or device (153) may be used to receive training signals (eg, 157) and accident signals (155) from the vehicle (111) and / or its ADAS (105).

[0073] In response to the accident signal (155), the controller (159) stops recording the incoming sensor data stream (151) into the accident data circular buffer (161) and starts copying the contents from the accident data circular buffer (161) to the non-volatile memory (165).

[0074] The non-volatile memory (165) has a plurality of slots (171, 173, ..., 179). Each slot is sufficient to store the entire contents of the accident data circular buffer (161). The controller (159) can use the slots (171, 173, ..., 179) in a round robin (cyclic) manner, so that after multiple accident signals, the slots (171, 173, ..., 179) store the latest accident sensor data set associated with and / or identified by the most recent accident signal.

[0075] In response to the training signal (157), the controller (159) may continue to buffer the incoming sensor data stream (151) into the training data circular buffer (163), such that the training data circular buffer (163) contains a sensor data set (partially recorded before the training signal (157) and partially recorded after the training signal (157). The controller (159) then copies the training sensor data set to an area (168) reserved for training sensor data in the non-volatile memory (165).

[0076] Optionally, the area (168) may also include a plurality of slots, each slot being sufficient to store the contents of the training data circular buffer (163).

[0077] Optionally, the data logger (101) includes a backup power source, such as a capacitor or a rechargeable battery.

[0078] Figure 4 A method of operating a data logger according to one embodiment is shown. For example, Figure 4 The method can be Figure 1 in the system Figure 2 Vehicles (111) Figure 3 The method is implemented in a data logger (101).

[0079] At blocks 201 and 211 , the data recorder ( 101 ) provides a first circular buffer ( 161 ) and a second circular buffer ( 163 ).

[0080] At block 203 , the data logger ( 101 ) buffers a first sensor data stream for a first duration (eg, 30 seconds) in a first circular buffer ( 161 ).

[0081] At block 213 , the data logger ( 101 ) buffers a second sensor data stream for a second duration (eg, 3 minutes) in a second circular buffer ( 163 ).

[0082] The first and second circular buffers (161 and 163) operate in parallel. Data is buffered in each circular buffer (161 and 163) in a circular manner so that when the buffer is full, the newest data overwrites the oldest data. For example, data is recorded from the beginning of the buffer to the end of the buffer. Once the end of the buffer is reached, the data will be written again from the beginning of the buffer to the end of the buffer as if the end of the buffer is after the beginning of the buffer.

[0083] At block 205, the data recorder (101) receives a request to record sensor data. The request may be an accident signal or a training signal.

[0084] At block 207 , the data recorder ( 101 ) determines whether the request is for recording an incident (eg, a collision or near collision).

[0085] If the request is to record an incident, then at block 209 the data recorder (101) copies the first content from the first circular buffer (161) to the non-volatile memory (165).

[0086] Optionally, the data logger (101) stops buffering incoming sensor data into the first circular buffer (161) for a period of time in response to a request to record an incident (e.g., to prevent existing data in the first circular buffer (161) from being overwritten and / or to preserve backup power in the event that an external power source to the data logger (101) is interrupted).

[0087] At block 221, the data recorder (101) transmits or provides first content from the non-volatile memory (165) to the server (119) after the incident.

[0088] At block 223, the server (119) reviews the incident based on the first content.

[0089] If the request is not to record an incident, then at block 219, the data recorder (101) copies the second content from the second circular buffer (163) to the non-volatile memory (165).

[0090] Optionally, the data recorder (101) continues to buffer incoming sensor data into the first circular buffer (161) for a period of time in response to the request, and thus buffers the second content containing the sensor data for a certain duration (partly before the request and partly after the request). After the period of continued buffering, the data recorder (101) may stop buffering for another period of time before completing copying the second content to the non-volatile memory (165).

[0091] At block 231 , the data recorder ( 101 ) transmits the second content from the non-volatile memory ( 165 ) to the server ( 119 ).

[0092] At block 233 , the server ( 119 ) improves an advanced driver assistance system (ADAS) based on the second content.

[0093] Figure 5 Another method of operating a data logger according to one embodiment is shown. For example, Figure 5 The method can be Figure 1 in the system Figure 2 Vehicles (111) Figure 3 The method is implemented in a data logger (101). Figure 5 The method can be used with Figure 4 methods are combined.

[0094] At block 251, the data recorder (101) provides a first circular buffer (161) for buffering sensor data related to the accident and a second circular buffer (163) for buffering sensor data not related to the accident.

[0095] At block 253, the data logger (101) provides non-volatile memory (165).

[0096] At block 255, the data recorder (101) divides the non-volatile memory (165) into a first portion (171, 173, ..., 179) and a second portion (167), wherein the storage capacity of the first portion is several times the capacity of the first circular buffer (161), and the second portion is sufficient to store the data buffered in the second circular buffer (163).

[0097] At block 257, the data recorder (101) organizes the first portion into a predetermined number of slots (171, 173, ..., 179), each slot having a capacity to store the data buffered in the first circular buffer (161).

[0098] At block 259, the data recorder (101) stores sensor data of up to a predetermined number of recent incidents in a predetermined number of slots (171, 173, ..., 179).

[0099] For example, the data recorder (101) copies the accident sensor data from the accident data circular buffer (161) to the first slot (171) in response to an initial accident signal. For each subsequent accident signal, the data recorder (101) stores the accident sensor data in the next slot (e.g., 173, ..., 179). After reaching the last slot (179), the data recorder (101) uses the first slot (171) as the next slot, as if the first slot (171) is the next slot after the last slot (179).

[0100] Optionally, the second portion (167) reserved for the training sensor data buffered in the training data circular buffer (163) is also configured to have a plurality of slots. Each slot stores an indicator identifying the priority of the training sensor data stored in the corresponding slot. When the slots in the second portion (167) are full, the data logger (101) reuses the slot with the lowest priority lower than the current training sensor data in the training data circular buffer (163).

[0101] In one example, an autonomous vehicle (111) has a sensor (103) configured to generate a sensor data stream (151) during operation of the autonomous vehicle (111) on a road (102). An advanced driver assistance system (ADAS) (105) of the vehicle (111) can be configured to autonomously operate the vehicle (111) on the road (102) based on the sensor data stream (151). The advanced driver assistance system (ADAS) (105) is further configured to generate an accident signal (155) in response to detection or prediction of an accident and to generate a training signal (157) in response to a failure in object detection, recognition, identification, or classification.

[0102] The data recorder (101) of the vehicle (111) in the example includes a non-volatile memory (165), a first circular buffer (161), a second circular buffer (163), and a controller (159). The first circular buffer (161) has a capacity to buffer a first segment (e.g., 30 seconds) of the sensor data stream (151); and the second circular buffer has a capacity to buffer a second segment (e.g., 3 minutes) of the sensor data stream (151). In the absence of an accident signal and a training signal, the first circular buffer (161) and the second circular buffer (163) are respectively configured to buffer the sensor data stream (151) in parallel and cyclically within their respective capacities. Before the generation of the accident signal (155) and the training signal (157), the sensor data stream segment buffered in the second circular buffer (163) may include the sensor data stream segment buffered in the first circular buffer (161).

[0103] In response to the accident signal (155), the controller (159) is configured to copy the sensor data stream segment from the first circular buffer (161) to the non-volatile memory (165). In response to the training signal (157), the controller (159) is configured to copy the sensor data stream segment from the second circular buffer (163) to the non-volatile memory (165).

[0104] The sensor data stream (151) from the sensor (103) may include the output of the sensor (103) (e.g., a digital camera, radar, lidar, or ultrasonic sonar, or any combination thereof). The advanced driver assistance system (ADAS) (105) may include a map (126) of the road (102) and an artificial neural network (125) for detecting, recognizing, identifying, and / or classifying objects on the road (102) based on the sensor data stream (151). In response to a mismatch between identification of objects in the map (126) of the road (102) and identification of corresponding objects by applying the sensor data stream (151) in the artificial neural network (125), the advanced driver assistance system (ADAS) (105) may generate a training signal (155).

[0105] Preferably, after the training signal (157), the autonomous vehicle (111) is configured to automatically transmit the sensor data stream segments copied to the non-volatile memory (165) to the remote server (119) in response to the training signal (157). The server (119) can be configured with a training and updating module (123) to update the map (126) of the road (102) based on the sensor data stream segments received from the vehicle (111), and / or update the artificial neural network (125) using machine learning techniques (e.g., supervised machine learning techniques).

[0106] In response to the accident signal (155), segments of the sensor data stream (151) generated before the accident signal (155) and / or buffered in the first circular buffer (161) at the time of the accident signal (155) are copied to the non-volatile memory (165). Optionally, the controller (159) can stop the first circular buffer (161) from buffering incoming segments of the sensor data stream (151) immediately after the accident signal (155) to prevent the loss of a portion of the segments of the sensor data stream (151) located in the first circular buffer (161) at the time of the accident signal (155) before at least a portion of the content is copied from the first circular buffer (161) to the non-volatile memory (165).

[0107] Optionally, in response to the training signal (157), segments of the sensor data stream (151) generated partially before the training signal (157) and partially after the training signal (157) are buffered in the second circular buffer (163) and copied to the non-volatile memory (165). For example, the controller (159) may continue to use the second circular buffer (161) to buffer segments of the sensor data stream (151) after the training signal (157) to complete buffering of segments to be copied to the non-volatile memory (165). Once the segments of the sensor data stream (151) to be copied to the non-volatile memory (165) are in the second circular buffer (163), the controller (159) may then stop the second circular buffer (163) from buffering further incoming segments of the sensor data stream (151).

[0108] In some embodiments, the accident data circular buffer (161) is not used. In response to the accident signal (155), the controller (159) copies a portion of the contents of the training data circular buffer (163) corresponding to the segment to be buffered in the accident data circular buffer (161) to a slot in the non-volatile memory (165). In other embodiments, the training circular buffer (163) is not used. In response to the training signal (157), the controller (159) copies at least a portion of the contents of the accident data circular buffer (161) corresponding to the segment to be buffered in the training data circular buffer (163) to a slot (173) in the non-volatile memory (165), while storing the segment of the sensor data stream (151) following the training signal (155) directly in the area (167) of the non-volatile memory (185). The combination of the segments copied from the accident data circular buffer (161) to the corresponding slot (e.g., 173) and the segments saved directly to the area (167) provides training sensor data corresponding to the training signal (157), as shown below in combination Figure 6 and 7 further discussed.

[0109] In some embodiments, in response to the accident signal (155), the controller (159) may stop operation of the training data circular buffer (163) to preserve power for copying data from the accident data circular buffer (161) to non-volatile memory (e.g., 165) (e.g., in response to detecting a loss of external power to the data recorder (101)).

[0110] The first and second circular buffers (161 and 163) may be volatile memory (e.g., DRAM or SRAM); and the non-volatile memory (165) may include TLC / QLC NAND flash memory. The first and second circular buffers (161 and 163) may buffer a common segment of the sensor data stream (151) before receiving an accident signal (155) or a training signal (157).

[0111] Optionally, the controller (159) divides the non-volatile memory (165) into a first region (171, 173, ..., 179) and a second region (167). In addition, the controller (159) organizes the first region (171, 173, ..., 179) into a plurality of slots (171, 173, ..., 179), wherein each slot (e.g., 171) has a storage capacity sufficient to store the entire contents of the first circular buffer (161). The sensor data stream segment associated with and / or identified by the training signal (157) is copied to the second region (167); and the sensor data stream segment associated with and / or identified by the accident signal (155) is copied to one of the slots (171, 173, ..., 179) in the first region. The controller (159) can use the slots according to a round-robin scheme to maintain up to a predetermined number of accident sensor data stream segments associated with and / or identified by the most recent accident signal generated by an advanced driver assistance system (ADAS) (105) of the autonomous vehicle (111) in the slots (171, 173, ..., 179).

[0112] Optionally, the controller (159) may further organize the second region (167) of the non-volatile memory (165) into slots, wherein each slot has a storage capacity sufficient to store the entire contents of the second circular buffer (163). The controller (159) may store (e.g., in the non-volatile memory or another memory) priority indicators for the data stored in the slots. In response to the training signal (157), the controller (159) selects a slot from the slots in the second region (167) based on the priority indicators of the existing data in the slots in the second region (167) and the priority indicator associated with the training signal (157). The training sensor data is copied from the second circular buffer (163) to the selected slot in the second region (167).

[0113] In some embodiments, at least a portion (e.g., 167) of the non-volatile memory (165) of the data logger (101) has a data logging performance level sufficient to allow the controller (159) to store the incoming sensor data stream in the portion (e.g., 167) without using a circular buffer. Thus, segments of the sensor data stream (151) following a request to save the data to the non-volatile memory (165) can be copied directly to the non-volatile memory (165) without passing through the circular buffer. In this case, the time required to store the sensor data stream (151) can be reduced. Figure 3 The circular buffers (161 and 163) of the data recorder (101) are further discussed below.

[0114] Figure 6 Another data logger according to one embodiment is shown. For example, Figure 6 The data logger (101) can be used in Figure 1 systems and / or Figure 2 in the vehicle (111).

[0115] Figure 6 The data logger (101) may store the sensor data stream (151) directly into area B (167) of the non-volatile memory (165).

[0116] For the life of the data logger (101), Figure 6 A controller (159) of the data logger (101) is configured to use a circular buffer (161) to select a portion of the sensor data stream (151) for storage in a non-volatile memory (165); and to use a trigger signal (156) to identify the segment of the sensor data stream (151) to be stored in the non-volatile memory (165).

[0117] Before the controller (159) receives the trigger signal (156), the controller (159) uses the circular buffer (161) to buffer the most recent segment of the sensor data stream (151). At the time of the trigger signal (156), the circular buffer (161) has the segment of the sensor data stream (151) generated before the trigger signal (156). Therefore, upon or after the arrival of the trigger signal (156), the desired portion of the sensor data stream (151) can be copied from the circular buffer (161) to the non-volatile memory (165).

[0118] After receiving the trigger signal (156), the controller (159) saves at least a desired portion of the sensor data stream segments in the circular buffer (161) and saves subsequent segments of the incoming sensor data stream (151) directly to region B (167) of the non-volatile memory. In parallel with (or after) recording the incoming segments of the sensor data stream (151) to region B (167), the desired portion of the sensor data stream segments in the circular buffer (161) can be copied to region A (166) of the non-volatile memory (165). The segments recorded directly to region B (167) and the segments copied from the circular buffer (161) to region A (166) provide a sensor data set associated with and / or identified by the trigger signal (156) (e.g., based on a time window relative to the arrival of the trigger signal (156)).

[0119] Typically, the trigger signal (156) can be an incident signal (155) or a training signal (155). In some embodiments, the trigger signal (156) includes a time window selector that identifies a time period before the trigger signal (156) and / or a time period after the trigger signal (156). The time period of the time window selector can be used to select a desired portion from the circular buffer (161) to copy into region A (166) and to select segments from the incoming sensor data stream (151) for direct storage into region B (167) (e.g., bypassing the circular buffer (161)).

[0120] After the sensor data set corresponding to the trigger signal (156) is stored in the non-volatile memory (165), the communication device (e.g., 153 and / or 139) can be configured to automatically transmit the sensor data set from areas (166 and 167) of the non-volatile memory (165) to the remote server (119) in a manner similar to that discussed above in conjunction with other embodiments of the data logger (101).

[0121] Figure 7 Another data logger according to an embodiment is shown. For example, Figure 7 The data logger (101) can be used in Figure 1 systems and / or Figure 2 in the vehicle (111).

[0122] Similar to Figure 6 a data logger (101), Figure 7 The data recorder (101) is capable of recording the sensor data stream (151) directly into area B (167).

[0123] In response to the accident signal (155), Figure 7 The controller (159) of the data logger (101) is similar to Figure 3 The data recorder (101) operates the circular buffer (161) in a manner such that the incident sensor data identified by the incident signal (155) is operated. For example, the incident sensor data identified by the incident signal (155) is copied from the circular buffer (161) to one of the slots (171, 173, ..., 179) of the non-volatile memory (165) (e.g., in a circular manner), so that the slot (171, 173, ..., 179) stores the incident sensor data associated with and / or identified by the most recently received incident signal (155).

[0124] In response to the training signal (157), Figure 7 The controller (159) of the data logger (101) is similar to Figure 6The circular buffer (161) is operated in the same manner as the data recorder (101) of the embodiment of the present invention operates its circular buffer (161). In parallel with (or after) saving the portion of the training sensor data generated after the training signal (157) directly to the area B (167) of the non-volatile memory (165), the portion of the training sensor data generated before the training signal (157) is copied from the circular buffer (161) to the area A (166).

[0125] In response to the training signal (157), Figure 7 The controller (159) of the data logger (101) may optionally pause buffering of the incoming sensor data stream (151) into the circular buffer (151).

[0126] Alternatively, instead of writing the incoming sensor data stream (151) following the training signal (157) directly into region B (167), the controller (159) may be configured to buffer into a portion of the circular buffer (161) that is vacated just as the non-volatile memory (165) is written.

[0127] For example, the controller (159) retrieves the oldest portion of the data from the circular buffer (161) while receiving data in the incoming sensor data stream (151). Newly received data from the sensor data stream (151) can be written to the storage location occupied by the oldest portion of the data just retrieved. In response to the training signal (157), the oldest portion of the data just retrieved from the circular buffer (161) can be written to the non-volatile memory (e.g., in regions 166 and 167) while newly received data from the sensor data stream (151) can be written to the circular buffer (161) for buffering. Thus, the sensor data stream (151) is continuously buffered in the circular buffer (161) while a portion of the buffered stream (151) can be selected by the training signal (157) and copied to the non-volatile memory (165).

[0128] Figure 7 The data logger (101) can be similar to Figure 3 The data recorder (101) provides the accident sensor data stored in the slots (171, 173, ..., 179) of the non-volatile memory (165) and / or the training sensor data stored in the areas (166 and 167) to the remote server (119).

[0129] Optionally, multiple slots may be organized in the non-volatile memory (165) for the training data set, wherein each slot has a corresponding area A (166) for accepting data from the circular buffer (161) and a corresponding area B (167) for storing data directly from the sensor data stream (151).

[0130] Figure 8-10 Methods of operating a data logger according to some embodiments are shown. For example, Figure 8-10 The method can be Figure 1 in the system Figure 2 Vehicles (111) Figure 6 or implemented in the data recorder (101) of 7.

[0131] Figure 8 A method is shown in which a controller (159) of a data logger (101) is configured to copy data from a circular buffer (161) to a non-volatile memory (165) in parallel with storing data from an incoming sensor data stream (151) directly into the non-volatile memory (165).

[0132] At block 271, the data logger (101) (e.g., Figure 6 The circular buffer (161) of FIG. 1 or 7) buffers the sensor data stream (151) by cyclically using the capacity of the buffer (161). For example, when the buffer (161) is full, the most recently received data of the sensor data stream (151) is stored in the storage location occupied by the oldest data previously obtained from the sensor data stream (151). Thus, the content of the buffer (161) can be continuously updated to contain the most recent segment of the sensor data stream (151) of a predetermined length (e.g., 30 seconds).

[0133] At block 273 , the data recorder ( 101 ) receives a trigger signal ( 156 ), such as a training signal ( 157 ) or an accident signal ( 155 ).

[0134] At block 275, the controller (159) of the data recorder (101) retrieves a first segment of the sensor data stream (151) from the circular buffer (161), wherein the first segment was generated in the data recorder (101) before the trigger signal (156) arrived (and / or received in the data recorder (101)). For example, the first segment of the sensor data stream (151) may include a 30 second video recording from a digital camera, radar, lidar, ultrasonic sonar, or any combination of such sensors.

[0135] At block 277 , the controller ( 159 ) of the data recorder ( 101 ) stores the first segment of the sensor data stream ( 151 ) retrieved from the circular buffer ( 161 ) into the non-volatile memory ( 165 ) of the data recorder ( 101 ).

[0136] In parallel with the operations performed in blocks 275 and 277, at block 276, the controller (159) of the data recorder (101) stores a second segment of the sensor data stream (151) in the non-volatile memory (165) of the data recorder (101), wherein the second segment of the sensor data stream (151) is generated in the data recorder (101) after the arrival of the trigger signal (156) (and / or received in the data recorder (101)). For example, the second segment of the sensor data stream (151) may include a two minute and thirty second video recording from a digital camera, radar, lidar, ultrasonic sonar, or any combination of such sensors.

[0137] Since the first segment of the sensor data stream (151) arrives at the data recorder (101) before the trigger signal (156), the first segment of the sensor data stream (151) is buffered via the circular buffer (161), so that after the trigger signal (156), the first segment of the sensor data stream (151) is available in the circular buffer (161) for copying to the non-volatile memory (165) of the data recorder (101). In contrast, the second segment of the sensor data stream (151) arrives at the data recorder (101) after the trigger signal (156), and the second segment of the sensor data stream (151) can be written to the non-volatile memory (165) of the data recorder (101) when it arrives without being buffered in any circular buffer.

[0138] At block 279 , the vehicle ( 111 ) and / or the data recorder ( 101 ) may transmit the first and second segments from the volatile memory ( 165 ) to the remote server ( 119 ) as a set of training sensor data (or accident sensor data).

[0139] Fig. 9 A method is shown in which a controller (159) of a data logger (101) is configured to copy data from an incoming sensor data stream (151) to a non-volatile memory (165) after storing the data directly in the non-volatile memory (165).

[0140] At block 271, the circular buffer (161) of the data logger (101) is filled by cyclically using the capacity of the buffer (161) (e.g., in a manner similar to Figure 8 271 of the block 271) to buffer the sensor data stream (151).

[0141] At block 273 , the data logger ( 101 ) receives a trigger signal ( 156 , such as 157 or 155 ).

[0142] At block 281, the controller (159) of the data logger (101) suspends buffering of new data from the sensor data stream (151) into the circular buffer (161). After the second segment of the sensor data stream (151) after the trigger signal (156) is saved to the non-volatile memory (165), the first segment of the sensor data stream (151) in the circular buffer (161) at the time of the trigger signal (156) can be copied to the non-volatile memory (165) without passing through the circular buffer (161).

[0143] At block 276, the controller (159) of the data recorder (101) stores a second segment of the sensor data stream (101) generated and / or received after the trigger signal (156) in the non-volatile memory (165) of the data recorder (101) (e.g., in a format similar to Figure 8 The manner in which block 276 operates).

[0144] After writing the second segment of the sensor data stream (101) to the non-volatile memory, at block 283, the controller (159) of the data recorder (101) copies the first segment of the sensor data stream (151) generated and / or received before the trigger signal to the non-volatile memory (165) of the data recorder (101).

[0145] After copying the first segment from the circular buffer (161) to the non-volatile memory (165), the circular buffer (161) may erase its contents and / or resume buffering of the incoming sensor data stream (151).

[0146] At block 279, the vehicle (111) and / or the data logger (101) may transmit the first and second segments from the volatile memory (165) to the remote server (119) (e.g., in a manner similar to Figure 8 operation 279).

[0147] In some embodiments, data in the circular buffer (161) can be retrieved according to a first-in-first-out (FIFO) scheme, such that the incoming sensor data stream (151) can be continuously buffered in the circular buffer (161), and the data in the circular buffer (161) can be optionally retrieved to be saved to the non-volatile memory (165). The circular buffer (161) effectively delays the sensor data stream (101) for a period of time (e.g., 30 seconds), such that in response to the trigger signal (156), a delayed segment retrieved from the circular buffer (161) can be selected to be stored in the non-volatile memory (165), wherein the delayed segment can include a first segment generated by the sensor (103) before the trigger signal (156) and a second segment generated by the sensor (103) after the trigger signal (156), as shown in FIG. Fig.10 method.

[0148] At block 301 , the controller ( 159 ) of the data logger ( 101 ) receives a new portion of the sensor data stream ( 151 ).

[0149] In parallel with receiving the new portion, at block 305, the controller (159) also retrieves from the circular buffer (161) the oldest portion of the segment of the sensor data stream (151) currently buffered in the circular buffer (161).

[0150] At block 303, the controller (159) buffers the new portion into the circular buffer (161) by overwriting the oldest portion in the circular buffer (161). Since the oldest portion has already been retrieved from the circular buffer (161), buffering of the new portion does not result in loss of the oldest portion.

[0151] If it is determined at block 307 that the retrieved portion should be saved (e.g., after a trigger signal has been received in the data logger (101)), then at block 309, the controller (159) stores the retrieved portion in the non-volatile memory (165) of the data logger (101) (e.g., in parallel with writing the new portion to the circular buffer (161) at block 303).

[0152] The buffering of block 303 and the storage of block 309 may be performed in parallel with the receipt of the next new portion of block 301 and the retrieval of the next oldest portion of block 305. Thus, new incoming portions of the sensor data stream (151) may be continuously buffered into the circular buffer while older delayed portions of the sensor data stream (151) may be retrieved from the circular buffer for storage into the non-volatile memory (165).

[0153] The present disclosure includes methods and apparatus for performing the above methods, including data processing systems for performing these methods, and computer-readable media containing instructions that, when executed on a data processing system, cause the system to perform these methods.

[0154] The server (119), computer system (131), and / or data logger (101) may each be implemented as one or more data processing systems.

[0155] A typical data processing system may contain an interconnect (eg, a bus and system core logic) that interconnects a microprocessor and memory. A microprocessor is typically coupled to a cache memory.

[0156] The interconnect interconnects the microprocessor and the memory together, and also interconnects them to input / output (I / O) devices via an I / O controller. The I / O devices may include display devices and / or peripheral devices, such as a mouse, keyboard, modem, network interface, printer, scanner, camera, and other devices known in the art. In one embodiment, when the data processing system is a server system, some I / O devices (e.g., printer, scanner, mouse, and / or keyboard) are optional.

[0157] The interconnection may include one or more buses interconnected by various bridges, controllers and / or adapters. In one embodiment, the I / O controller includes a USB (Universal Serial Bus) adapter for controlling USB peripherals and / or an IEEE-1394 bus adapter for controlling IEEE-1394 peripherals.

[0158] The memory may include one or more of: ROM (read only memory), volatile RAM (random access memory), and non-volatile memory (eg, hard drive, flash memory, etc.).

[0159] Volatile RAM is usually implemented as dynamic RAM (DRAM), which requires continuous power to refresh or maintain the data in the memory. Non-volatile memory is usually a magnetic hard drive, magneto-optical drive, optical drive (e.g., DVDRAM), or other type of storage system that can maintain data even after power is removed from the system. Non-volatile memory can also be random access memory.

[0160] The non-volatile memory may be a local device directly coupled to the rest of the components in the data processing system. Non-volatile memory that is remote from the system may also be used, such as a network storage device coupled to the data processing system through a network interface (e.g., a modem or Ethernet interface).

[0161] In the present disclosure, in order to simplify the description, some functions and operations are described as being performed or caused by software codes. However, such expressions are also used to specify that the functions are generated by a processor (eg, a microprocessor) executing code / instructions.

[0162] Alternatively or in combination, dedicated circuitry (with or without software instructions) may be used to implement the functions and operations described herein, such as using an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). Embodiments may be implemented using hardwired circuitry (without software instructions) or in combination with software instructions. Thus, the described techniques are not limited to any specific combination of hardware circuitry and software, nor to any specific source of instructions executed by a data processing system.

[0163] Although one embodiment can be implemented in fully functional computers and computer systems, various embodiments can be distributed in various forms as computing products and can be applied regardless of the specific type of machine or computer readable media used to actually implement the distribution.

[0164] At least some aspects disclosed may be embodied at least in part in software. That is, the techniques may be implemented in a computer system or other data processing system in response to its processor (e.g., microprocessor) executing sequences of instructions contained in a memory (e.g., ROM, volatile RAM, non-volatile memory, cache, or remote storage device).

[0165] The routines executed to implement the embodiments may be implemented as part of an operating system or as a specific application, component, program, object, module, or sequence of instructions referred to as a “computer program.” A computer program typically includes one or more instructions that are located at different times in various memories and storage devices in a computer, and when read and executed by one or more processors in a computer, the instructions cause the computer to perform the operations necessary to perform the elements involved in the various aspects.

[0166] Machine-readable media can be used to store software and data, which, when executed by a data processing system, cause the system to perform various methods. Executable software and data can be stored in various places, including, for example, ROM, volatile RAM, non-volatile memory and / or cache. Parts of the present software and / or data may be stored in any one of these storage devices. In addition, data and instructions can be obtained from a centralized server or a peer-to-peer network. Different parts of data and instructions can be obtained from different centralized servers and / or peer-to-peer networks at different times and in different communication sessions or the same communication session. Data and instructions can be obtained completely before executing the application. Alternatively, parts of data and instructions can be obtained dynamically and timely when execution is required. Therefore, it is not required that data and instructions be completely located on machine-readable media at a specific moment.

[0167] Examples of computer-readable media include, but are not limited to, non-transitory, recordable and non-recordable types of media, such as volatile and non-volatile storage devices, read-only memory (ROM), random access memory (RAM), flash memory devices, floppy disks and other removable disks, magnetic disk storage media, optical storage media (e.g., compact disk read-only memory (CD ROM), digital versatile disk (DVD), etc.), etc. Computer-readable media may store instructions.

[0168] The instructions may also be embodied in digital and analog communication links, for electrical, optical, acoustical or other forms of propagated signals, such as carrier waves, infrared signals, digital signals, etc. However, propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.) are not tangible machine-readable media and are not configured to store instructions.

[0169] Generally, a machine-readable medium includes any mechanism that provides (ie, stores and / or transmits) information in a form accessible by a machine (eg, a computer, a network device, a personal digital assistant, a manufacturing tool, any device with a set of one or more processors, etc.).

[0170] In various embodiments, hardwired circuitry may be used in combination with software instructions to implement the techniques. Thus, the techniques are not limited to any specific combination of hardware circuitry and software, nor to any specific source of instructions executed by a data processing system.

[0171] The above description and drawings are illustrative and should not be construed as limiting. Many specific details are described to provide a thorough understanding. However, in some cases, well-known or conventional details are not described to avoid confusing the description. References to one embodiment in this disclosure do not necessarily refer to the same embodiment; and, such references refer to at least one.

[0172] In the foregoing description, the present disclosure has been described with reference to specific exemplary embodiments of the present disclosure. It will be apparent that various modifications may be made thereto without departing from the broader spirit and scope set forth in the following claims. Accordingly, the description and drawings are to be regarded as illustrative rather than restrictive.

Claims

1. An autonomous vehicle comprising: a sensor configured to generate a sensor data stream during operation of the autonomous vehicle on a road; an advanced driver assistance system configured to operate the autonomous vehicle on the road based on the sensor data stream, wherein the advanced driver assistance system is further configured to generate a training signal in response to a failure in object detection, recognition, identification, or classification, and to generate an accident signal in response to detection or prediction of an accident; and Non-volatile memory; a circular buffer configured to buffer the sensor data stream; and A controller configured to receive a training signal and in response, store a first segment of the sensor data stream in the non-volatile memory, wherein the first segment is longer than a second segment of the sensor data stream buffered in the circular buffer during the training signal, and to stop buffering data from the sensor data stream in the circular buffer in response to the accident signal.

2. The autonomous vehicle of claim 1, wherein the first segment includes the second segment.

3. The autonomous vehicle of claim 2 , wherein the controller is configured to, in response to the training signal, copy the second segment from the circular buffer to the non-volatile memory and write a third segment to the non-volatile memory, the third segment following the second segment in the sensor data stream without passing through any circular buffer.

4. The autonomous vehicle of claim 3, wherein the controller is configured to copy the second segment to the non-volatile memory in parallel with writing the third segment to the non-volatile memory in response to the training signal.

5. The autonomous vehicle of claim 2, wherein the controller is configured to retrieve an oldest portion from the circular buffer while receiving a portion from the sensor data stream and buffer the received portion into a memory cell occupied by the oldest portion.

6. The autonomous vehicle of claim 5, wherein the controller is configured to write the oldest portion retrieved from the circular buffer in parallel with buffering the received portion into the circular buffer in response to the training signal.

7. The autonomous vehicle of claim 1, wherein the sensor comprises a digital camera, radar, lidar, or ultrasonic sonar, or any combination thereof.

8. The autonomous vehicle of claim 7 , wherein the advanced driver assistance system comprises a map of the road and an artificial neural network; and the advanced driver assistance system is configured to generate the training signal in response to a mismatch between an identification of an object in the map of the road and an identification of a corresponding object via applying the sensor data stream in the artificial neural network.

9. The autonomous vehicle of claim 8, wherein after the training signal, the autonomous vehicle is configured to transmit the first segment of the sensor data stream from the non-volatile memory to a remote server, the remote server being configured to use the first segment of the sensor data stream to update at least a map of the road, the artificial neural network, or both the map and the artificial neural network.

10. The autonomous vehicle of claim 9, wherein the controller is further configured to receive the distress signal and in response, copy the contents of the circular buffer into the non-volatile memory.

11. A data recorder for an autonomous vehicle, comprising: Non-volatile memory; a communication interface configured to receive a sensor data stream from a sensor of the autonomous vehicle; a first circular buffer; and A controller configured to: In the absence of a trigger signal, continuously buffering the sensor data stream into the first circular buffer; In response to a training signal received as the trigger signal in the communication interface: continuing to buffer the sensor data stream into the first circular buffer; copying at least a portion of the contents of the first circular buffer into the non-volatile memory; and storing a portion of the sensor data stream subsequent to the trigger signal in the non-volatile memory without using a second circular buffer to buffer the portion of the sensor data stream; and In response to an accident signal received as the trigger signal in the communication interface, buffering of the sensor data stream in the first circular buffer is stopped.

12. The data recorder of claim 11, wherein the trigger signal is a training signal generated by an advanced driver assistance system of the vehicle in response to a failure in object detection, recognition, identification or classification.

13. The data recorder of claim 12, wherein the controller is configured to receive a time window indicator and select the portion of the content and the portion of the sensor data stream according to the time window indicator.

14. The data recorder of claim 13, wherein the controller is configured to simultaneously copy the portion of the content and store the portion of the sensor data stream.

15. The data logger of claim 14, wherein the first circular buffer comprises volatile memory; And the non-volatile memory includes a NAND flash memory.

16. A method for recording a sensor data stream, comprising: receiving, in a data recorder of an autonomous vehicle, said sensor data stream generated by sensors of said autonomous vehicle during operation of said autonomous vehicle on a road; In the absence of any trigger signal, continuously buffering the sensor data stream into a circular buffer of the data logger; In the data recorder, receiving a trigger signal generated by an advanced driver assistance system of the autonomous vehicle; and In response to the trigger signal being a training signal: continuing to buffer the sensor data stream into the circular buffer; copying a first segment of the sensor data stream from the circular buffer to a non-volatile memory of the data recorder; storing a second segment of the sensor data stream in the non-volatile memory, the second segment of the sensor data stream being generated after the trigger signal; transmitting from the non-volatile memory to a remote server; and In response to the trigger signal being an accident signal: Buffering of the sensor data stream into the circular buffer is stopped.

17. The method according to claim 16, further comprising: detecting a mismatch between an identification of an object provided in a map of the road and an identification of the object made by applying the first segment of the sensor data stream in an artificial neural network of the advanced driver assistance system of the autonomous vehicle; generating the trigger signal in response to the mismatch; and A map of the road, the artificial neural network, or both the map and the artificial neural network are updated using the first and second segments of the sensor data stream transmitted from the non-volatile memory to the server.

18. The method according to claim 17, further comprising: generating said sensor data stream by said sensor comprising a digital camera, radar, lidar or ultrasonic sonar or any combination thereof; The copying and the storing are performed in parallel.

19. The method according to claim 18, further comprising: detecting or predicting an accident by the advanced driver assistance system; and In response to the detection or prediction of the incident, data is copied from the circular buffer to a slot cyclically selected from a plurality of slots in the non-volatile memory.

Images