Alternate frame processing operation using predicted frame comparison.

Abstract

Frames from the image stream are processed by independently operating digital signal processors (DSPs) (404A, 404B), with only the frame checking microprocessors (402A, 402B) operating in lockstep mode. In one example, two DSPs operate on alternating frames. Each DSP processes a frame and generates a prediction for the next frame. The lockstep microprocessors (402A, 402B) develop their own next-frame predictions. The lockstep microprocessors compare the issued frame with the previously developed predicted frame for consistency. If these predictions are close enough, the issued frame passes the test. The lockstep microprocessors then compare the issued frame with the two previous frames for a similar consistency check. If the previous frames are also close enough, the issued frame is acceptable.

Description

[Technical field]

[0001] The field relates to fault detection in image processing operations.

[0002] The use of electronic devices in automobiles is increasing day by day. In addition to the traditional engine controller, transmission controller, infotainment unit, body controller, etc., the emergence of numerous safety and autonomous systems has significantly increased the processing inside the automobile. For example, adaptive cruise control uses intercommunication between a radar system, an engine controller, and a transmission controller. As another example, in a bird's-eye view display, the output from several different cameras located at various positions is provided to a processor that processes the received video to develop a bird's-eye view image, which is then provided to the infotainment system for display to the driver. This increase in the number and variety of input sensors places a heavy burden on the system-on-chip (SoC) device that receives the sensor data. In addition, the sensor data is often used by multiple processes, which increases the demands on the SoC device. The reliability requirements of safety systems that use sensor data often require at least duplication of computation blocks, further complicating the burden. Summary of the Invention

[0003] Images or frames from one or more image streams are processed by independently operating digital signal processors (DSPs) and hardware-assisted logic, with only the frame checking microprocessor operating in lock-step mode. In one example, two DSPs are operating on an image stream. Instead of each DSP operating on each frame in lock-step mode, one DSP operates on the even frames and the other operates on the odd frames. Each DSP processes a frame for a given operation as needed and generates a prediction for the next frame. For example, one DSP operates on frame 1 and generates a prediction for frame 2. The lock-step microprocessors develop their own next frame predictions, in this example the frame 2 prediction. When the DSPs complete frame processing and issue the frame and the next frame prediction, the lock-step processors compare the issued frame with the previously developed predicted frame for consistency. If the predictions are close enough, the issued frame passes the test. The lock-step processors then compare the issued frame with the previous two frames to perform a similar consistency check. For example, a lockstep processor may compare frames 0 and 1 with issued frame 2. If the preceding frames are also close enough, the issued frame is acceptable and no error is signaled. If any comparison determines that the frames are too different, an error is signaled and appropriate safety precautions are taken.

[0004] In another example, frame prediction is not used and a hardware checker is provided to compare the current frame with more previous frames, for example, four previous frames. The hardware checker provides the comparison results to the lockstep processor to compare against an acceptable variation limit. If the comparison results exceed the variation limit, an error is provided.

[0005] Because each DSP does not process every frame, but only every other frame, in some cases developing next frame prediction, a significant portion of the DSP bandwidth is freed up to perform other functions in addition to specific image processing tasks. Although each frame is checked in multiple ways against many samples, safety is not reduced at the expense of processing bandwidth. This allows for greater capacity for a given SoC, or fewer SoCs may be used than would otherwise be required. [Brief description of the drawings]

[0006] For a detailed description of the various examples, reference is made to the accompanying drawings.

[0007] [Figure 1] FIG. 2 is a diagram of a vehicle and the field of view of various sensors.

[0008] [Diagram 2] FIG. 2 is a block diagram of a sensor system in the vehicle of FIG. 1.

[0009] [Figure 3A] FIG. 1 is a block diagram of a conventional SoC operating in ASIL D mode.

[0010] [Figure 3B] FIG. 3B illustrates a frame processed by the SoC of FIG. 3A.

[0011] [Figure 4A] This is an example of an SoC that operates in ASIL D mode.

[0012] [Figure 4B] FIG. 4B illustrates a frame processed by the SoC of FIG. 4A.

[0013] [Diagram 5] FIG. 4B is a block diagram of an example for determining a comparison value of the SoC of FIG. 4A.

[0014] [Figure 5A] FIG. 6 is a block diagram of an SoC used in FIGS. 4A and 5.

[0015] [Figure 6] FIG. 6 is a block diagram of an example for determining a comparison value of the SoCs of FIGS. 4A and 5.

[0016] [Figure 7] FIG. 4B is a block diagram of a second example for determining a comparison value for the SoC of FIG. 4A.

[0017] [Figure 8A] 4B is a flowchart illustrating the operation of the SoC of FIG. 4A in accordance with FIG. 4B. [Figure 8B] 4B is a flowchart illustrating the operation of the SoC of FIG. 4A in accordance with FIG. 4B. [Figure 8C] 4B is a flowchart illustrating the operation of the SoC of FIG. 4A in accordance with FIG. 4B. [Figure 8D] 4B is a flowchart illustrating the operation of the SoC of FIG. 4A in accordance with FIG. 4B.

[0018] [Figure 9A] 8 is a flow chart illustrating the operation of the SoC of FIG. 4A in accordance with FIG. 7. [Figure 9B] 8 is a flow chart illustrating the operation of the SoC of FIG. 4A in accordance with FIG. 7. [Figure 9C] 8 is a flow chart illustrating the operation of the SoC of FIG. 4A in accordance with FIG. 7. [Figure 9D] 8 is a flow chart illustrating the operation of the SoC of FIG. 4A in accordance with FIG. 7. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Referring now to FIG. 1, a vehicle 100 is shown. The vehicle 100 includes a series of cameras or optical sensors. A left camera 102 and a right camera 104 provide images from the front of the vehicle 100 for lane departure warning, traffic sign recognition, collision warning, and object detection. A left LIDAR (Light Detection and Ranging) sensor 106 and a right LIDAR sensor 108 provide images from the front of the vehicle 100 for lane and object detection. These cameras and LIDAR sensors provide input to various advanced driver assistance systems (ADAS). The cameras and LIDAR sensors are merely examples, and many other sensors such as radar and ultrasonic could also be used.

[0020] 2, cameras 102 and 104 are connected to a front camera module 202. LIDAR sensors 106 and 108 are connected to a LIDAR module 204. The front camera module 202 and the LIDAR module 204 are connected to a sensor fusion module 210 that integrates the various sensor outputs deployed by the other modules. An autonomous processing module 212 is connected to the sensor fusion module 210 to perform autonomous processing for vehicle operation. More or fewer sensors may be connected to a given module, and multiple sensor types may be provided on a single module.

[0021] In automotive applications, safety is a priority for many of the electrical systems. ISO26262 defines various Automotive Safety Integrity Levels (ASIL). The ASIL level used for a given system is based on severity, probability of exposure, and controllability. Probability of exposure has five classes ranging from very low probability to high probability (E0-E4). Severity has four classes ranging from no fault to life-threatening fault (survivability unknown), fatal fault (S0-S3). Controllability, meaning controllability by the driver, rather than by the vehicle's electronic systems, has four classes ranging from generally controllable to difficult or uncontrollable. These values ​​are combined to give ASIL levels A-D, with A being the lowest level and D being the highest. Collision warning, lane departure, and autonomous driving are generally classified as ASIL D.

[0022] FIG. 3A is an example of an SoC 300 used in a module, such as modules 202, 204, 210, or 212, configured for ASIL D operation. The components within the SoC 300 are broadly categorized as a processor 302, a digital signal processor (DSP) 304, and hardware-assisted logic 306. In the SoC 300, the processor 302 is two microprocessors 302A, 302B, such as ARM processors, configured to operate in lockstep. In lockstep operation, each processor runs identical software and operates on the same input data. Detection hardware monitors differences in the outputs of the microprocessors 302A, 302B and provides an error indication if a difference is detected. Similarly, the DSP 304 includes two DSPs 304A, 304B operating in lockstep. The hardware-assisted logic 306 has identical hardware-assisted logic 306A, 306B operating in lockstep.

[0023] Lockstep operation achieves ASIL D operation, but at the cost of effectively doubling the required silicon area of ​​the SoC, thereby limiting the capabilities of the SoC.

[0024] 3B illustrates that each frame is operated by each device in each block, such as microprocessors 303A, 302B, DSPs 304A, 304B, and hardware-assisted logic 306A, 306B. In FIG. 3B, the top row shows frames operated by device "A" and the bottom row shows frames operated by device "B." This shows that each frame from each image source is operated by each device in the lockstep pair.

[0025] 4A is a block diagram of a first example of a SoC 400 that may be used in modules 202, 204, 210, or 212 to form an ASIL D compliant image processing system. In SoC 400, only processor 402 is formed by lockstep microprocessors 402A, 402B. DSPs D0 404A, D1 404B operate independently. Hardware assist logic 406A, 406B operate independently.

[0026] FIG. 4B illustrates the basic operation of SoC 400. In this example, DSP D0 404A operates only on odd frames, and DSP D1 404B operates only on even frames, so DSP D0 404A and DSP D1 404B operate on different frames. Each DSP D0, D1 404A, 404B predicts the next frame along with whether it is an odd or even frame. Lockstep processor 402 develops its own prediction of the next frame and then compares the prediction made by the DSP and the processor itself with the actual frame, and also compares the actual frame with the two previous frames. For example, as shown in FIG. 4B, for frame 3, lockstep processor 402 uses actual frame 3, previous frames 2 and 1, and a prediction made using frame 2 to determine whether actual frame 3 is within an acceptable deviation of these predictions from the previous frame. Then, for frame 4, lockstep processor 402 evaluates actual frame 4 using actual frame 4, previous frames 3 and 2, and the prediction made using frame 3. If the frame is outside the acceptable deviation, lockstep processor 402 provides an error signal.

[0027] Note the gap between successive frames for each DSP D0, D1 404A, 404B. This gap is slightly less than the time required for the DSP to process a frame. The DSP can use this time for other processing tasks other than the illustrated frame processing, which is done to comply with ASIL D. Compared to FIG. 3B, where there is effectively just a nominal gap or time between frames, in FIG. 4B, this gap is much larger, so that other important processing can be performed by the DSPs D0 404A, D1 404B. By operating as shown in FIG. 4B, almost one DSP's worth of processing is regained compared to FIG. 3B. Depending on the actual capabilities of the DSP, the DSP may perform frame operations in less time than the nearly 100% duty cycle shown in FIG. 3B, and the improvement in FIG. 4B is only slightly less than the time required to process a frame, which is a large improvement in the ability to perform other tasks other than frame processing.

[0028] 5, there is shown DSP D0 504A. In DSP 504A there is a set of selectable control registers 503. DSP D0 504A is connected to a multi-core shared memory controller (MSMC) 510. Connected to MSMC 510 is L3 RAM 512. In L3 RAM 512 is located diagnostic memory D0 552A along with DSP D0 frame buffer 554A, both of which are considered data memory areas in L3 RAM 512, as opposed to program memory areas that store instructions or programs executed by DSP D0 504A and other processors, such as lockstep processors 402A, 402B. Diagnostic memory D0 552A and DSP D0 frame buffer 554A are located in known locations to simplify operation. DSP D0 frame buffer 554A may include additional scratch pad memory used by DSP D0 504A to unpack a particular frame of interest. Snoop and copy block 550A is connected to selected control registers 503 in DSP D0 504A, DSP D0 frame buffer 554A, diagnostic memory D0 552A, and general purpose timer 592. In one example, snoop and copy block 550A is a hardware block that performs read and write operations. In another example, snoop and copy block 550A is a software block where instructions executed by the DSP perform read and write operations. When snoop and copy block 550A is operated, it takes a timestamp from general purpose timer 592 and copies selected control register 503 values ​​from DSP D0 504A and a frame from DSP D0 frame buffer 554A to diagnostic memory 552, thus creating a timestamp snapshot of the operation of DSP D0 504A.

[0029] Two sources trigger the operation of snoop and copy block 550A. The first source is instructions executed in DSP D0 504A, and the second source is general-purpose timer 592. DSP D0 504A may execute instructions to perform frame operations. In one example, additional instructions are provided, which execute in DSP D0 504A to trigger snoop and copy block 550A at the completion of each frame. This is illustrated in FIG. 5 by compiler block 590 connected to snoop and copy block 550A. While the software version of snoop and copy block 550A executes, it sets register bits, etc. to trigger the hardware version of snoop and copy block 550A. As a result, at the completion of each frame processing, the timestamp, control register values, and DSP frame are placed in diagnostic memory D0 552A. By triggering this snoop and copy operation upon the completion of every frame, the alternating frame skew operations of the DSPs are synchronized so that comparison between the contents of the diagnostic memories D0 552A, D1 552B can be made more easily. In addition, a general purpose timer 592 is configured to trigger write events of the hardware version of the snoop and copy block 550A. Preferably, the general purpose timer 592 is configured so that these write events occur at consistent intervals relative to the frame calculations so that the two DSPs can be synchronized. The snoop and copy block 550A captures the general purpose timer 592 timestamp, reads selected control registers 503 and the DSP D0 frame buffer 554A, and all values ​​are written to the diagnostic memory D0 552A. In this way, by capturing the timestamp, frame information, and control registers after every frame processing completion event and upon a synchronous write event, the alternating frame operations of the DSPs are effectively synchronized to simplify the comparison operation.

[0030] 5A is a block diagram of a first example of a SoC 500 that may be used as SoC 400. A series of more powerful microprocessors 501, such as an ARM A72 or A53 core, form the main general-purpose processing block of SoC 500. A lockstep processor 502 with individual microprocessors 502A, 502B, such as an ARM R5F core, is equivalent to lockstep processor 402. Two DSPs, DSP 0 504A and DSP 1 504B, are equivalent to DSPs D0 404A, D1 404B. A simpler microprocessor 506, such as one or more ARM R5F cores, provides general control capabilities in SoC 500. A high-speed interconnect 508 connects microprocessor 501, lockstep processor 502, DSP 0 504A, DSP 1 504B, and microprocessor 506 to various other components in SoC 500. For example, a multi-core shared memory controller 510 including an on-board memory or L3 RAM 512 is connected to the high-speed interconnect 508 and acts as an on-board RAM and controller for the SoC 500. A DDR memory controller system 514 is connected to the high-speed interconnect 508 and acts as an external memory interface to external DRAM memory. Similarly, a video acceleration module 516 and a radar processing accelerator (PAC) module 518 are connected to the high-speed interconnect 508. Two vision processing accelerator modules 520A, 520B are connected to the high-speed interconnect 508 and represent the hardware assisted logic 406A, 406B, although many of these blocks are from the hardware assisted logic, which is a combination of blocks not shown. A depth and motion PAC module 522 is connected to the high-speed interconnect 508. A graphics acceleration module 524 is connected to the high-speed interconnect 508. A display subsystem 526 is connected to the high speed interconnect 508 and includes conversion logic 528 and output logic 530, which allows operation with and connection to a variety of video monitors, where appropriate. A system services block 532, which includes items such as a DMA controller, a memory management unit, general purpose I / O, and a mailbox, is provided for the normal operation of SoC 500.Serial connectivity module 534 is connected to high speed interconnect 508 and includes modules as would typically be found in an SoC. Vehicle connectivity module 536 provides interconnects for external communication interfaces such as PCIe block 538, USB block 540, and Ethernet switch 542. Capture / MIPI module 544 includes a 4-lane CSI-2 compliant transmit block 546 and a 4-lane CSI-2 receive module and hub. Details of the CSI-2 receive module and hub are provided below.

[0031] An MCU island 560 is provided as a secondary subsystem to handle the operation of the integrated SoC 500 when other components are powered off to save energy. An MCU ARM processor 562 acts as a master and is coupled to the high-speed interconnect 508 via an isolation interface 561. An MCU general purpose I / O (GPIO) block 564 acts as a slave. An MCU RAM 566 is provided to act as a local memory for the MCU ARM processor 562. An additional external communication interface, a CAN bus block 568, is connected to enable operation in a traditional CAN bus environment in the vehicle 100. An Ethernet MAC (Media Access Control) block 570 is provided for further connectivity in the vehicle 100. A non-volatile memory (NVM) (not shown) is connected to the MCU ARM processor 562 via an NVRAM interface 571. The MCU ARM processor 562 acts as a safety processor to monitor the operation of the SoC 500 to ensure that it operates properly.

[0032] In one example, hardware versions of snoop and copy blocks 550A, 550B are provided to cooperate with their associated DSP 0 504A, DSP 1 504B. Diagnostic memories D0 552A, D1 552B to cooperate with their associated DSP 0 504A, DSP 1 504B are provided in known locations in L3 RAM 512. DSP 0 and DSP 1 frame buffers 554A, 554B are provided in known locations in L3 RAM 512. Hardware checkers 505A and 505B are provided as an example and are described in more detail below.

[0033] The above is one example SoC provided for purposes of illustration, and many other examples of SoCs are possible, with varying numbers of processors, DSPs, accelerators, etc.

[0034] Figure 6 is a variation of Figure 5, showing DSP D0 504A and DSP D1 504B. A common general-purpose timer 592 is connected to the snoop and copy blocks 550A, 550B in each of DSPs D0 504A, D1 504B. When the snoop and copy blocks 550A, 550B are triggered as shown in Figure 5, control register values, frame data, and timestamps are passed to the respective diagnostic memories D0 552A, D1 552B. General-purpose timer 592 also provides trigger instructions to lockstep processor 502 to cause lockstep processor 502 to read diagnostic memories D0 552A, D1 552B and compare the data in diagnostic memories D0 552A, D1 552B to determine whether the operation of DSPs D0 504A, D1 504B differs by an acceptable amount or by more than a given level, and if so, provides an error interrupt or message.

[0035] In one example, this diagnostic memory comparison operation is triggered immediately after the snoop and copy block completes the transfer operation. In both these diagnostic memory comparisons and the frame and frame prediction operations described above, not all of the data needs to match exactly. Some areas, such as the control register data, should be identical, but the frame data, especially the unrolled scratch memory values ​​that implement the frame operations, can change by a selected amount and still be considered good. "Close enough" in this comparison is based on the fact that in many cases, the vehicle 100 is moving and therefore the images are slightly different. In other cases where the cameras are configured for an oblique or slightly oblique gaze direction, such as image streams from left and right forward looking cameras, when one DSP processes the left image stream and the other DSP processes the right image stream, the images are different but close enough that the comparison and prediction are accurate enough for safety purposes. To account for vehicle movement and gaze direction between frames, some differences in the images are acceptable and do not indicate a component failure requiring ASIL D intervention. If the image is within 5% of various dimensions calculated for the image, such as distance and angle relative to the object's location, the image is deemed acceptable and free of faults. A tolerance below 5% will increase the number of false positives, which will impair operation. Since faults are generally complete and therefore the difference is very large, tolerances above 5% may be used, but lower numbers are preferred since faults are not always complete. 5% has been determined to be a good balance between false positives and false negatives. In one example, the tolerance or acceptance level is determined by developing the moving variance of the image and then setting the 3 sigma value as the acceptance limit.

[0036] In the above example, object distance and orientation angle were used as values ​​to check for acceptable differences. These are example variables, any target variable created for an image can be used so that tolerances can be determined for multiple dimensions. These tolerances for target variables can also be determined statistically by analysis of variance for calculations over a number of prior time cycles.

[0037] FIG. 6 further illustrates the operation 600 of the snoop and copy blocks 550A, 550B. For a given snoop and copy operation, an event indication or handler is developed (602) indicating a trigger source for that snoop and copy operation, including a timestamp of the operation in some examples. Control register information is read (604). This is followed by reading a data packet of the respective frame information (606). Finally, these developed and read values ​​are written to the diagnostic memory (608). Preferably, a circular buffer pointer is incremented so that multiple snapshots can be stored sequentially in the diagnostic memory. Since the size of each snapshot in the diagnostic memory is known, this allows the hardware checker 505A, 505B or the lockstep processor 502 to easily find the relevant frame through the respective diagnostic memories 552A, 552B for the two different DSPs.

[0038] In one example, the hardware snoop and copy blocks 550A, 550B are effectively direct memory access (DMA) blocks configured to collect control register information from DSPs D0 504A, D1 504B and move it to diagnostic memory, and copy frame information to diagnostic memory, whose location is advanced with each operation using a circular buffer pointer.

[0039] In one example, the software versions of snoop and copy blocks 550A, 550B are interrupt driven tasks of each DSP D0 504A, D1 504B. In the interrupt routine, the DSP reads the desired control register information from the stack, reads the image information from the frame buffer, and writes both to diagnostic memory.

[0040] As shown in FIG. 7, preferably, the diagnostic memories D0 552A, D1 552B store frames 1, 3, 5 or 2, 4, 6 from both DSP 1 504B and DSP 0 504A, i.e., frames T N , TN-1 , and T N-2 5, etc., and includes data snapshots from various frames, such as frames 1-5, 2-6, 3-8, 4-9, 5-10, 5-11, 5-12, 5-13, 5-14, 5-15, 5-16, 5-17, 5-18, 5-19, 6-20, 6-21, 6-22, 6-23, 6-24, 6-25, 6-26, 6-27, 6-28, 6-29, 7-30, 7-31, 7-32, 7-33, 7-34, 7-35, 7-40, 7-41, 7-42, 7-43, 7-44, 7-45, 7-50, 7-51, 7-52, 7-65, 7-66, 7-77, 7-80, 7-90, 7-10, 7-11, 7-12, 7-13, 7-14, 7-15, 7-16, 7-17, 7-18, 7-19, 7-22, 7-23, 7-24, 7-25, 7-31, 7-44, 7-51, 7-14, 7-15, 7-25, 7-16, 7-26, 7-32, 7-45, 7-52, 7-17, 7-18, 7-27, 7-33, 7-45, 7-51, 7-19, 7-28, 7-34, 7-45, 7-52, 7-19, 7-29, 7-35

[0041] Because the snapshots are provided to known locations in diagnostic memories D0 552A, D1 552B, which are themselves in known locations, the checking of each frame can be done by hardware checkers 505A, 505B, reducing the load on lockstep processor 502. In one example, hardware checkers 505A, 505B include a simple ALU that reads the associated frame and register values ​​from diagnostic memory 552 and compares those values. For example, if frame 6 is T N Then, the hardware checker 0 505A detects frame 5 (D1_T N-2 ), 3(D1_T N-1 ), 4(D0_T N-1 ), and 2(D0_T N-2), and compares frame 6 to each of these frames. The results of the comparisons are provided to lockstep processor 502, which then checks the results against limits of acceptable differences. If the comparisons indicate differences that exceed the acceptable level, lockstep processor 502 provides an error indication. The use of hardware checkers 505A, 505B reduces the processing required of lockstep processor 502. The comparisons made by hardware checkers 505A, 505B do not include predicted values ​​as do the software checks made by lockstep processor 502, but the use of additional frames provides sufficient data to substitute for predicted values.

[0042] 8A-8D, a flow chart of the operation of DSP0, DSP1, and a lockstep processor without a hardware checker will now be described with approximate time relationships. In step 802, DSP0 processes frame 1 and develops a prediction for frame 2. At the same time, DSP1 in step 804 performs other non-frame processing tasks. In step 806, shown shortly before DSP0 completes the frame processing in step 802, the lockstep processor reads DSP1's frame 1 prediction from diagnostic memory and develops its own frame 1 prediction in step 808. In step 810, DSP0 issues a prediction for frame 1 and frame 2. This issuing of the frame 1 and frame 2 predictions involves the snoop and copy block copying the frames to diagnostic memory, as well as the values ​​of the DSP registers if they are stored therein. Any such issuing of frames will involve snoop and copy block operations, which will be omitted below for clarity. After frame 1 is issued, the lockstep processor reads frame 1 from the diagnostic memory in step 812 and compares DSP1 and the lockstep processor's frame 1 prediction to the actual frame 1 in step 814. At this time, DSP0 has proceeded with other non-frame processing tasks in step 816. Also at this time, DSP1 has begun processing frame 2 and developed a prediction for frame 3 in step 818. After the comparison in step 814, the lockstep processor evaluates or determines whether the comparison indicates that the actual frame 1 was close enough to the predicted frame 1 in step 820. If the comparison does not indicate that it was close enough, an error interrupt is issued in step 822 so that higher level processing can evaluate for ASIL D concerns. If the comparison was successful, then in step 824 the lockstep processor compares the previous two frames, frame 0 and frame -1, to frame 1. If the frames do not fall within the acceptable limits of difference as in step 826, an error interrupt is issued in step 828.

[0043] If the comparison is successful, DSP0's frame 2 prediction is read in step 829, and then the lockstep processor develops the frame 2 prediction in step 830. At this time, DSP1 is issuing frame 2 and frame 3 predictions in step 832. The lockstep processor reads frame 2 in step 834 and compares DSP0 and the lockstep processor's frame 2 predictions to the actual frame 2 in step 836. If the difference is beyond the acceptable limits as determined in step 838, an error interrupt is issued in step 840. If the difference is within the acceptable limits, operation proceeds to step 842, where frames 0 and 1 are compared to frame 2. At this time, DSP0 has begun processing frame 3 and is developing a prediction for frame 4 in step 844. DSP1 is busy with other processing tasks in step 846. If it is determined in step 848 that the three-frame comparison is not acceptable, an error is issued in step 850. If so, in step 851, the lockstep processor reads DSP1's frame 3 prediction that was issued in step 832. In step 852, DSP0 issues its frame 3 and frame 4 predictions. In step 854, the lockstep processor develops its frame prediction for frame 3. DSP0 begins performing other tasks in step 856, and the lockstep processor reads frame 3 in step 858.

[0044] In step 860, the lockstep processor compares DSP1 and the lockstep processor's frame 3 predictions to the actual frame 3. If outside acceptable limits as determined in step 862, an error interrupt is issued in step 864. If acceptable, the lockstep processor compares frames 1, 2, and 3 in step 866. DSP1 is now processing frame 4 and developing a prediction for frame 5 in step 868. The lockstep processor determines whether the 3-frame comparison was acceptable in step 870. If not, an error interrupt is issued in step 871. If acceptable, the lockstep processor reads DSP0's frame 4 prediction in step 872 and the lockstep processor performs its own frame 4 prediction in step 874. DSP1 now issues frame 4 and frame 5 predictions in step 876. This allows the lockstep processor to read frame 4 in step 878. DSP1 proceeds to other tasks other than frame processing in step 880. In step 882, the lockstep processors compare the frame 4 predictions of DSP0 and the lockstep processor to the actual frame 4. In step 884, DSP0 begins processing frame 5 and develops a prediction for frame 6. If the comparison in step 882 indicates an unacceptable difference as determined in step 886, then an error interrupt is issued in step 888.

[0045] If the differences are within acceptable limits in step 886, then the lockstep processor compares frames 2 and 3 to frame 4 in step 890. If the frames are too different as determined in step 892, then an error interrupt is issued in step 894. If the frames are close enough, then the lockstep processor reads DSP1's frame 5 prediction in step 896 and develops its own frame 5 prediction in step 898. At this time, DSP0 has completed its frame processing and issues frame 5 and frame 6 predictions in step 900. The lockstep processor reads frame 5 in step 902 at essentially the same time that DSP0 proceeds to other tasks in step 904. The lockstep processor compares DSP1's and ARM's frame 5 predictions to the actual frame 5 in step 906. At approximately this time, DSP1 has begun processing frame 6 and developed a prediction for frame 7 in step 908. In step 910, the lockstep processor determines if the comparison in step 906 was unacceptable. If not, an error interrupt is issued in step 912. If acceptable, in step 914 the lockstep processor compares frames 3, 4, and 5. In step 916, the lockstep processor determines if the error was not acceptable, in which case an error interrupt is issued in step 918.

[0046] Frame processing continues in the same manner until it is no longer necessary.

[0047] 8A-8D, the DSP may spend a significant amount of time performing other tasks rather than performing lockstep frame processing, thus allowing the SoC to increase performance for a given DSP, or to use a less powerful DSP while still maintaining ASIL D levels of safety.

[0048] In the above description, frame prediction is implemented in both the DSP and the lockstep processor, but in one example, only one of the DSP and the lockstep processor develops frame prediction, thereby freeing up some bandwidth in either the lockstep processor or the DSP.

[0049] 9A-9D are flow charts illustrating the operation of DSP0, DSP1, the lockstep processor, and the hardware checker in approximate time relationships. In step 930, DSP0 processes frame 1. In step 932, the hardware checker has previously read DSP1 frames 0, -2, and DSP0 frames -1, -3. Concurrently with step 930, DSP1 performs other non-frame processing tasks in step 934. In step 936, DSP0 issues frame 1. After frame 1 is issued, in step 938, based on general-purpose timer 592, the hardware checker reads frame 1, and in step 940, DSP0 frame 1 is compared with DSP1 frames 0, -2, and DSP0 frames -1, -3. At this time, in step 942, DSP0 proceeds to other non-frame processing tasks. Also, simultaneously, in step 944, DSP1 starts processing frame 2. After the comparison in step 940, the hardware checker provides the comparison result to the lockstep processor in step 946. In step 948, the lockstep processor evaluates or checks the comparison result against allowed deviation limits. In step 950, the lockstep processor determines if the comparison result is within acceptable limits. If not, an error interrupt is issued in step 952 so that higher level processing can evaluate for ASIL D concerns. If the comparison was successful, the operation ends in the lockstep processor until the next comparison result is received from the hardware checker.

[0050] DSP1 finishes processing frame 2, issues frame 2 in step 954, and proceeds to other tasks in step 956. After frame 2 is issued, the hardware checker reads frame 2 in step 958 and compares DSP1 frame 2 with DSP1 frames 0, -2 and DSP0 frames 1, -1 in step 960. At this time, DSP0 has started processing frame 3 in step 962. After the comparison in step 960, the hardware checker provides the comparison result to the lockstep processor in step 964. In step 966, the lockstep processor checks the comparison result for allowed deviation limits. In step 968, the lockstep processor determines if the comparison result is within acceptable limits. If not, an error interrupt is issued in step 970 so that higher level processing can evaluate for ASIL D concerns. If the comparison was successful, the operation ends in the lockstep processor until the next comparison result is received from the hardware checker.

[0051] In step 972, DSP0 issues frame 3 and proceeds to other tasks in step 974. After frame 3 is issued, in step 976, the hardware checker reads frame 3 and compares DSP0 frame 3 with DSP1 frames 2, 0 and DSP0 frames 1, -1 in step 978. At this time, DSP1 has started processing frame 4 in step 980. After the comparison in step 978, the hardware checker provides the comparison result to the lockstep processor in step 982. In step 984, the lockstep processor checks the comparison result against allowed deviation limits. In step 986, the lockstep processor determines whether the comparison result is within acceptable limits. If not, an error interrupt is issued in step 988 so that higher level processing can evaluate for ASIL D concerns. If the comparison was successful, the operation ends in the lockstep processor until the next comparison result is received from the hardware checker.

[0052] DSP1 issues frame 4 in step 990 and proceeds to other tasks in step 991. After frame 4 is issued, the hardware checker reads frame 4 in step 992 and compares DSP1 frame 4 with DSP1 frames 2, 0 and DSP0 frames 3, 1 in step 994. At this time, DSP0 has started processing frame 5 in step 996. After the comparison in step 994, the hardware checker provides the comparison result to the lockstep processor in step 998. In step 1000, the lockstep processor checks the comparison result against the allowed deviation limits. In step 1002, the lockstep processor determines if the comparison result is within acceptable limits. If not, an error interrupt is issued in step 1004 so that higher level processing can evaluate for ASIL D concerns. If the comparison was successful, the operation ends in the lockstep processor until the next comparison result is received from the hardware checker.

[0053] In step 1006, DSP0 issues frame 5 and proceeds to other tasks in step 1008. After frame 5 is issued, the hardware checker reads frame 5 in step 1010 and compares DSP0 frame 5 with DSP1 frames 4, 2 and DSP0 frames 3, 1 in step 1012. At this time, DSP1 has started processing frame 6 in step 1014. After the comparison in step 1012, the hardware checker provides the comparison result to the lockstep processor in step 1016. In step 1018, the lockstep processor checks the comparison result against acceptable deviation limits. In step 1020, the lockstep processor determines whether the comparison result is within acceptable limits. If not, an error interrupt is issued in step 1022 so that higher level processing can evaluate for ASIL D concerns. If the comparison was successful, the operation ends in the lockstep processor until the next comparison result is received from the hardware checker.

[0054] Frame processing continues in the same manner until completed.

[0055] 9A-9D, like FIGs. 8A-8D, illustrate that the DSP may spend a significant amount of time performing other tasks rather than performing lockstep frame processing. The load on the lockstep processor may also be lightened, allowing it more time to perform other tasks. This may increase the performance of the image processing system of the SoC for a given DSP and lockstep processor, or a less powerful DSP and lockstep processor may be used while still maintaining ASIL D levels of safety.

[0056] In the descriptions of Figures 8A-8D and 9A-9D, a comparison of a frame and a predicted frame is described, but if the example also stores DSP register values ​​for each frame, then this comparison includes a comparison of DSP register values.

[0057] As mentioned above, various factors are used to determine whether a processed frame is acceptable or if a reportable error has occurred. These factors include frame prediction by the DSP and lockstep processor, pre-processed frame, and DSP register values. All of these factors are considered alternate frame information because they indicate what the data for the next frame should be close to if the next frame is error-free. The frame prediction and pre-processed frame are alternate frame versions, and the DSP register values ​​are alternate information about the frame, information related to the evolution of the frame, or alternate frame versions. Thus, alternate frame information encompasses all three items: frame prediction, pre-processed frame, and DSP register values.

[0058] In the above description, we have focused on the description of the operation with a single image stream, and have described different DSPs handling even and odd frames. When two related image streams are being processed and these image streams are sufficiently related or overlapping, in one example, each DSP can process a separate image stream, each DSP processes each image in its assigned stream, so that the DSPs again work on different frames and not on the same frames. For example, in the case of a left-right pair of image streams, one DSP processes the right stream and one DSP processes the left stream, but the comparison and, if a prediction is made, the prediction is made for both streams. In another example, in the case of such two image streams, one DSP can process the even images from each image stream, and the other DSP can process the odd images from each image stream. In this example, when a prediction is made, the prediction and the comparison are made for frames of the same image stream and not for frames of the other image stream. So a first DSP may process frame 1 of image stream 1 and predict frame 2, then a first DSP may process frame 1 of image stream 2 and predict frame 2.

[0059] In the above description, the DSP includes register values ​​in the comparison operations for additional information to detect error conditions. In some instances, the register values ​​are not used and the snoop and copy block simply copies the frame data to the diagnostic memory.

[0060] In the above discussion and in the flow charts of Figures 8A-8D and 9A-9D, it has been shown that an error is indicated for a single frame that differs by more than an acceptable range. In one design, instead of using a single frame, three consecutive failed frames are used. If the first frame fails, it is discarded and the next frame is compared to the criteria used for the first frame. If this second frame also fails, it is also discarded and the next frame is compared. If this third frame also fails, an error indication is provided. This provides a simple filtering action to prevent a single transient error from causing an error condition.

[0061] While the above description has focused on image processing, other streams may be processed similarly. For example, radar signal processing, lidar signal processing, and object level sensor fusion processing may all be done similarly and achieve similar improvements in available DSP processing bandwidth. The tolerance determination is based on the target variables developed for the relevant streams.

[0062] The above description is illustrative, and not limiting. For example, the above examples may be used in combination with each other. Many other examples will become apparent upon review of the above description. The scope of the claims should therefore be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. In the appended claims, the terms "including" and "in which" are used as the plain-English equivalents of the terms "comprising" and "wherein," respectively.

Claims

1. 1. A system comprising: a first processor configured to provide a first prediction associated with a first frame and a second frame based on the first frame; a second processor configured to provide the second frame; a third processor coupled to the first processor and the second processor, comparing the second frame from the second processor to the first prediction associated with the second frame from the first processor to determine a first difference associated with the second frame; determining whether the first difference exceeds a threshold; determining whether to assert a first error based on whether the first difference exceeds the threshold; the third processor; Including, the system.

2. 2. The system of claim 1, the first processor is further configured to provide a set of first frames including the first frame; the second processor is further configured to provide a second set of frames including the second frame; The system, wherein the first set of frames excludes every frame in the second set of frames, and the second set of frames excludes every frame in the first set of frames.

3. 3. The system of claim 2, The system, wherein the first set of frames and the second set of frames are alternating frames of an image stream.

4. 2. The system of claim 1, a fourth processor coupled to the first processor and the second processor, contemporaneously with a comparison by the third processor of the second frame with the first prediction associated with the second frame, comparing the second frame from the second processor with the first prediction associated with the second frame from the first processor to determine a second difference associated with the second frame; determining whether the second difference exceeds the threshold value concurrently with the third processor determining whether the first difference exceeds the threshold value; determining whether to assert a second error based on whether the second difference exceeds the threshold value concurrently with the third processor determining whether to assert the first error. The system further comprising the fourth processor configured to:

5. 2. The system of claim 1, The third processor: determining a second prediction associated with the second frame; comparing the second frame from the second processor to the second prediction associated with the second frame; determining whether to assert the first error based on a comparison of the second frame to the second prediction associated with the second frame; The system further comprises:

6. 2. The system of claim 1, The third processor: comparing the second frame from the second processor with the first frame from the first processor; determining whether to assert a second error based on a comparison of the second frame to the first frame; The system further comprises:

7. 7. The system of claim 6, the second processor is further configured to provide a third frame that immediately precedes the first frame; The third processor: comparing the second frame from the second processor with the third frame from the second processor; determining whether to assert the second error further based on a comparison of the second frame to the third frame; The system further comprises:

8. 2. The system of claim 1, a memory coupled to the first processor, the second processor, and the third processor, receiving the first frame from the first processor; receiving the second frame from the second processor; providing the first frame and the second frame to the third processor; The system further comprises the memory configured to:

9. 9. The system of claim 8, a circuit block coupled to the first processor, the second processor, and the memory, causing the first frame to be provided by the first processor and received by the memory; storing the first frame in the memory with a first time stamp; causing the second frame to be provided by the second processor and received by the memory; storing the second frame in the memory with a second time stamp; The system further comprises the circuit block configured to:

10. 10. The system of claim 9, further comprising a timer circuit coupled to the circuit block; the circuit block responds to the timer circuit by causing the first frame to be provided by the first processor and received by the memory; storing the first frame in the memory with a first time stamp; causing the second frame to be provided by the second processor and received by the memory; storing the second frame in the memory with a second time stamp; The system is further configured to execute.

11. 10. The system of claim 9, the first processor includes a first set of control registers configured to store a first set of values ​​associated with the first frame; the second processor includes a second set of control registers configured to store a second set of values ​​associated with the second frame; The circuit block comprises: storing said first frame in said memory together with said first set of values; storing said second frame in said memory together with said second set of values; The system further comprises:

12. 2. The system of claim 1, the first processor is further configured to perform a first non-frame processing task simultaneously with the second processor providing the second frame; The system, wherein the second processor is further configured to perform a second non-frame processing task simultaneously with the first processor providing the first frame.

13. 1. A system comprising: a first digital signal processor (DSP) configured to provide a first prediction relating to a first frame and a second frame based on the first frame; a second digital signal processor (DSP) configured to provide the second frame; a plurality of processors, each coupled to the first DSP and the second DSP, each configured to: comparing the second frame from the second DSP to the first prediction associated with the second frame from the first DSP; determining whether to assert a first error based on a comparison of the second frame to the first prediction associated with the second frame; the plurality of processors configured to Including, the system.

14. 14. The system of claim 13, the first DSP is further configured to provide a set of odd frames that includes the first frame without providing any frame of the set of even frames; The system, wherein the second DSP is further configured to provide the set of even frames that includes the second frame without providing any frame of the set of odd frames.

15. 14. The system of claim 13, each of said plurality of processors, simultaneously with the remainder of said plurality of processors; determining a respective second prediction associated with the second frame; comparing the second frame from the second DSP to the respective second prediction; determining whether to assert the first error based on a comparison of the second frame to the respective second prediction; The system further comprises:

16. 14. The system of claim 13, each of said plurality of processors, simultaneously with the remainder of said plurality of processors; comparing the second frame from the second DSP with the first frame from the first DSP; determining whether to assert a second error based on a comparison of the second frame to the first frame; The system further comprises:

17. 14. The system of claim 13, the second DSP is further configured to provide a third frame immediately preceding the first frame; each of said plurality of processors, simultaneously with the remainder of said plurality of processors; comparing the second frame from the second DSP to the third frame from the second DSP; determining whether to assert a second error further based on a comparison of the second frame to the third frame; The system further comprises:

18. 14. The system of claim 13, a memory coupled to the first DSP, the second DSP and the plurality of processors; a circuit block coupled to the first DSP, the second DSP, and the memory, having the first frame provided by the first DSP and received by the memory; storing the first frame in the memory with a first time stamp; having the second frame provided by the second DSP and received by the memory; storing the second frame in the memory with a second time stamp; The circuit block is configured as follows: The system further comprises:

19. 20. The system of claim 18, further comprising a timer circuit coupled to the circuit block; the circuit block responds to the timer circuit by having the first frame provided by the first DSP and received by the memory; storing the first frame in the memory with a first time stamp; having the second frame provided by the second DSP and received by the memory; storing the second frame in the memory with a second time stamp; The system is further configured to execute.

20. 20. The system of claim 18, the first DSP includes a first set of control registers configured to store a first set of values ​​associated with the first frame; the second DSP includes a second set of control registers configured to store a second set of values ​​associated with the second frame; The circuit block comprises: storing said first frame in said memory together with said first set of values; storing said second frame in said memory together with said second set of values; The system further comprises:

Images