Rearview using rear view camera

Abstract

The application discloses looking around using a rear view camera, and relates to a circuit comprising a first processor core (106), a second processor core (104), and a video processing circuit. The first processor core (106) is configured to program the video processing circuitry to provide a rear view image processing path (202) and pass rear view images from the rear view image processing path (202) to the surround view image processing path (302). The second processor core (104) is configured to program the video processing circuitry to provide a look-around image processing path (302), receive a rear view image from the rear view image processing path (202), and provide a look-around image based on the rear view image.

Description

Background Art

[0001] Advanced driver assistance systems (ADAS) provide features such as rearview cameras, electronic stability control, and vision-based pedestrian detection systems. Many of these systems rely on computer vision processing to detect objects in the field of view of one or more cameras. Surround view camera systems are an ADAS technology that allows the driver to see a 360-degree overhead view of the vehicle's surroundings. Such systems typically include four to six wide-angle cameras mounted around the vehicle, with each camera facing a different direction. Based on these camera inputs, a composite view of the vehicle's surroundings is synthesized and displayed in real time. Summary of the invention

[0002] In one example, a circuit includes a first processor core, a second processor core, and a video processing circuit. The first processor core is configured to program the video processing circuit to provide a rear view image processing path and transfer a rear view image from the rear view image processing path to a surround view image processing path. The second processor core is configured to program the video processing circuit to provide a surround view image processing path, receive a rear view image from the rear view image processing path, and provide a surround view image based on the rear view image.

[0003] In another example, a method includes programming a video processing circuit to provide a rear view image processing path, and generating a rear view image in the rear view image processing path. The method also includes programming the video processing circuit to provide a surround view image processing path, and transferring the rear view image from the rear view image processing path to the surround view image processing path. The method also includes generating a surround view image based on the rear view image in the surround view image processing path.

[0004] In another example, a system includes a camera interface, a video processing circuit, a first processor core, a second processor core, and a display interface. The camera interface is configured to receive a rear-view video signal, a first side-view video signal, a second side-view video signal, and a front-view video signal. The video processing circuit is coupled to the camera interface. The first processor core is coupled to the video processing circuit. The first processor core is configured to program the video processing circuit to provide a rear-view image processing path, provide a rear-view image based on the rear-view video signal, and transfer the rear-view image from the rear-view image processing path to the surround-view image processing path. The second processor core is coupled to the video processing circuit, and the second processor core is configured to program the video processing circuit to provide a surround-view image processing path, receive a rear-view image from the rear-view image processing path, and provide a surround-view image based on the rear-view image. The display interface is coupled to the video processing circuit, and the display interface is configured to provide a rear-view image and a surround-view image to a video display device. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1is an example of a video signal flow in a system using rear view camera images in a surround view.

[0006] Figure 2 is a block diagram of example processing of rear view camera images before initializing surround view processing.

[0007] Figure 3 Is used for Figure 2 A block diagram of example processing of videos from multiple cameras to generate a surround view.

[0008] Figure 4 is suitable for implementation Figure 2 and Figure 3 Block diagram of example processing circuits for rear-view processing and surround-view processing.

[0009] Figure 5 is a flow chart of an example method for providing a surround view image based on an image provided by a rearview image processing path. DETAILED DESCRIPTION

[0010] Figure 1 1 is an example of a video signal stream in a system 100 that uses a rear view camera image in a surround view. The system 100 can be implemented in a vehicle (such as a car) to provide a display of a rear view image and a surround view image. The system 100 includes a video processing subsystem 102, cameras 110, 112, 114, and 116, and a display device 108. The video processing subsystem 102 is coupled to cameras 110, 112, 114, and 116, and a display device 108. Camera 110 can be a rear view camera positioned to capture images from the rear of the vehicle. Camera 112 can be a side view camera positioned to capture images from a first side (e.g., driver's side) of the vehicle. Camera 114 can be a side view camera positioned to capture images from a second side (e.g., passenger side) of the vehicle. Camera 116 can be a front view camera positioned to capture images from the front of the vehicle. Cameras 110, 112, 114, and 116 can include lenses, image sensors, video transmission circuit systems, and the like. The display device 108 displays the image received from the video processing subsystem 102. The display device 108 may include a display panel (such as a liquid crystal display panel, an organic light emitting diode display panel, etc.), a display driver circuit system, and a video receiver circuit system. The video processing subsystem 102 may also or alternatively transmit the image to a remote device via Ethernet or other networking means for display and / or analysis.

[0011] Video processing subsystem 102 includes processor cores 104 and 106. Processor core 104 may be configured to execute a general-purpose operating system that provides many functions, but is relatively slow to initialize (e.g., boot up). Processor core 106 may be configured to execute a "lightweight" operating system (e.g., a real-time operating system) that initializes relatively quickly. For example, processor core 106 may boot up and begin application execution in less than a second, whereas processor core 104 may take several seconds to initialize.

[0012] When processor core 106 has been initialized, and when processor core 104 is initializing, processor core 106 configures video processing subsystem 102 to receive video from cameras 110, 112, 114, and 116, generate a rear view image, and provide the rear view image to display device 108. In some examples, processor core 106 may be initialized and provide a rear view image to display device 108 within approximately 2 seconds after initialization of processor core 106 begins (e.g., approximately 2 seconds after power-up or reset). When configuring video processing subsystem 102, processor core 106 may activate rear view camera capture module 118 to receive video from camera 110, and activate sensor capture module 124 to receive video from cameras 112, 114, and 116. Processor core 106 may also activate split / copy module 120. The split / replicate module 120 receives the rear view camera video provided by the camera capture module 118 and replicates the received video stream to generate a rear view video stream 122 , which is transmitted to the display device 108 for display, and another video stream, which is provided to the surround view module 126 .

[0013] When the processor core 104 has been initialized, which may be a considerable time (e.g., a few seconds) after the processor core 106 has completed initialization and configured the video processing subsystem 102 for rear-view display, the processor core 104 configures the video processing subsystem 102 to process the video streams provided by the segmentation / replication module 120 and the sensor capture module 124, and generate a surround view image. The configuration of the video processing subsystem 102 by the processor core 104 includes activating the surround view module 126. The surround view module 126 receives the video streams from the segmentation / replication module 120 and the sensor capture module 124, and processes the video streams (e.g., stitching together the rear image, the front image, and the side image) to generate a surround view or a "bird's eye view". The surround view module 126 can operate in parallel with the generation of the rear-view video stream 122.

[0014] In some embodiments of rear view image processing and surround view image processing, as described above, rear view image processing is initialized before surround view processing, but rear view image processing is disabled before surround view processing is activated, which increases the delay and complexity of the transition from rear view to surround view. In system 100, by duplicating the rear view video stream, the rear view video display provides a small delay (e.g., less than 2 seconds), and the complexity of switching from rear view to surround view is avoided by providing rear view and surround view in parallel. Other embodiments of rear view and surround view use a first rear view camera for rear view and a second rear view camera for surround view to allow fast switching between views. System 100 allows fast switching between rear view and surround view without using two rear view cameras.

[0015] Figure 2 2 is a block diagram of an example processing 200 of a rear view camera image prior to initializing a surround view process. Figure 2 , the processor core 104 is initializing (in the boot process), and the initialization of the processor core 106 is completed. The processor core 106 configures the rear view image processing path 202, and configures the cameras 110, 112, 114, and 116 to stream the video to the rear view image processing path 202. The rear view image processing path 202 includes a capture node 204, an image signal processing (ISP) node 208, a lens distortion correction (LDC) node 212, and a display node 216. The capture node 204 receives the video stream from the cameras 110, 112, 114, and 116, and outputs an image 206 (e.g., a rear view image, a front view image, a first side view image, and a second side view image). The image signal processing node 208 receives the image 206 and processes the received image (e.g., converting the RAW image sensor data into a processed YUV or RGB image) to generate a processed image 210. Based on the processed image 210, the rear view image is provided to the lens distortion correction node 212. The lens distortion correction node 212 processes the rearward-view image to correct for distortion introduced by the lens of the camera 110 to produce the rearward-view image 214. The display node 216 may transmit the rearward-view image 214 to the display device 108 for display.

[0016] Figure 3 2 is a block diagram of an example process 300 for generating a surround view of a video from multiple cameras using a rear view image generated in a rear view image processing path 202. In the process 300, the rear view image processing path 202 is as described with respect to Figure 2The processor core 104 is initially initialized and the processor core 104 configures the surround view image processing path 302. The surround view image processing path 302 includes a surround view node 304. The surround view node 304 receives the processed images 210 from the rear view image processing path 202 and stitches the images together to create a panoramic surround view (SRV) image 306. The surround view image 306 shows a 360° view of the surrounding environment of the vehicle. The stitching process can align the distortion corrected images and blend the images together.

[0017] The display node 216 receives the surround view image 306 from the surround view image processing path 302. The display node 216 may display the rear view image 214 and / or the surround view image 306 as needed based on the vehicle operation. For example, if the vehicle is in reverse, the display node 216 may transmit the rear view image 214 to the display device 108 for display, and in other vehicle operation scenarios, the display node 216 may transmit the surround view image 306 to the display device 108 for display.

[0018] Figure 4 4 is a block diagram of an example processing circuit 400 suitable for implementing process 200 and process 300. An example of processing circuit 400 may be provided in a system on chip suitable for automotive video processing. Processing circuit 400 includes processor 402, camera interface 404, vision pre-processing accelerator (VPAC) 406, graphics processor (GPU) 408, display interface 410, and memory 418. Processor 402, camera interface 404, VPAC 406, GPU 408, display interface 410, and memory 418 are coupled via interconnect memory 420. Processor 402 may include real-time operating system (RTOS) core 412, high-level operating system (HLOS) core 414, digital signal processor (DSP) core 416, and / or other processor cores. RTOS core 412 includes a processor core selected to execute a real-time operating system. HLOS core 414 includes a processor core selected to execute a high-level (e.g., general-purpose) operating system (such as LINUX). RTOS core 412 may be used to implement processor core 106. One or more of the HLOS cores 414 may be used to implement the processor core 104. The RTOS core 412 and the HLOS core 414 may be implemented using the same or different processor architectures.

[0019] Camera interface 404 includes circuitry for interfacing a camera to processing circuit 400. For example, camera interface 404 may include a multiple mobile industry processor interface camera serial interface (MIPI-CSI) for interfacing processing circuit 400 to cameras 110, 112, 114, and 116. Camera interface 404 may be used to implement capture node 204 to receive video streams transmitted by cameras 110, 112, 114, and 116.

[0020] VPAC 406 is a hardware accelerator that can be configured to accelerate image processing tasks such as color processing and enhancement, noise filtering, wide dynamic range processing, lens distortion correction, pixel remapping for denoising, real-time scale generation, real-time pyramid generation, etc. VPAC 406 can be used to implement image signal processing node 208 and / or lens distortion correction node 212.

[0021] The GPU 408 can be programmed to accelerate three-dimensional (3D) and two-dimensional (2D) graphics and computing applications. The GPU 408 can be used to implement the look around node 304.

[0022] The display interface 410 is a display subsystem that supports one or more high-resolution display outputs. It may include a display controller (DISPC) and a frame buffer decompression core (FBDC). The DISPC may support multi-layer blending and transparency for each of its display outputs. The DISPC may also support a write-back pipeline with scaling to achieve a combination of memory to memory and / or capture display outputs for Ethernet video encoding. The display interface 410 may include a MIPI digital serial interface (DSI) controller and / or a transmitter host controller that complies with the Video Electronics Standards Association Display Port 1 (VESA DP1) for transmitting video to the display device 108. The display interface 410 may be used to implement the display node 216.

[0023] The various video handling circuits of processing circuitry 400 may be referred to individually or collectively as video processing circuitry. For example, camera interface 404, VPAC 406, GPU 408, and / or display interface 410 may be referred to as video processing circuitry.

[0024] The memory 418 may include semiconductor memory, including volatile or non-volatile memory, static or dynamic random access memory, or other types of non-transitory computer-readable media for storing programming and data used or provided by the processor 402, the camera interface 404, the VPAC 406, the GPU 408, and / or the display interface 410. The memory 418 may provide storage for the images 206, the processed images 210, the rear view images 214, and the surround view images 306, and provide storage for instructions executed by the processor 402, the VPAC 406, the GPU 408, the camera interface 404, and / or the display interface 410.

[0025] Figure 5 Flowchart of an example method 500 for providing surround view images based on images provided by a rear view processing stream. Although depicted sequentially for convenience, at least some of the actions shown may be performed in a different order and / or in parallel. In addition, some embodiments may perform only some of the operations shown. The operations of method 500 may be performed by processing circuit 400, as illustrated in process 200 and process 300.

[0026] In block 502, one of the RTOS cores 412 is initialized (started) to operate as the processor core 106. The RTOS core 412 may start relatively quickly (eg, less than a second) after power-up or reset.

[0027] In step 504, the startup of the processor core 106 is completed, and the processor core 106 initializes the rear view image processing path 202. For example, the RTOS core 412 can configure the camera interface 404 to operate as the capture node 204, configure the VPAC 406 to operate as the image signal processing node 208 and the lens distortion correction node 212, and configure the display interface 410 to operate as the display node 216.

[0028] In block 506, camera interface 404 captures images transmitted by cameras 110, 112, 114, and 116. Camera interface 404 may store the received video in memory 418.

[0029] In block 508, the image captured in block 506 is processed. For example, VPAC 406 may process the captured image to convert RAW image data received from cameras 110, 112, 114, and 116 into a YUV or RGB image.

[0030] In block 510, lens distortion correction is applied to the rearview image generated in block 508. For example, VPAC 406 may process the rearview image to provide lens distortion correction.

[0031] In block 512, the rear view image is displayed. For example, the display interface 410 may transmit the rear view image generated in block 510 to the display device 108 for display. The time required to perform the operations of blocks 502-512 may be two seconds or less.

[0032] In block 514, one of the HLOS cores 414 is initialized (started) to run as the processor core 104. The HLOS core 414 may start relatively slowly (e.g., more than two seconds) after power-up or reset. The initialization of the HLOS core 414 may begin approximately at the same time as the initialization of the RTOS core 412.

[0033] In block 516, initialization of the processor core 104 is completed, and the processor core 104 initializes the surround view image processing path 302. For example, the HLOS core 414 may configure the GPU 408 to operate as a surround view node 304. The surround view image processing path 302 operates concurrently with the rear view image processing path 202. In some examples of the method 500, the operations of blocks 502-512 may be completed prior to the operations of block 516.

[0034] In block 518 , the surround view image processing pathway 302 receives the processed image 210 from the rear view image processing pathway 202 .

[0035] In block 520 , look-around processing is applied to the image received from the rear view image processing path 202 . For example, the look-around image processing path 302 (look-around node 304 ) processes the processed image 210 to generate the look-around image 306 .

[0036] In block 522, the surround view image 306 is displayed. For example, the display node 216 transmits the surround view image 306 to the display device 108 for display.

[0037] The same reference numbers or other reference designators are used in the drawings to indicate the same or similar (functionally and / or structural) features.

[0038] In this description, the term "coupled" may cover connections, communications, or signal paths that achieve a functional relationship consistent with this description. For example, if device A generates a signal for controlling device B to perform an action: (a) in the first example, device A is coupled to device B through a direct connection; or (b) in the second example, if the intermediate component C does not change the functional relationship between device A and device B, device A is coupled to device B through the intermediate component C, so that device B is controlled by device A via the control signal generated by device A.

[0039] Additionally, in this specification, the expression “based on” means “based at least in part on.” Thus, if X is based on Y, then X may be a function of Y and any number of other factors.

[0040] A device that is "configured to" perform a task or function may be configured (e.g., programmed and / or hardwired) by the manufacturer at the time of manufacture to perform that function, and / or may be configurable (or reconfigurable) by a user after manufacture to perform that function and / or other additional or alternative functions. Configuration may be programmable through the device's firmware and / or software, through the construction and / or layout of the device's hardware components and interconnections, or a combination thereof.

[0041] Circuits or devices described herein as including certain components may instead be adapted to be coupled to those components to form the described circuit systems or devices. For example, a structure described as including one or more active or passive elements or subsystems may instead include only a subset of the elements, and may be adapted to be coupled to at least some of the elements during or after manufacture (e.g., by an end user and / or a third party) to form the described structure.

[0042] The circuits described herein may be reconfigured to include additional or different components to provide functionality at least partially similar to that available prior to replacement of the components.

[0043] Although some elements in the described examples are included in the integrated circuit and other elements are outside the integrated circuit, in other example embodiments, more or fewer features may be incorporated into the integrated circuit. In addition, some or all of the features shown as outside the integrated circuit may be included in the integrated circuit, and / or some features shown as within the integrated circuit may be incorporated outside the integrated circuit. As used herein, the term "integrated circuit" means one or more circuits that: (i) are incorporated in / on a semiconductor substrate; (ii) are incorporated in a single semiconductor package; (iii) are incorporated into the same module; and / or (iv) are incorporated in / on the same printed circuit board.

[0044] Modifications in the described embodiments are possible, and other embodiments are possible, within the scope of the claims.

Claims

1. A circuit comprising: A first processor core, a second processor core and a video processing circuit, wherein: The first processor core is configured to program the video processing circuit to: Provide a rear view image processing path; and transferring a rear view image from the rear view image processing path to a surround view image processing path; The second processor core is configured to program the video processing circuit to: Providing the surround view image processing path; receiving the rearward-view image from the rearward-view image processing path; and A surround view image is provided based on the rear view image.

2. The circuit of claim 1 , wherein the first processor core is configured to program the video processing circuit to provide the rear view image processing path before the second processor core programs the video processing circuit to provide the surround view image processing path.

3. The circuit of claim 1, wherein the rear view image processing path is configured to operate simultaneously with the surround view image processing path.

4. The circuit according to claim 1, further comprising a display interface, wherein: The rearview image processing path is configured to pass the rearview image to the display interface; and The surround view image processing path is configured to pass the surround view image to the display interface.

5. The circuit of claim 1 , further comprising a camera interface, wherein: The rearview image processing path is configured as follows: receiving a rear view video signal from the camera interface; and The rearview image is generated based on the rearview video signal.

6. The circuit of claim 5, wherein the rearview image processing path is configured to: receiving a first side view video signal, a second side view video signal, and a front view video signal from the camera interface; generating a first side view image, a second side view image, and a front view image based on the first side view video signal, the second side view video signal, and the front view video signal; and The first side view image, the second side view image, and the front view image are passed to the surround view image processing path.

7. The circuit of claim 6, wherein the surround view image processing path is configured to generate the surround view image based on the rear view image, the first side view image, the second side view image, and the front view image received from the rear view image processing path.

8. A method comprising: programming the video processing circuit to provide a rear view image processing path; generating a rearview image in the rearview image processing path; programming the video processing circuit to provide a surround view image processing path; transferring the rear view image from the rear view image processing path to the surround view image processing path; as well as In the surround view image processing path, a surround view image is generated based on the rear view image.

9. The method according to claim 8, further comprising: programming the rearview image processing path via a first processor core; as well as The surround view image processing path is programmed by a second processor core.

10. The method of claim 8, further comprising generating the rearward view image in the rearward view image processing path prior to operation of the surround view image processing path.

11. The method of claim 8, further comprising operating the rear view image processing path simultaneously with the surround view image processing path.

12. The method according to claim 8, further comprising: providing the rear view image to a display interface; providing the surround view image to the display interface; displaying the rear view image; as well as The surround view image is displayed.

13. The method according to claim 8, further comprising: receiving a rear view video signal, a first side view video signal, a second side view video signal, and a front view video signal in the rear view image processing path; as well as The rearview video signal is processed by the rearview image processing path to generate the rearview image.

14. The method according to claim 13, further comprising: processing the first side view video signal, the second side view video signal, and the front view video signal in the rear view image processing path to generate a first side view image, a second side view image, and a front view image; and The first side view image, the second side view image, and the front view image are passed to the surround view image processing path.

15. The method of claim 14, further comprising processing the rear view image, the first side view image, the second side view image, and the front view image received from the rear view image processing path in the surround view image processing path to generate the surround view image.

16. A system comprising: A camera interface, a video processing circuit, a first processor core, a second processor core and a display interface, wherein: The camera interface is configured to receive a rear view video signal, a first side view video signal, a second side view video signal, and a front view video signal; The video processing circuit is coupled to the camera interface; The first processor core is coupled to the video processing circuit, the first processor core being configured to program the video processing circuit to: Provides a rear view image processing path; providing a rear view image based on the rear view video signal; and transferring the rear view image from the rear view image processing path to a surround view image processing path; The second processor core is coupled to the video processing circuit, the second processor core being configured to program the video processing circuit to: Providing the surround view image processing path; receiving the rearward-view image from the rearward-view image processing path; and providing a surround view image based on the rear view image; and The display interface is coupled to the video processing circuit, and the display interface is configured to provide the rear view image and the surround view image to a video display device.

17. The system of claim 16, wherein the first processor core is configured to program the video processing circuit to provide the rear view image processing path before the second processor core programs the video processing circuit to provide the surround view image processing path.

18. The system of claim 16, wherein the rear view image processing path is configured to operate simultaneously with the surround view image processing path.

19. The system of claim 16, wherein: The rearview image processing path is configured as follows: receiving the first side view video signal, the second side view video signal, and the front view video signal from the camera interface; generating a first side view image, a second side view image, and a front view image based on the first side view video signal, the second side view video signal, and the front view video signal; and passing the first side view image, the second side view image, and the front view image to the surround view image processing path; and The surround view image processing path is configured to generate the surround view image based on the rear view image, the first side view image, the second side view image, and the front view image received from the rear view image processing path.

20. The system of claim 16, further comprising: a rear view camera, a first side view camera, a second side view camera, and a front view camera coupled to the camera interface; as well as A video display device is coupled to the display interface.

Images