Streaming media rearview mirror system based on FPGA

Abstract

The invention discloses a streaming media rearview mirror system based on an FPGA. The streaming media rearview mirror system comprises a camera decoding unit, a DDR image caching unit, a DDR arbitration unit, an image splicing algorithm unit, a coordinate interpolation unit, an image fusion unit, an OSD superposition unit and an image output unit. The camera decoding comprises a vehicle-mounted camera and FPGA (Field Programmable Gate Array) processing; the DDR image cache unit comprises an RAM (Random Access Memory) cache and a DDR cache; the DDR arbitration unit comprises an idle state, a sorting state, a response state, an interruption state and a single-point independent channel. According to the streaming media rearview mirror system based on the FPGA, the camera decoding unit and the DDR image caching unit are arranged, the camera decoding unit can use three cameras to shoot the rear portion of a vehicle at the same time, meanwhile, the sensitivity degree of the FPGA to a power source is greatly reduced, meanwhile, due to flexibility and variability of the FPGA, transmission can be carried out on different cameras, meanwhile, the FPGA is started up fast, and the system is convenient to use. An independent special circuit can be used by the FPGA, delay can be reduced, and the device is convenient to use.

Description

Technical Field

[0001] The present invention relates to the technical field of streaming media rearview mirrors, and in particular to a streaming media rearview mirror system based on FPGA. Background Art

[0002] The streaming media rearview mirror is a vehicle-mounted peripheral electronic product. The streaming media rearview mirror can transmit the rear view to the rearview mirror through the camera next to the high-mounted brake light at the rear of the vehicle. At the same time, the streaming media rearview mirror is equivalent to the reversing image. At the same time, the streaming media rearview mirror can make the rear view wider, and the rear view is no longer blocked by objects in the car such as headrests, rear passengers and cargo. At the same time, people pay more and more attention to driving safety, and the demand for rearview mirrors also increases. Therefore, the streaming media rearview mirror is developing rapidly. The streaming media rearview mirror uses the rear camera of the vehicle to capture the rear view of the vehicle in real time and display it in the central rearview mirror without loss or delay. Compared with The advantages of traditional rearview mirrors are: significantly reducing blind spots, enhancing night vision capabilities, and having a video recording function to prevent rear-end collisions. However, existing streaming media rearview mirrors are divided into single-camera solutions and multi-camera + CPU solutions when in use. When a single-camera solution is in use, the image quality may be blurred due to interference from analog signals and low resolution, which can easily cause the driver to misjudge the distance of the image, and this method has no video recording function. When a multi-camera + CPU solution is in use, the CPU system often requires very complex peripheral circuits, which can easily cause a freeze or flashing of the image when the car is started. At the same time, it can easily cause excessive delays after video processing, which is inconvenient for the driver and extremely inconvenient to use. Summary of the invention

[0003] The purpose of the present invention is to provide a streaming media rearview mirror system based on FPGA to solve the problem in the above background technology that the delay after video processing is too large, the image quality is relatively fuzzy, and it is easy to cause the driver to misjudge the distance sense of the picture.

[0004] To achieve the above object, the present invention provides the following technical solutions: a streaming media rearview mirror system based on FPGA, comprising camera decoding, a DDR image cache unit, a DDR arbitration unit, an image stitching algorithm unit, a coordinate interpolation unit, an image fusion unit, an OSD overlay unit and an image output unit; Camera decoding includes: vehicle-mounted camera and FPGA processing; The DDR image cache unit includes: RAM cache and DDR cache; The DDR arbitration unit includes: idle state, sorting state, response state, interrupt state and single-point independent channel: The vehicle rear view is captured by the on-board camera. The on-board cameras have different signal formats such as AHD, digital LVDS and MIPI. FPGA processing can decode different camera signals. The video signal generated by FPGA processing is transmitted to the DDR image cache unit for subsequent processing. The signal is first transmitted to the RAM cache, and then to the DDR arbitration unit to generate a data request. After the request is successful, the signal can be transmitted to the DDR cache to cache the video. The DDR arbitration unit contains a variety of DDR operations. It collects DDR requests through the idle state and fixes the collection window time. It sorts the urgency of DDR operations through the sorting state and sets VIP channels for urgent operations. In the response state, it responds according to the previous queue and pulls up the corresponding response signal line to indicate the DDR channel number being processed. In the interrupt state, when the DDR channel needs to be accessed immediately, the existing DDR channel can be interrupted and forced to be blocked. The single-point independent channel reorganizes and classifies the input coordinates. The DDR arbitration unit and the DDR image buffer unit transmit the signal information to the image stitching algorithm unit, which can stitch the images between different AHD, LVDS and MIPI cameras, and perform rectangular correction on the four-point coordinates of the camera graphics; The image stitching algorithm unit and the DDR arbitration unit transmit the signal information to the coordinate interpolation unit. Through the single-point operation channel and the coordinate interpolation unit in the arbitration unit, bilinear interpolation calculation is performed on each point obtained in the DDR. At the same time, the decimal part of the coordinate is the interpolation coefficient K. The formula of the single-point operation channel is: K*X+(1-K)Y; The coordinate interpolation unit transmits the processed video information to the image fusion unit, and the image fusion unit performs image fusion at the transition point between the AHD, LVDS and MIPI cameras; The image fusion unit transmits the processed video to the OSD overlay unit, and the OSD overlay unit can be used to increase the OSD interactivity between the entire rearview mirror system, which can facilitate certain function settings for later users. At the same time, after the camera screen is completely restored by the image fusion unit, the video can be transmitted to the image output unit and displayed by the image output unit.

[0005] Preferably, when the vehicle-mounted camera is in the AHD signal format, the FPGA is required to perform a deinterleaving operation therein. When the signal format is LVDS or MIPI, the FPGA parses the video data stream according to the corresponding protocol, so that the FPGA processes and decodes different camera signals.

[0006] Preferably, the DDR image cache unit first pre-stores the signal through the RAM cache, and after the DDR arbitration unit agrees, the signal can be cached inside the DDR cache in a matrix format.

[0007] Preferably, the mechanism of the DDR arbitration unit is a request response model, which collects medical consultation requests in an idle state and queues them in the order of their collection time. A VIP channel is set inside the sorting state to give priority to VIP channel requests.

[0008] Preferably, the field of view of the AHD, digital LVDS and MIPI is basically in fisheye form, and the image stitching algorithm unit performs projection transformation according to the filled-in camera height and performs rectangular correction according to the four-point coordinates.

[0009] Preferably, the image fusion unit processes the transitional images of AHD, digital LVDS and MIPI splicing in a gradual manner.

[0010] Preferably, the OSD overlay unit internally overlays a font library function, and the image output unit is configured as a vehicle-mounted display screen, and the image output unit can display images captured by AHD, digital HDMI and LVDS.

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows: the FPGA-based streaming media rearview mirror system: 1. Camera decoding and DDR image cache units are set up. Camera decoding can use three cameras to shoot the rear of the vehicle at the same time. At the same time, the sensitivity of FPGA to power supply is greatly reduced. At the same time, due to its flexibility and variability, FPGA can transmit to different cameras. At the same time, FPGA starts up faster and can use a separate dedicated circuit to reduce delays and facilitate the use of the device; 2. An image stitching algorithm unit, a coordinate interpolation unit and an image fusion unit are set up. The image stitching algorithm unit can stitch the cameras together to increase the clarity of the camera images. The image fusion unit can fuse the transition points between the cameras. At the same time, the coordinate interpolation unit can perform interpolation calculations on the images to make the stitching between the camera images more accurate, thereby improving the integrity of the camera images. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a schematic diagram of the system structure of the present invention; Figure 2 It is a schematic diagram of the process structure of the present invention; Figure 3 It is a schematic diagram of the structure of the DDR arbitration unit of the present invention. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0014] See also Figure 1-3 ,The present invention provides a technical solution: a streaming media rearview mirror system based on FPGA, including camera decoding, DDR image cache unit, DDR arbitration unit, image stitching algorithm unit, coordinate interpolation unit, image fusion unit, OSD overlay unit and image output unit; Camera decoding includes: vehicle-mounted camera and FPGA processing; The DDR image cache unit includes: RAM cache and DDR cache; The DDR arbitration unit includes: idle state, sorting state, response state, interrupt state and single-point independent channel: The vehicle rear view is captured by the on-board camera. The on-board cameras have different signal formats such as AHD, digital LVDS and MIPI. FPGA processing can decode different camera signals. The video signal generated by FPGA processing is transmitted to the DDR image cache unit for subsequent processing. The signal is first transmitted to the RAM cache, and then to the DDR arbitration unit to generate a data request. After the request is successful, the signal can be transmitted to the DDR cache to cache the video. The DDR arbitration unit contains a variety of DDR operations. It collects DDR requests through the idle state and fixes the collection window time. It sorts the urgency of DDR operations through the sorting state and sets VIP channels for urgent operations. In the response state, it responds according to the previous queue and pulls up the corresponding response signal line to indicate the DDR channel number being processed. In the interrupt state, when the DDR channel needs to be accessed immediately, the existing DDR channel can be interrupted and forced to be blocked. The single-point independent channel reorganizes and classifies the input coordinates. The DDR arbitration unit and the DDR image buffer unit transmit the signal information to the image stitching algorithm unit, which can stitch the images between different AHD, LVDS and MIPI cameras, and perform rectangular correction on the four-point coordinates of the camera graphics; The image stitching algorithm unit and the DDR arbitration unit transmit the signal information to the coordinate interpolation unit. Through the single-point operation channel and the coordinate interpolation unit in the arbitration unit, bilinear interpolation calculation is performed on each point obtained in the DDR. At the same time, the decimal part of the coordinate is the interpolation coefficient K. The formula of the single-point operation channel is: K*X+(1-K)Y; The coordinate interpolation unit transmits the processed video information to the image fusion unit, and the image fusion unit performs image fusion at the transition point between the AHD, LVDS and MIPI cameras; The image fusion unit transmits the processed video to the OSD overlay unit, which can increase the OSD interactivity between the entire rearview mirror system, and facilitate certain function settings in the later user use. At the same time, after the camera screen is completely restored by the image fusion unit, the video can be transmitted to the image output unit, and the video can be displayed by the image output unit; The rear side of the car is photographed simultaneously by the on-board camera, and the three cameras AHD, LVDS and MIPI are photographed simultaneously. The signals after AHD, LVDS and MIPI are photographed are transmitted to FPGA for processing, and the images after AHD, LVDS and MIPI are decoded by FPGA. When the signal format is AHD, FPGA is required to perform deinterleaving inside, and the channel number needs to be parsed out according to the packet header FF-00-00-XY and then the data stream is extracted respectively. When the signal format is LVDS or MIPI, FPGA parses the video data stream HS, VS, DE or DATA according to the corresponding protocol. At the same time, the camera decodes and transmits the decoded data to the inside of the DDR image cache unit. At this time, the RAM cache can store the signal, and the signal can be transmitted to the inside of the DDR arbitration unit, and judged by the DDR arbitration unit. When the request to the DDR arbitration unit is successful, the DDR cache can store the signal. At the same time, the DDR arbitration unit can judge and sort the transmitted signals to maintain the continuity of the DDR requests inside the DDR arbitration unit. At this time, the idle state sorts the DDR requests and arranges them according to the order of the DDR request input. At the same time, the sorting state can set a VIP channel inside the DDR arbitration unit. The VIP channel processes the urgent DDR requests. At the same time, the response state can sort according to the DDR request time and perform cyclic sorting. The interrupt state can interrupt the current request when an urgent DDR request appears, and the urgent DDR request can be jammed. The single-point independent channel can reorganize and classify the DDR operations to ensure the maximum efficiency of each activation of DDR. The DDR image cache unit transmits the signal to the image stitching algorithm unit, and the images taken by three different cameras are stitched together by the image stitching algorithm unit. The image stitching algorithm unit first performs barrel correction through the stitching algorithm, and then performs projection transformation according to the filled-in camera height. Finally, a rectangular correction is performed according to the four-point coordinates. The signal processed by the image stitching algorithm unit is transmitted to the coordinate interpolation unit. Four points are obtained in the DDR for bilinear interpolation calculation. The pixels in the row are calculated through the K*X+(1-K)Y formula, which is convenient for the camera image to be interpolated by the output Fixing and splicing are performed. At this time, the signal passing through the coordinate interpolation unit is transmitted to the image fusion unit. The image fusion unit fuses the two adjacent images in a gradual manner, eliminates the line marks between the images, and makes the image clearer. The OSD overlay unit can add system settings to the monitoring images generated by the monitoring. Finally, the camera video after splicing is transmitted to the video screen and then to the image output unit. The video is transmitted to the image output unit for display. In the video transmitted to the image output unit, a single AHD, digital LVDS and MIPI video can be displayed.

[0015] When the vehicle-mounted camera uses the AHD signal format, the FPGA needs to perform deinterleaving operations. When the signal format is LVDS or MIPI, the FPGA parses the video data stream according to the corresponding protocol, so that the FPGA can process and decode different camera signals. FPGA can decode different signals and forms of vehicle-mounted cameras, which can improve the efficiency of device decoding.

[0016] In the DDR image cache unit, the signal is first pre-stored in the RAM cache. After the DDR arbitration unit agrees, the signal can be cached in the DDR cache in a matrix format. In the DDR image cache unit, the signal needs to be transmitted to the DDR arbitration unit first. After the request of the DDR arbitration unit is successful, the signal will be stored in the DDR cache.

[0017] The mechanism of the DDR arbitration unit is a request-response model. It collects medical consultation requests in the idle state and queues them in the order of their collection time. A VIP channel is set inside the sorting state to give priority to VIP channel requests. The DDR arbitration unit can arrange and process the DDR operations, and can also perform queue processing on the DDR operations, so as to facilitate the operation of urgent DDR operation requests.

[0018] The field of view of AHD, digital LVDS and MIPI is basically in fisheye form. The image stitching algorithm unit performs projection transformation according to the filled-in camera height and performs rectangle correction according to the four-point coordinates; AHD, digital LVDS and MIPI simultaneously shoot the rear of the vehicle. At the same time, AHD, digital LVDS and MIPI have a large shooting range, and the image stitching algorithm unit stitches and restores the images.

[0019] The image fusion unit uses a gradual transition method to process the transition images between AHD, digital LVDS and MIPI splicing; The image fusion unit can process the connection of the images taken by the camera to increase the clarity of the images.

[0020] The OSD overlay unit has internal overlay font function, and the image output unit is set as a vehicle-mounted display screen. The image output unit can display images captured by AHD, digital HDMI and LVDS; The spliced ​​video is transmitted to the image output unit, the video is displayed by the image output unit, and the rear side of the vehicle is observed by the image output unit.

[0021] Working principle: When using the FPGA-based streaming media rearview mirror system, a DDR image cache unit and a DDR arbitration unit are set up to improve the clarity of the device video screen, improve the safety of vehicle driving, and increase the overall practicality.

[0022] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A streaming media rearview mirror system based on FPGA, comprising the following features: Camera decoding, DDR image cache unit, DDR arbitration unit, image stitching algorithm unit, coordinate interpolation unit, image fusion unit, OSD overlay unit and image output unit; Camera decoding includes: vehicle-mounted camera and FPGA processing; The DDR image cache unit includes: RAM cache and DDR cache; The DDR arbitration unit includes: idle state, sorting state, response state, interrupt state and single-point independent channel: The vehicle rear view is captured by the on-board camera. The on-board cameras have different signal formats such as AHD, digital LVDS and MIPI. FPGA processing can decode different camera signals. The video signal generated by FPGA processing is transmitted to the DDR image cache unit for subsequent processing. The signal is first transmitted to the RAM cache, and then to the DDR arbitration unit to generate a data request. After the request is successful, the signal can be transmitted to the DDR cache to cache the video. The DDR arbitration unit contains a variety of DDR operations. It collects DDR requests through the idle state and fixes the collection window time. It sorts the urgency of DDR operations through the sorting state and sets VIP channels for urgent operations. In the response state, it responds according to the previous queue and pulls up the corresponding response signal line to indicate the DDR channel number being processed. In the interrupt state, when the DDR channel needs to be accessed immediately, the existing DDR channel can be interrupted and forced to be blocked. The single-point independent channel reorganizes and classifies the input coordinates. The DDR arbitration unit and the DDR image buffer unit transmit the signal information to the image stitching algorithm unit, which can stitch the images between different AHD, LVDS and MIPI cameras, and perform rectangular correction on the four-point coordinates of the camera graphics; The image stitching algorithm unit and the DDR arbitration unit transmit the signal information to the coordinate interpolation unit. Through the single-point operation channel in the arbitration unit and the coordinate interpolation unit, bilinear interpolation calculation is performed on each point obtained in the DDR. At the same time, the decimal part of the coordinate is the interpolation coefficient K. The formula of the single-point operation channel is: K*X+(1-K)Y; The coordinate interpolation unit transmits the processed video information to the image fusion unit, and the image fusion unit performs image fusion at the transition point between the AHD, LVDS and MIPI cameras; The image fusion unit transmits the processed video to the OSD overlay unit, and the OSD overlay unit can be used to increase the OSD interactivity between the entire rearview mirror system, which can facilitate certain function settings for later users. At the same time, after the camera screen is completely restored by the image fusion unit, the video can be transmitted to the image output unit and displayed by the image output unit.

2. The FPGA-based streaming media rearview mirror system according to claim 1, characterized in that: When the vehicle-mounted camera uses the AHD signal format, the FPGA needs to perform deinterleaving operations therein. When the signal format is LVDS or MIPI, the FPGA parses the video data stream according to the corresponding protocol, so that the FPGA processes and decodes different camera signals.

3. The FPGA-based streaming media rearview mirror system according to claim 2, characterized in that: The DDR image cache unit first pre-stores the signal through the RAM cache, and after the DDR arbitration unit agrees, the signal can be cached inside the DDR cache in a matrix format.

4. The FPGA-based streaming media rearview mirror system according to claim 1, characterized in that: The mechanism of the DDR arbitration unit is a request response model, which collects application requests in an idle state and queues them in the order of their collection time. A VIP channel is set inside the sorting state to give priority to VIP channel requests.

5. The FPGA-based streaming media rearview mirror system according to claim 2, characterized in that: The field of view of the AHD, digital LVDS and MIPI is basically in fisheye form, and the image stitching algorithm unit performs projection transformation according to the filled-in camera height and performs rectangular correction according to the four-point coordinates.

6. The FPGA-based streaming media rearview mirror system according to claim 2, characterized in that: The image fusion unit processes the transitional images of AHD, digital LVDS and MIPI splicing in a gradual manner.

7. The FPGA-based streaming media rearview mirror system according to claim 2, characterized in that: The OSD overlay unit has a font library function, and the image output unit is configured as a vehicle-mounted display screen. The image output unit can display images captured by AHD, digital HDMI and LVDS.

Images