Panoramic image display system and method

Abstract

The invention discloses a panoramic image display system and method. In the system, each fisheye camera is used for collecting images in a corresponding shooting range; the main control chip is used for applying for the memory of the direct memory access buffer area through the image display service and transmitting a memory address to the real-time rendering service; splicing and rendering the acquired images through a real-time rendering service to obtain a panoramic image of the vehicle, and storing the panoramic image in a memory; and when receiving the panoramic image in the memory, the image display service sends the panoramic image to the display controller in the form of serial transmission protocol data, so that the display controller displays the panoramic image in the vehicle-mounted screen. By deploying a real-time rendering service and an image display service on the main control chip and matching various software and hardware modules and interface protocols among the fisheye camera, the main control chip and the display controller, the panoramic image of the vehicle can be quickly, accurately and efficiently displayed, and the image quality and efficiency of panoramic image display are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular, to a panoramic image display system and method. Background Art

[0002] With the development of automobile technology, panoramic imaging functions are increasingly used in mainstream smart cockpits. Existing in-vehicle panoramic imaging solutions are often deployed on infotainment controllers (CSCs), and CSCs often use Qualcomm chips, which do not have strong image processing capabilities and low power consumption advantages, and the image quality is often low and blurry. In addition, existing in-vehicle panoramic imaging solutions often use Ethernet signal interaction, which requires the initialization of Ethernet services deployed on the chip. The cold start time is as long as more than 13 seconds, and the panoramic image display speed is slow. In summary, how to balance image quality and efficiency in displaying panoramic images is an urgent problem to be solved. Summary of the invention

[0003] The present application provides a panoramic image display system and method to efficiently determine appropriate encoding parameters and improve the image quality and efficiency of panoramic image display.

[0004] In a first aspect, an embodiment of the present application provides a panoramic image display system, comprising: a main control chip, a display controller, a vehicle screen, and a plurality of fisheye cameras arranged at different positions of a vehicle, wherein the shooting range of each of the fisheye cameras covers all directions around the vehicle, and the main control chip is respectively connected to the display controller and each of the fisheye cameras;

[0005] Each of the fisheye cameras is used to collect images within a corresponding shooting range;

[0006] The main control chip is used to apply for a memory of a direct memory access buffer through an image display service, and transmit the address of the memory to a real-time rendering service; stitch and render the images collected by each of the fisheye cameras through the real-time rendering service to obtain a panoramic image of the vehicle and store it in the memory; when the image display service receives the panoramic image in the memory, the panoramic image is sent to the display controller in the form of serial transmission protocol data;

[0007] The display controller is used to display the panoramic image on the vehicle-mounted screen.

[0008] In a second aspect, the embodiment of the present application further provides a panoramic image display method, including:

[0009] Each of the fisheye cameras collects images within a corresponding shooting range;

[0010] The main control chip applies for a memory of a direct memory access buffer through an image display service, and transmits the address of the memory to a real-time rendering service; the images collected by each of the fisheye cameras are spliced ​​and rendered through the real-time rendering service to obtain a panoramic image of the vehicle and store it in the memory; when the panoramic image in the memory is received through the image display service, the panoramic image is sent to the display controller in the form of serial transmission protocol data;

[0011] The display controller displays the panoramic image on the vehicle-mounted screen according to the serial transmission protocol data.

[0012] The embodiments of the present application provide a panoramic image display system and method, which can quickly, accurately and efficiently display the panoramic image of the vehicle by deploying real-time rendering services and image display services on the main control chip, and coordinating various software and hardware modules and interface protocols between the fisheye camera, the main control chip and the display controller, thereby improving the image quality and efficiency of the panoramic image display. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0014] Figure 1 A schematic diagram of the structure of a panoramic image display system provided in an embodiment of the present application;

[0015] Figure 2 A schematic diagram of a panoramic image display process provided by an embodiment of the present application;

[0016] Figure 3 A schematic diagram of a real-time rendering service architecture provided in an embodiment of the present application.

[0017] Figure 4 A flowchart of a panoramic image display method provided in an embodiment of the present application;

[0018] Figure 5 A schematic diagram of an image display service processing process provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only the parts related to the present application, rather than all structures, are shown in the accompanying drawings.

[0020] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0021] It should be noted that the concepts of "first", "second", etc. mentioned in the embodiments of the present application are only used to distinguish different devices, modules, units or other objects, and are not used to limit the order or interdependence of the functions performed by these devices, modules, units or other objects.

[0022] In addition, the embodiments in the present application and the features in the embodiments may be combined with each other if there is no conflict.

[0023] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0024] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the relevant content of the solution.

[0025] Figure 1 A schematic diagram of the structure of a panoramic image display system provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the panoramic image display includes: a main control chip 120, a display controller 130, a vehicle screen 140 and a plurality of fisheye cameras 110 arranged at different positions of the vehicle, the shooting range of each fisheye camera 110 covers all directions around the vehicle, the main control chip 120 is respectively connected to the display controller 130 and each fisheye camera 110; each fisheye camera 110 is respectively used to collect images within the corresponding shooting range; the main control chip 120 is used to apply for the memory of the direct memory access buffer through the image display service, and pass the address of the memory to the real-time rendering service; the images collected by each fisheye camera 110 are spliced ​​and rendered through the real-time rendering service to obtain a panoramic image of the vehicle and store it in the memory; when the panoramic image in the memory is received through the image display service, the panoramic image is sent to the display controller 130 in the form of serial transmission protocol data; the display controller 130 is used to display the panoramic image on the vehicle screen 140.

[0026] Exemplarily, the system includes four surround fisheye cameras 110, a main control chip 120 (for example, a TDA4VM main control chip) for processing the images captured by each fisheye camera 110 to obtain a panoramic image, and a display controller 130 for displaying the panoramic image on a vehicle screen 140, the display controller 130 is, for example, a vehicle computer or an infotainment controller (CSC), etc. The main control chip 120 is deployed with a real-time rendering service and an image display service.

[0027] Specifically, the four surround-view fisheye cameras 110 can be used to collect images of the front, rear, left and right of the vehicle respectively and send them to the main control chip 120; in the main control chip 120, the image display service can be used to apply for direct memory access (DMA) buffer memory and pass the address to the real-time rendering service. After the real-time rendering service obtains the memory address, it stitches and renders the surround-view images, analyzes the current calibration parameters, and then notifies the image display service that the rendering is completed; the image display service outputs the rendered panoramic image to the display controller 130. After receiving the rendered panoramic image data, the display controller 130 displays the rendered panoramic image through the vehicle screen 140.

[0028] In one embodiment, each fisheye camera 110 includes an image signal processor (ISP), and the image processor is used to adjust the quality of the collected image.

[0029] In one embodiment, the main control chip 120 is also used to convert the images collected by each fisheye camera 110 into Mobile Industry Processor Interface (MIPI) data through a deserialization chip, and transmit it to the real-time rendering service through the camera serial interface.

[0030] In one embodiment, the image display service is specifically used to output MIPI data through the camera serial interface, convert the MIPI data into serial transmission protocol data through a serialization chip, and send the serial transmission protocol data to the display controller 130 through a low-voltage differential signal (LVDS) coaxial cable.

[0031] Figure 2 A schematic diagram of a panoramic image display process provided by an embodiment of the present application. Figure 2 As shown in the figure, the panoramic image display process mainly includes:

[0032] Each surround fisheye camera (Camera-0, Camera-1, Camera-2, Camera-3) collects images from the front, back, left, and right of the vehicle respectively;

[0033] Each surround fisheye camera adjusts the image quality of the camera through the hardware ISP;

[0034] Each fisheye camera sends GMSL2 protocol data, which is converted into MIPI data by the deserialization chip (such as MAX9296) and then collected to the main control chip (such as TDA4VM).

[0035] The main control chip applies for DMABuffer memory through the image display service and passes the address to the real-time rendering service;

[0036] The real-time rendering service obtains images from four surround fisheye cameras;

[0037] After obtaining the memory address, the real-time rendering service stitches and renders the surround image;

[0038] The image display service outputs MIPI data through the CSITX hardware on the main control chip and outputs GMSL2 protocol data through the serialization chip (such as MAX9295);

[0039] The display controller receives the GMSL2 protocol data sent by the main control chip and displays the rendered image on the large in-vehicle screen.

[0040] In one embodiment, the real-time rendering service includes:

[0041] The application logic layer is used to receive and process the images collected by each fisheye camera 110 and call the kernel layer;

[0042] The kernel layer is used to stitch and render the images collected by the fisheye cameras 110 to obtain a panoramic image of the vehicle;

[0043] The platform adaptation layer is used to provide corresponding interfaces for cross-platform panoramic image display.

[0044] Figure 3 A schematic diagram of a real-time rendering service architecture provided in an embodiment of the present application. Figure 3As shown in the figure, the application logic layer is mainly responsible for receiving external signals, signal processing, state maintenance and calling the kernel layer interface of the panoramic image (Around View Monitor, AVM), so as to realize the external message interaction of AVM and realize rendering synchronization. Among them, the AVM state machine is used to realize the functions of view state management, function start and stop, initialization (Init) / deinitialization (DeInit) logic control and / or quick start, etc. The signal response and mapping (Mapping) module is used to process the whole vehicle signal and interact with the user through the screen. In addition, there are function modules such as vulnerability (Debug) detection, fault diagnosis, view layout / model layout, configuration of viewing angle and model parameters, and analysis of radar distance logic.

[0045] The kernel layer mainly provides modeling and rendering of bird's-eye view, single-channel view, bird's-eye view trajectory, single-channel trajectory, car model, radar and blind spot filling through the graphics processing unit (GPU). In addition, the AVM algorithm module can consider 2D coordinate mapping, distortion correction, top trajectory, color balance, single-channel perspective transformation, single-channel trajectory and side perspective transformation, etc. The CPU scheduling module can be used for cache control, management of rendering threads, color balance threads and various model resources, etc. This process can be processed using the digital radiography (DR) algorithm;

[0046] The platform adaptation layer mainly shields the Kernel layer from changes in hardware, operating system (OS), and driver changes, and provides an abstract interface for subsequent cross-platform porting development, including shader creation, texture buffer binding, and / or graphics library (Embedded-System Graphics Library, EGL) initialization.

[0047] The panoramic image display system provided in the embodiment of the present application can use the GPU of the main control chip to process and splice the surround view images, and add different GPU models for rendering, so as to achieve better display effect and better user experience; through the coordinated transmission and interaction of various software, hardware and protocols, it can solve the problems of long cold start time and slow panoramic image display speed. In theory, it can increase the startup speed of the vehicle's panoramic image to within 7 seconds, providing the driver with a more convenient and safe driving experience; in addition, the existing on-board panoramic image solutions often use multiple threads to splice and render the same memory buffer. The software architecture of the present method uses multi-GPU model rendering and single-thread drawing, which can solve the risk of panoramic image freeze caused by multi-thread operation and the risk of panoramic image delay caused by thread switching.

[0048] Figure 4This is a flowchart of a panoramic image display method provided in an embodiment of the present application. The panoramic image display method is applicable to the panoramic image display system described in any of the above embodiments. For details not described in this embodiment, please refer to any of the above embodiments. Figure 4 As shown, the method specifically comprises the following steps:

[0049] S210. Each of the fisheye cameras collects images within a corresponding shooting range.

[0050] S220, the main control chip applies for the memory of the direct memory access buffer through the image display service, and transmits the address of the memory to the real-time rendering service; the images collected by each of the fisheye cameras are stitched and rendered through the real-time rendering service to obtain a panoramic image of the vehicle and store it in the memory; when the image display service receives the panoramic image in the memory, the panoramic image is sent to the display controller in the form of serial transmission protocol data.

[0051] S230: The display controller displays the panoramic image on the vehicle-mounted screen.

[0052] Figure 5 A schematic diagram of an image display service processing process provided by an embodiment of the present application. Figure 5 As shown in the figure, the processing of image display service mainly includes:

[0053] 1) Image display service starts;

[0054] 2) Determine whether the buffer image sent by the real-time rendering service is received, if yes, proceed to step 6), if not, proceed to step 3);

[0055] 3) Obtaining panoramic fisheye camera images;

[0056] 4) Use CPU to render and stitch single view;

[0057] 5) Send the rendering buffer to the display controller and return to step 2);

[0058] 6) Send the buffer image sent by the real-time rendering service to the display controller;

[0059] 7) The display controller displays the received panoramic image through the vehicle screen.

[0060] In one embodiment, each fisheye camera includes an image processor; before the main control chip stitches and renders the images collected by each fisheye camera through the real-time rendering service, the method further includes:

[0061] S2110. Each image processor adjusts the quality of the image collected by the corresponding fisheye camera.

[0062] In one embodiment, before the main control chip stitches and renders the images collected by each fisheye camera through the real-time rendering service, the method further includes:

[0063] S2210, the main control chip converts the images collected by each fisheye camera into MIPI data of the mobile industry processor interface through the deserialization chip, and transmits it to the real-time rendering service through the camera serial interface.

[0064] In one embodiment, the panoramic image is sent to the display controller in the form of serial transmission protocol data, including:

[0065] The MIPI data is output through the camera serial interface, and the MIPI data is converted into serial transmission protocol data through the serialization chip, and the serial transmission protocol data is sent to the display controller through the low voltage differential signal LVDS coaxial cable.

[0066] In one embodiment, the method further comprises:

[0067] S2220, the real-time rendering service receives and processes the images collected by each fisheye camera based on the application logic layer and calls the kernel layer;

[0068] S2220. Provide a corresponding interface for cross-platform panoramic image display based on the platform adaptation layer.

[0069] The panoramic image display method provided in the embodiment of the present application can be applied to the panoramic image display system provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0070] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0071] The computer programs for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer programs are executed by the processor, the functions / operations specified in the flow charts and / or block diagrams are implemented. The computer programs may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0072] Computer readable storage medium can be a tangible medium that can contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. Computer readable storage medium can include but is not limited to electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. Alternatively, computer readable storage medium can be a machine readable signal medium. More specific examples of machine readable storage medium can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0073] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0074] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0075] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution of this application can be achieved, and this document is not limited here.

[0076] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A panoramic image display system, characterized in that: include: A main control chip, a display controller, a vehicle screen, and a plurality of fisheye cameras arranged at different positions of the vehicle, wherein the shooting range of each of the fisheye cameras covers all directions around the vehicle, and the main control chip is connected to the display controller and each of the fisheye cameras respectively; Each of the fisheye cameras is used to collect images within a corresponding shooting range; The main control chip is used to apply for a memory of a direct memory access buffer through an image display service, and transmit the address of the memory to a real-time rendering service; splicing and rendering the images captured by the fisheye cameras through the real-time rendering service to obtain a panoramic image of the vehicle and store it in the memory; When receiving the panoramic image in the memory through the image display service, the panoramic image is sent to the display controller in the form of serial transmission protocol data; The display controller is used to display the panoramic image on the vehicle-mounted screen.

2. The system according to claim 1, characterized in that Each of the fisheye cameras includes an image processor, and the image processor is used to adjust the quality of the collected image.

3. The system according to claim 1, characterized in that The main control chip is also used to convert the images collected by each fisheye camera into mobile industry processor interface MIPI data through a deserialization chip, and transmit it to the real-time rendering service through a camera serial interface.

4. The system according to claim 1, characterized in that The image display service is specifically used to output MIPI data through the camera serial interface, convert the MIPI data into serial transmission protocol data through a serialization chip, and send the serial transmission protocol data to the display controller through a low voltage differential signal LVDS coaxial cable.

5. The system according to claim 1, characterized in that The real-time rendering service includes: The application logic layer is used to receive and process the images collected by each of the fisheye cameras and call the kernel layer; A kernel layer, used for stitching and rendering the images collected by the fisheye cameras to obtain a panoramic image of the vehicle; The platform adaptation layer is used to provide corresponding interfaces for cross-platform panoramic image display.

6. A panoramic image display method, characterized in that: Applied to a panoramic image display system, the panoramic image display system comprises: a main control chip, a display controller, a vehicle screen, and a plurality of fisheye cameras arranged at different positions of the vehicle, the shooting range of each fisheye camera covering all directions around the vehicle; the method comprises: Each of the fisheye cameras collects images within a corresponding shooting range; The main control chip applies for a memory of a direct memory access buffer through an image display service, and transmits the address of the memory to a real-time rendering service; the images collected by each of the fisheye cameras are spliced ​​and rendered through the real-time rendering service to obtain a panoramic image of the vehicle and store it in the memory; when the panoramic image in the memory is received through the image display service, the panoramic image is sent to the display controller in the form of serial transmission protocol data; The display controller displays the panoramic image on the vehicle-mounted screen according to the serial transmission protocol data.

7. The method according to claim 6, characterized in that Each of the fisheye cameras includes an image processor; before the main control chip stitches and renders the images collected by each of the fisheye cameras through the real-time rendering service, the method further includes: Each of the image processors adjusts the quality of the image captured by the corresponding fisheye camera.

8. The method according to claim 6, characterized in that Before the main control chip stitches and renders the images collected by each of the fisheye cameras through the real-time rendering service, the method further includes: The main control chip converts the images collected by each fisheye camera into mobile industry processor interface MIPI data through a deserialization chip, and transmits the data to the real-time rendering service through a camera serial interface.

9. The method according to claim 6, characterized in that The method includes sending the panoramic image to the display controller in the form of serial transmission protocol data, comprising: The MIPI data is output through the camera serial interface, and the MIPI data is converted into serial transmission protocol data through a serialization chip, and the serial transmission protocol data is sent to the display controller through a low voltage differential signal LVDS coaxial cable.

10. The method according to claim 6, characterized in that Also includes: The real-time rendering service receives and processes the images collected by each of the fisheye cameras based on the application logic layer and calls the kernel layer; Based on the platform adaptation layer, a corresponding interface is provided for cross-platform panoramic image display.

Images