Method, device and vehicle for improving ISP real-time performance

Abstract

The application discloses a method, device and vehicle for improving ISP real-time performance. The method keeps the accuracy of ISP control parameters of driving area blocks by carrying out block processing on ISP original data, and realizes key processing on driving area blocks related to vehicle driving. The ISP control parameters of background area blocks are reduced or processed by empty symbols, and the background area blocks not related to vehicle driving are processed by virtualization or blanking. The data processing of the background area blocks is reduced by extracting the background area blocks separately, so that the efficiency of restoring normal images by ISP processing is improved, and the timeliness of image acquisition is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and in particular to a method and device for improving ISP real-time performance and a vehicle. BACKGROUND

[0002] With the increase of China's economic strength and the improvement of people's living standards, new energy vehicles with automatic driving technology have also gained more attention and development. In the automatic driving technology, considering the cost and difficulty of technology implementation, cameras are still mainly used for visual perception and machine recognition at present.

[0003] At present, in order to improve the accuracy and accuracy of vehicle visual perception, automobile manufacturers use more cameras, and the resolution of each camera is also increasing. However, the camera is limited by the photoelectric conversion mechanism and can only produce monochrome raw data similar to grid arrangement, and cannot produce normal RGB images that can be viewed by the human eye, and needs to restore the monochrome raw data to normal images through ISP (Image Signal Processing). The existing traditional ISP cannot realize real-time continuous shooting of multiple cameras at high resolution, so as to cause it difficult to guarantee the real-time performance of the obtained images.

[0004] Therefore, in the process of real-time image acquisition in the prior art, there is a problem of low timeliness of image acquisition due to the fact that the traditional ISP cannot realize continuous shooting. SUMMARY

[0005] Therefore, it is necessary to provide a method and device for improving ISP real-time performance and a vehicle, so as to solve the problem of low timeliness of image acquisition due to the fact that the traditional ISP cannot realize continuous shooting in the process of real-time image acquisition in the prior art.

[0006] In order to solve the above problems, the present application provides a method for improving ISP real-time performance, comprising:

[0007] obtaining ISP raw data and performing block processing on the ISP raw data to obtain a plurality of ISP raw data blocks corresponding thereto, wherein the ISP raw data blocks are marked with corresponding position information;

[0008] classifying the ISP raw data blocks according to the position information to determine driving area blocks and background area blocks;

[0009] adjusting the initial ISP control parameters of the driving area blocks and the background area blocks respectively to obtain corrected ISP control parameters;

[0010] processing the ISP raw data based on the corrected ISP control parameters to determine updated ISP images.

[0011] Further, the ISP original data is processed in blocks to obtain a plurality of ISP original data blocks corresponding thereto, including:

[0012] A minimum data block size is preset;

[0013] Based on the minimum data block size, an area where the ISP original data is located is divided into a plurality of ISP original data blocks of the same size.

[0014] Further, the ISP original data block includes a packet group and a data error correction code;

[0015] The packet group includes a packet header, a size of the packet group, a number of cameras, position information of the ISP original data block, size information of the ISP original data block, and an image processing effect of the ISP original data block.

[0016] Further, the ISP original data block is classified according to the position information to determine a driving area block and a background area block, including:

[0017] A preset position number is obtained;

[0018] Based on the position information, an area block in the ISP original data block that matches the preset position number is determined as the driving area block;

[0019] Other area blocks in the ISP original data block except the driving area block are determined as the background area block;

[0020] The position number corresponds to the position information of the ISP original data block one by one.

[0021] Further, the initial ISP control parameters of the driving area block and the background area block are adjusted respectively to obtain corrected ISP control parameters, including:

[0022] The data error correction code corresponding to the driving area block is maintained to determine accurate ISP control parameters of the driving area block;

[0023] The data error correction code corresponding to the background area block is revised to be a null character to determine null ISP control parameters of the background area block.

[0024] Further, the ISP original data is processed based on the corrected ISP control parameters to determine an updated ISP image, including:

[0025] The driving area block is processed based on the accurate ISP control parameters to determine a driving image block corresponding to the driving area block;

[0026] The background area block is processed based on the null ISP control parameters to determine a background image block corresponding to the background area block;

[0027] The driving image block and the background image block are fused and spliced to determine the updated ISP image.

[0028] Further, the driving area block is processed based on the accurate ISP control parameter, and the previous process further comprises:

[0029] ISP preprocessing is performed on the ISP original data in the driving area block.

[0030] The ISP preprocessing comprises sensor black level correction processing, sensor bad point correction processing, sensor bottom noise removal processing, sensor lens correction processing and sensor flick removal processing.

[0031] Further, the initial ISP control parameter comprises a brightness parameter, a gamma parameter, an automatic exposure parameter, a white balance parameter, and other parameters related to image effects.

[0032] In order to solve the above problems, the application further provides a device for improving the real-time performance of ISP, which is applied to vehicle automatic driving, comprising:

[0033] The ISP original data block acquisition module is configured to acquire ISP original data and perform block processing on the ISP original data to obtain a plurality of ISP original data blocks corresponding thereto, wherein the ISP original data blocks are marked with corresponding position information.

[0034] The classification module is configured to classify the ISP original data blocks according to the position information to determine driving area blocks and background area blocks.

[0035] The ISP control parameter correction module is configured to adjust the initial ISP control parameters of the driving area blocks and the background area blocks respectively to obtain corrected ISP control parameters.

[0036] The updated ISP image determination module is configured to process the ISP original data based on the corrected ISP control parameters to determine an updated ISP image.

[0037] In order to solve the above problems, the application further provides a vehicle comprising a processor and a memory, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the method for improving the real-time performance of ISP as described above.

[0038] The beneficial effects of the above technical scheme are: the method, device and vehicle for improving ISP real-time are provided, the ISP original data is processed by block, the accuracy of the ISP control parameters of the driving area block is maintained, the driving area block related to the vehicle driving is processed, the ISP control parameters of the background area block are deleted or processed by empty symbols, the background area block irrelevant to the vehicle driving is processed by virtualization or blanking, the background area block is extracted separately, the data processing of the background area block is reduced, the efficiency of the normal image restored by ISP processing is improved, and the timeliness of image acquisition is improved. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The flowchart of the embodiment of the method for improving ISP real-time provided by the application is shown.

[0040] Figure 2 The flowchart of the embodiment of the method for improving ISP real-time provided by the application is shown.

[0041] Figure 3 The flowchart of the embodiment of the method for improving ISP real-time provided by the application is shown.

[0042] Figure 4 The flowchart of the embodiment of the method for improving ISP real-time provided by the application is shown.

[0043] Figure 5 The first two steps of the process diagram of the embodiment of the method for improving ISP real-time provided by the application are shown.

[0044] Figure 6 The third step of the process diagram of the embodiment of the method for improving ISP real-time provided by the application is shown.

[0045] Figure 7 The fourth step of the process diagram of the embodiment of the method for improving ISP real-time provided by the application is shown.

[0046] Figure 8 The fifth step of the process diagram of the embodiment of the method for improving ISP real-time provided by the application is shown.

[0047] Figure 9 The sixth step of the process diagram of the embodiment of the method for improving ISP real-time provided by the application is shown.

[0048] Figure 10 The seventh step of the process diagram of the embodiment of the method for improving ISP real-time provided by the application is shown.

[0049] Figure 11 Resulting diagram of an embodiment of processing an image according to conventional ISP technology provided by the present application;

[0050] Figure 12 Resulting diagram of an embodiment of processing an image according to the technical solution of the present application provided by the present application;

[0051] Figure 13 Structural diagram of an embodiment of the device for improving ISP real-time provided by the present application;

[0052] Figure 14 Structural diagram of an embodiment of the electronic device provided by the present application. DETAILED DESCRIPTION

[0053] The preferred embodiments of the present application will be described in detail below with reference to the accompanying drawings, which form a part of this application. The accompanying drawings, together with the description, illustrate the principles of the present application, and are not intended to limit the scope of the present application.

[0054] Before the embodiments are described, the ISP is described first:

[0055] ISP (Image Signal Processing) is mainly used to process the output signal of the front-end image sensor unit to match different manufacturers' image sensors.

[0056] With the increase of China's economic strength and the improvement of people's living standards, new energy vehicles with automatic driving technology have also gained more and more attention and development. In the automatic driving technology, considering the cost and difficulty of technical implementation, cameras are still mainly used for visual perception and machine recognition at present.

[0057] In order to improve the accuracy and accuracy of visual perception, and avoid the visual detection blind area in vehicle driving, more and more automobile manufacturers use more cameras (at present, in L3 and above automatic driving, generally 4 surround view, 6 around view, and 2 front view are used), and the resolution of each camera is also increasing (from 640x480 to 1920x1080 to 3840x2160). In addition, due to the limitation of photoelectric conversion mechanism, the camera can only produce single-color raw data arranged in a grid-like manner, and cannot produce normal RGB images that can be viewed by the human eye. Therefore, in order to produce normal viewable images, the camera raw data needs to be sent to the ISP (Image Signal Processing) module to restore it to a normal image.

[0058] However, due to the lagging development of ISP technology, the currently available ISP modules are still the traditional ISP modules used in ordinary smartphones or digital cameras, and they use traditional image processing algorithms to restore normal images. Since smartphones and digital cameras generally only have one main camera, and the screen is small, resulting in a correspondingly low image resolution, and since mobile phone cameras do not require real-time and continuous shooting (they can only take pictures one at a time), existing traditional ISPs are limited by the performance requirements of mobile phone cameras and the cost of consumer electronics products. This means that, in terms of image processing principles, they cannot achieve real-time continuous shooting (i.e., meet a certain frame rate) at high resolution by multiple cameras.

[0059] In the field of autonomous driving, due to the high speed of vehicles, it is necessary to have multiple cameras processed and restored into normal images simultaneously and continuously in real time so that machine vision modules or AI modules can perform visual target detection.

[0060] Therefore, in the process of real-time image acquisition, the existing technology suffers from the problem of low timeliness in image acquisition because traditional ISPs cannot achieve continuous shooting.

[0061] To address the aforementioned problems, the present invention provides a method, apparatus, and vehicle for improving the real-time performance of ISP, which will be described in detail below.

[0062] like Figure 1 As shown, Figure 1 A flowchart illustrating an embodiment of the method for improving ISP real-time performance provided by the present invention includes:

[0063] Step S101: Obtain ISP raw data and divide the ISP raw data into blocks to obtain multiple corresponding ISP raw data blocks, wherein the ISP raw data blocks are marked with corresponding location information;

[0064] Step S102: Classify the ISP raw data blocks according to the location information to determine the driving area block and the background area block;

[0065] Step S103: Adjust the initial ISP control parameters of the driving area block and the background area block respectively to obtain the corrected ISP control parameters;

[0066] Step S104: Process the raw ISP data based on the corrected ISP control parameters to determine the updated ISP image.

[0067] In this embodiment, first, the ISP original data is acquired, and the ISP original data is processed in blocks to obtain a plurality of ISP original data blocks corresponding thereto, wherein the ISP original data blocks are marked with corresponding position information; then, the ISP original data blocks are classified according to the position information to determine driving area blocks and background area blocks; next, the initial ISP control parameters of the driving area blocks and the background area blocks are adjusted respectively to obtain corrected ISP control parameters; finally, the ISP original data is processed based on the corrected ISP control parameters to determine an updated ISP image.

[0068] In this embodiment, by processing the ISP original data in blocks, the accuracy of the ISP control parameters of the driving area blocks is maintained, the driving area blocks related to vehicle driving are processed as the focus, the ISP control parameters of the background area blocks are reduced or processed as empty symbols, the background area blocks unrelated to vehicle driving are processed as virtual or blank, etc., the background area blocks are extracted separately, the data processing of the background area blocks is reduced, the efficiency of restoring normal images through ISP processing is improved, and the timeliness of image acquisition is improved.

[0069] As a preferred embodiment, in step S101, in order to process the ISP original data in blocks, first, a minimum data block size is set in advance; then, the area where the ISP original data is located is divided into a plurality of ISP original data blocks of the same size based on the minimum data block size.

[0070] It should be noted that the minimum data block size is a regular pattern, and by equally dividing the ISP original data into multiple parts without distinction, it is beneficial for subsequent data processing.

[0071] In a specific embodiment, the minimum data block size is preferably a square or a rectangle.

[0072] In other embodiments, the shape and size of the minimum data block size can also be adjusted according to actual needs, such as a triangle, a trapezoid, etc.

[0073] Further, the ISP original data block includes a packet group and a data error correction code.

[0074] The packet group includes a packet header, the size of the packet group, the number of cameras, the position information of the ISP original data block, the size information of the ISP original data block, and the image processing effect of the ISP original data block.

[0075] It should be noted that, generally, the input ISP raw data comes from multiple cameras, and the ISP raw data is simultaneously divided into blocks, and the multiple cameras are independently and in parallel divided into blocks. Since the resolution, transmission rate and frame rate of the input camera raw data are the same, the block operation action, operation sequence, operation mode and operation speed are also the same. That is, the raw data of multiple cameras can complete the same size of block at the same time.

[0076] Further, the ISP raw data of the camera is divided into blocks due to the following reasons:

[0077] First, with the rapid development of the field of autonomous driving, the resolution of the current vehicle-mounted camera is getting larger and larger (most of the current vehicle cameras are 1920x1080, and some reach 3840x2160). Due to the limitations of the early circuit implementation mechanism, the traditional ISP used in mobile phone digital cameras only supports generating low-resolution images. In order to use the traditional ISP for real-time image processing, the complete image data with large resolution needs to be divided into blocks so that the ISP can directly process each small block image.

[0078] Second, in the field of autonomous driving, not all information on the image is concerned, only the image related to the driving area is interested, such as the current lane vehicles / pedestrians / signboards, opposite lane vehicles / pedestrians, and vehicles driving in the same direction behind the vehicle. These information only accounts for a part of the complete image. Through block division, the traditional ISP only needs to process the driving area related region in depth and comprehensively, and does not process or only simply processes other irrelevant regions. This avoids the difficulty of the traditional ISP in processing large resolution data, improves the real-time processing capability of the traditional ISP, and is also conducive to reducing the recognition complexity, improving the recognition accuracy and speed in the subsequent process of recognizing the current area.

[0079] As a preferred embodiment, in step S102, in order to determine the driving area block and the background area block, as shown in Figure 2 , Figure 2 The flowchart of an embodiment of determining the driving area block and the background area block provided by the present application comprises:

[0080] Step S121: obtaining a preset position number;

[0081] Step S122: determining, based on the position information, that the area block in the ISP raw data block matched with the preset position number is the driving area block;

[0082] Step S123: determining that the other area blocks in the ISP raw data block except the driving area block are the background area blocks;

[0083] The position number corresponds to the position information of the ISP original data block.

[0084] In this embodiment, first, a preset position number is obtained; then, based on the position information, a region block in the ISP original data block that matches the preset position number is determined as a driving region block; finally, other region blocks in the ISP original data block except the driving region block are determined as background region blocks; wherein the position number corresponds to the position information of the ISP original data block.

[0085] In this embodiment, the ISP original data block is numbered, and the preset position number corresponding to the key region is set in advance according to the actual situation and experience; the driving region block and the background region block are determined by comparing the position number in the position information, thereby effectively improving the data processing efficiency.

[0086] As a preferred embodiment, in step S121, since the position between the vehicle itself and the driving background such as the road is relatively fixed during the automatic driving of the vehicle, the preset position number can be obtained after the ISP original data block is encoded.

[0087] As a preferred embodiment, in step S103, in order to obtain the corrected ISP control parameter, the initial ISP control parameter is obtained, and the initial ISP control parameter is adjusted to obtain the corrected ISP control parameter. Figure 3 As shown in the flowchart of an embodiment of obtaining the corrected ISP control parameter provided by the present application, the initial ISP control parameter is obtained, and the initial ISP control parameter is adjusted to obtain the corrected ISP control parameter. Figure 3 As shown in the flowchart of an embodiment of obtaining the corrected ISP control parameter provided by the present application, the initial ISP control parameter is obtained, and the initial ISP control parameter is adjusted to obtain the corrected ISP control parameter.

[0088] Step S131: keeping the data error correction code corresponding to the driving region block, and determining the accurate ISP control parameter of the driving region block;

[0089] Step S132: revising the data error correction code corresponding to the background region block to be empty characters, and determining the empty ISP control parameter of the background region block.

[0090] In this embodiment, by adjusting the data error correction code of the driving region block and the background region block, the ISP control parameter of the driving region block and the background region block is corrected, and by revising the data error correction code of the background region block to be empty characters, the data of the background region block can be ignored in the subsequent data processing process, the efficiency of restoring the normal image through the ISP processing is improved, and the timeliness of image acquisition is improved.

[0091] As a preferred embodiment, the initial ISP control parameter includes brightness parameter, gamma parameter, automatic exposure parameter, white balance parameter, and other parameters related to image effect.

[0092] In a specific embodiment, the types and quantities of ISP control parameters can be adaptively adjusted according to actual needs to meet the current technical requirements as much as possible.

[0093] In a specific embodiment, the error correction code of the group package contains the error correction code of the package header, and if the data package is a relatively important image area (such as a driving road), the error correction code of the camera block data is contained, and if it is not an important image area (such as a sky white cloud), the camera block data error correction code is not contained, but the "null" is added to fill the space of the error correction code to maintain the pipeline performance of the synchronization processing. In this way, each group package is an independent data unit, which can ensure that each important image data can be correctly processed when sent to the ISP for processing, and can ensure that the image that does not care is processed quickly.

[0094] As a preferred embodiment, in step S104, in order to determine the updated ISP image, as shown in the following formula: Figure 4 Figure 4 The flowchart of an embodiment of determining the updated ISP image provided by the present application comprises:

[0095] Step S141: performing image processing on the driving area block based on the accurate ISP control parameter to determine the driving image block corresponding to the driving area block;

[0096] Step S142: performing image processing on the background area block based on the empty ISP control parameter to determine the background image block corresponding to the background area block;

[0097] Step S143: fusing and splicing the driving image block and the background image block to determine the updated ISP image.

[0098] In this embodiment, first, the driving area block is processed based on the accurate ISP control parameter to determine the driving image block corresponding to the driving area block; then, the background area block is processed based on the empty ISP control parameter to determine the background image block corresponding to the background area block; finally, the driving image block and the background image block are fused and spliced to determine the updated ISP image.

[0099] In this embodiment, by separately processing the driving area block and the background area block, the corresponding driving image block and the background image block are obtained, and the updated ISP image is determined by fusing and splicing, which ensures the integrity of the ISP image and avoids data loss in the subsequent data processing process.

[0100] As a preferred embodiment, in step S141, in order to improve the image data processing accuracy, the ISP original data in the driving area block can also be preprocessed before the driving area block is processed.

[0101] ​The ISP preprocessing includes one or more of the following: Sensor black level correction processing, Sensor bad pixel correction processing, Sensor noise reduction processing, Sensor lens correction processing, and Sensor flicker removal processing, or any combination thereof.

[0102] In one specific embodiment, after the raw data from multiple cameras is simultaneously segmented, they are integrated and arranged sequentially, specifically including:

[0103] First: Input raw data from multiple cameras simultaneously;

[0104] Second: Divide the input multi-camera data into blocks; such as... Figure 5 As shown, Figure 5 This is a schematic diagram of the first two steps of an embodiment of ISP processing of four sets of raw data provided by the present invention.

[0105] Third: Arrange the multi-camera data blocks sequentially and continuously; for example... Figure 6 As shown, Figure 6 This is a schematic diagram of the third step in an embodiment of ISP processing of four sets of raw data provided by the present invention.

[0106] Fourth: Real-time preprocessing by multi-camera ISP; such as Figure 7 As shown, Figure 7 This is a schematic diagram of the fourth step in an embodiment of the present invention for performing ISP processing on four sets of raw data.

[0107] Fifth: Continuously process segmented data from multiple cameras using a traditional ISP module; such as... Figure 8 As shown, Figure 8 This is a schematic diagram of the fifth step in an embodiment of the present invention for performing ISP processing on four sets of raw data.

[0108] Sixth: Stitching and fusing segmented images from multiple cameras' ISPs; such as... Figure 9 As shown, Figure 9 This is a schematic diagram of the sixth step in an embodiment of the present invention for performing ISP processing on four sets of raw data.

[0109] Seventh: Simultaneously generate normal and complete images from multiple cameras; such as Figure 10 As shown, Figure 10 This is a schematic diagram of the seventh step in an embodiment of the present invention for performing ISP processing on four sets of raw data.

[0110] By dividing the data of each camera into blocks and interleaving them sequentially to form a data group, the 0th data group is output at time T0, the 1st data group is output at time T1, and the corresponding nth data group is output at each time Tn.

[0111] For example: the raw data block (0,0) from camera 1 -> raw data block (0,0) from camera 2 -> raw data block (0,0) from camera 3 -> raw data block (0,0) from camera 4 constitutes data group 0 output. The subsequent raw data block (0,1) from camera 1 -> raw data block (0,1) from camera 2 -> raw data block (0,1) from camera 3 -> raw data block (0,1) from camera 4 constitutes data group 1 output. This process continues until all data groups are output to a traditional ISP for processing.

[0112] This arrangement allows the ISP to apply different processing effects to each image segment. Since not all areas in images captured during autonomous driving are of interest, the ISP focuses only on the lower central segment relevant to the driving road, ignoring other segments (such as the sky, clouds, buildings, and trees). This allows the ISP to perform targeted, in-depth processing on road-related image segments, resulting in better processing, clearer images, and richer details, which is more beneficial for subsequent AI detection modules to perform object detection (detecting vehicles, cyclists, pedestrians, signs, traffic lights, etc.). Image segments that are not of interest to the ISP are simply processed or left unprocessed.

[0113] Traditional image processing ISPs (Internet Service Providers) face a challenge: at a given resolution, higher-quality, more detailed image processing consumes more processing time and resources (such as storage space, bus bandwidth, computational logic, software configuration, and parameter scheduling). Since the total resources of an electronic system are finite, the increased resource requirements of the ISP actually slow down the system's ability to provide those resources, leading to longer processing times and poorer real-time performance. Conversely, simpler image processing by the ISP results in faster processing, requires fewer resources, and achieves better real-time performance.

[0114] Therefore, this application achieves a good balance between processing effect and processing time by performing more detailed ISP processing only on the image areas of interest and simple processing on the image areas of no interest. This reduces processing time and makes the process more real-time while achieving the desired ISP image effect.

[0115] Furthermore, in order to intuitively demonstrate the technical effects of this technical solution, such as... Figure 11 As shown, Figure 11A result schematic diagram of an embodiment of processing an image according to the traditional ISP technology provided by the present application is shown in Fig. 1. Figure 12 As shown in Fig. 1, Figure 12 A result schematic diagram of an embodiment of processing an image according to the technical solution of the present application provided by the present application is shown in Fig. 2.

[0116] By comparing the two result schematic diagrams, it can be obviously found that the image obtained according to the technical solution of the present application retains the content related to the vehicle and the driving of the vehicle, and the irrelevant part is processed to be blurred, thereby reducing the data processing amount and improving the efficiency of restoring the normal image as a whole.

[0117] In the above manner, by performing the block processing on the ISP original data, the accuracy of the ISP control parameters of the driving area block is maintained, the driving area block related to the driving of the vehicle is processed as the focus, the ISP control parameters of the background area block are reduced or processed as empty symbols, the background area block irrelevant to the driving of the vehicle is processed as blurred or blank, etc., by extracting the background area block separately, the data processing on the background area block is reduced, thereby improving the efficiency of restoring the normal image through the ISP processing as a whole and improving the timeliness of image acquisition.

[0118] In order to solve the above problems, the present application further provides a device for improving the real-time performance of ISP, as shown in Fig. 3, Figure 13 As shown in Fig. 3, Figure 13 A structure schematic diagram of an embodiment of the device for improving the real-time performance of ISP provided by the present application, the device for improving the real-time performance of ISP 1300 comprises:

[0119] An ISP original data block acquisition module 1301 is configured to acquire ISP original data, perform block processing on the ISP original data, and obtain a plurality of ISP original data blocks corresponding thereto, wherein the ISP original data blocks are marked with corresponding position information;

[0120] A classification module 1302 is configured to classify the ISP original data blocks according to the position information, and determine driving area blocks and background area blocks;

[0121] An ISP control parameter correction module 1303 is configured to adjust initial ISP control parameters of the driving area blocks and the background area blocks respectively, and obtain corrected ISP control parameters;

[0122] An updated ISP image determination module 1304 is configured to process the ISP original data based on the corrected ISP control parameters, and determine an updated ISP image.

[0123] The application also provides a vehicle comprising a processor and a memory, wherein the memory stores a computer program which, when executed by the processor, implements the method for improving ISP real-time performance as described above. The memory stores the program for improving ISP real-time performance.

[0124] The memory can be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device in some embodiments. The memory can also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. of the computer device in other embodiments. Further, the memory can include both the internal storage unit and the external storage device of the computer device. The memory is used to store application software and various data installed on the computer device, such as program codes installed on the computer device. The memory can also be used to temporarily store data that has been output or will be output. In an embodiment, the program for improving ISP real-time performance can be executed by the processor, thereby implementing the method for improving ISP real-time performance of the embodiments of the application.

[0125] The processor can be a central processing unit (CPU), a microprocessor or other data processing chip in some embodiments, which is used to run program codes or process data stored in the memory, such as executing the program for improving ISP real-time performance.

[0126] Further, the application also provides an electronic device for real-time ISP processing of multi-camera data, as shown in Figure 14 The structure schematic diagram of an embodiment of the electronic device provided by the application is shown in Figure 14 The structure schematic diagram of an embodiment of the electronic device provided by the application is shown in

[0127] The electronic device comprises a system control unit 100, a camera signal interface unit 101, a camera data synchronization unit 102, a camera data blocking unit 103, a camera blocking group packaging unit 104, an ISP blocking preprocessing unit 105, an ISP real-time processing unit 106, a blocking ISP fusion splicing unit 107, and a multi-camera image real-time output unit 108. The operation of each unit is described as follows.

[0128] After the system is powered on, the system control unit first initializes other functional units, completes relevant configurations, and performs detection. When each unit is running normally, raw data input from the camera is started. The mipi signals generated by each camera are sent to the camera signal interface unit for independent parallel and simultaneous mipi deserialization to recover the camera raw data. These raw data are respectively cached to the camera data synchronization unit. Due to the differences in camera characteristics, start time, and transmission jitter, the data is written in sequence when written to the cache. After caching, it is read out at the same clock to meet the needs of subsequent unit processing. In this way, multiple synchronized camera raw data are output.

[0129] The synchronized camera raw data are respectively sent to the camera data blocking unit, where camera data blocking processing is simultaneously and in parallel performed. After the blocking data is processed, it is stored in the respective cache for subsequent unit processing.

[0130] The camera blocking packet unit reads the blocking data from the cache of the previous unit and sends it to the unit for grouping operation of multiple camera data according to the method, generates the corresponding packet header and error correction code, forms a continuous data packet, and converts the multiple camera data into a group packet data.

[0131] When the blocking packet data is sent to the ISP blocking preprocessing unit, ISP preprocessing operation is performed, that is, the blocking data is converted into ISP preprocessing data, and the corresponding blocking ISP control parameters, ISP correction parameters of each camera, and packet header and check code are generated according to the ISP requirements of the blocking data. After completion, it is sent to the cache.

[0132] The traditional ISP function module in the ISP real-time processing unit performs ISP image processing according to the preprocessing data and configuration parameters. The configuration information and intermediate calculation data during operation are respectively saved to the respective cache.

[0133] The processed ISP blocking image is sent to the blocking ISP fusion splicing unit, which splits and restores multiple camera images. The blocking image and blocking information of each camera are respectively and independently cached and then sent to the splicing and fusion module of each camera to realize a complete image. Each module has its own calculation cache and runs independently without affecting each other.

[0134] The complete image spliced by each camera is sent to the multi-camera real-time output unit, where two caches are used for row caching of the image of each camera. The cache read-write ping-pong operation mode is adopted to continuously write and read the cache. In this way, each camera simultaneously and in real time outputs a complete and normal image.

[0135] In a specific embodiment, four cameras are preferred for data acquisition.

[0136] In other embodiments, the number of cameras can also be adjusted according to actual needs.

[0137] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database, or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0138] The above description is only a preferred embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed by the present application can be easily thought by those skilled in the art, and should be covered within the protection scope of the present application.

Claims

1. A method for improving the real-time performance of an ISP, applied to automatic driving of a vehicle, characterized in that, The method comprises the following steps: acquiring ISP original data and performing block processing on the ISP original data to obtain a plurality of ISP original data blocks corresponding thereto, wherein the ISP original data blocks are marked with corresponding position information, and the ISP original data blocks comprise a packet group, a data error correction code; wherein the packet group comprises a packet header, the size of the packet group, the number of cameras, the position information of the ISP original data blocks, the size information of the ISP original data blocks, and the image processing effect of the ISP original data blocks; classifying the ISP original data blocks according to the position information to determine driving area blocks and background area blocks, comprising: acquiring a preset position number; determining, based on the position information, that an area block in the ISP original data blocks that matches the preset position number is the driving area block; determining that other area blocks in the ISP original data blocks except the driving area block are background area blocks; wherein the position number corresponds one-to-one to the position information of the ISP original data blocks; adjusting initial ISP control parameters of the driving area blocks and the background area blocks respectively to obtain corrected ISP control parameters; processing the ISP original data based on the corrected ISP control parameters to determine updated ISP images.

2. The method for improving ISP real-time performance according to claim 1, wherein, The block processing on the ISP original data to obtain a plurality of ISP original data blocks corresponding thereto comprises: pre-setting a minimum data block size; dividing an area where the ISP original data is located into a plurality of ISP original data blocks of the same size based on the minimum data block size.

3. The method for improving ISP real-time performance according to claim 1, wherein, The adjusting of the initial ISP control parameters of the driving area blocks and the background area blocks respectively to obtain corrected ISP control parameters comprises: maintaining the data error correction code corresponding to the driving area blocks to determine accurate ISP control parameters of the driving area blocks; revising the data error correction code corresponding to the background area blocks to be empty characters to determine empty ISP control parameters of the background area blocks.

4. The method for improving ISP real-time performance according to claim 3, characterized in that, The processing of the ISP original data based on the corrected ISP control parameters to determine updated ISP images comprises: performing image processing on the driving area blocks based on the accurate ISP control parameters to determine driving image blocks corresponding to the driving area blocks; performing image processing on the background area blocks based on the empty ISP control parameters to determine background image blocks corresponding to the background area blocks; fusing and splicing the driving image blocks and the background image blocks to determine updated ISP images.

5. The method for improving ISP real-time performance according to claim 4, characterized in that, The image processing on the driving area blocks based on the accurate ISP control parameters further comprises: performing ISP preprocessing on the ISP original data in the driving area blocks; wherein the ISP preprocessing comprises Sensor black level correction processing, Sensor bad point correction processing, Sensor bottom noise removal processing, Sensor lens correction processing, and Sensor Flick removal processing.

6. The method for improving ISP real-time performance according to claim 3, wherein, The initial ISP control parameters include brightness parameters, gamma parameters, automatic exposure parameters, white balance parameters, and other parameters related to image effects.

7. An apparatus for improving ISP real-time, applied to automatic driving of a vehicle, comprising: The method comprises the following steps: An ISP original data block acquisition module is configured to acquire ISP original data and perform block processing on the ISP original data to obtain a plurality of ISP original data blocks corresponding thereto, wherein the ISP original data blocks are marked with corresponding position information, and the ISP original data blocks comprise a packet group and a data error correction code; wherein the packet group comprises a packet header, the size of the packet group, the number of cameras, the position information of the ISP original data blocks, the size information of the ISP original data blocks, and the image processing effect of the ISP original data blocks; A classification module is configured to classify the ISP original data blocks according to the position information to determine driving area blocks and background area blocks, comprising: acquiring a preset position number; determining, based on the position information, that an area block in the ISP original data blocks that matches the preset position number is the driving area block; determining that other area blocks in the ISP original data blocks other than the driving area block are background area blocks; wherein the position number corresponds one-to-one to the position information of the ISP original data blocks; An ISP control parameter correction module is configured to adjust initial ISP control parameters of the driving area blocks and the background area blocks respectively to obtain corrected ISP control parameters; An updated ISP image determination module is configured to process the ISP original data based on the corrected ISP control parameters to determine an updated ISP image.

8. A vehicle characterized by comprising: The method comprises a processor and a memory, and the memory stores a computer program, and the computer program is executed by the processor to implement the method for improving the real-time performance of ISP according to any one of claims 1-6.

Images