Image processing method, readable medium and electronic device

Abstract

The present application relates to an image processing method, a readable medium and an electronic device. The method comprises: obtaining image features of an image to be processed and an image of a previous frame of the image to be processed; determining the segmentation point corresponding to the image to be processed according to the image features of the previous frame of the image and preset rules; segmenting the image to be processed using the determined segmentation point to obtain a plurality of sub-images to be processed; performing image preprocessing on the plurality of sub-images to be processed respectively using a plurality of image signal processors to obtain a plurality of preprocessed sub-images. The technical solution of the present application determines the segmentation point of the image to be processed by using the image features of the previous frame of the image to be processed in combination with preset rules, thereby achieving dynamic segmentation of the image to be processed, and uses a plurality of image signal processors to collaboratively process the plurality of sub-images to be processed obtained by segmentation, which can not only improve the accuracy of image processing, but also improve the real-time performance of image processing and reduce latency.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, a readable medium and an electronic device. Background Art

[0002] With the development of image processing technology, more and more image processing is involved in technical fields such as the Internet of Things and autonomous driving. As a commonly used device, the Image Signal Processor (ISP) is usually deployed in electronic devices involved in image processing, so that the electronic device can use the ISP to perform some preprocessing on the image, and then other processors use the ISP preprocessed image for post-processing such as target detection and face recognition. For example, the camera of the electronic device collects the original image, such as the Bayer format image (generally with the suffix .raw), and after a series of preprocessing by the ISP, such as black level compensation, lens shading correction, bad pixel correction, etc., it is transmitted to the central processing unit (CPU) of the electronic device for target detection.

[0003] When the amount of data of the original image to be processed is large, for example, when the resolution of the image is high, for application scenarios with high requirements for real-time image processing, such as autonomous driving application scenarios, if the real-time processing capability of the ISP is limited, it may cause a large delay in image processing and a poor user experience. Summary of the invention

[0004] In view of this, embodiments of the present application provide an image processing method, a readable medium, and an electronic device.

[0005] The technical solution of the present application determines the segmentation points of the image to be processed by utilizing the image features of the previous frame of the image to be processed in combination with preset rules, thereby realizing dynamic segmentation of the image to be processed, and utilizing multiple image signal processors to collaboratively process the multiple sub-images to be processed obtained by segmentation, which can not only improve the accuracy of image processing, but also improve the real-time performance of image processing and reduce latency.

[0006] In a first aspect, the present application provides an image processing method for an electronic device having multiple image signal processors, the method comprising:

[0007] Obtaining image features of an image to be processed and an image frame previous to the image to be processed;

[0008] Determine the segmentation point corresponding to the image to be processed according to the image features of the previous frame and preset rules;

[0009] Using the determined segmentation points to segment the image to be processed, to obtain a plurality of sub-images to be processed;

[0010] A plurality of image signal processors are used to perform image preprocessing on a plurality of sub-images to be processed respectively, so as to obtain a plurality of preprocessed sub-images.

[0011] In a possible implementation of the first aspect, the image to be processed is an image captured in real time by the electronic device.

[0012] Optionally, the image to be processed is an image captured in real time by the electronic device through a camera.

[0013] In a possible implementation of the first aspect, the time interval between the image to be processed and the previous frame image is less than a time threshold, that is, the processed image is a non-first frame image among multiple frames of images collected within a period of time.

[0014] In a possible implementation of the first aspect, the image feature includes a grayscale value distribution of the image, and determining a segmentation point corresponding to the image to be processed according to the image feature of the previous frame of image and a preset rule includes:

[0015] Pre-segment the previous frame image to obtain multiple minimum segmentation units corresponding to the previous frame image;

[0016] Count the sum of the gray values ​​of all pixels in each minimum segmentation unit;

[0017] Based on the statistical sum of the gray values ​​corresponding to each minimum segmentation unit, the segmentation point corresponding to the image to be processed is determined.

[0018] In a possible implementation of the first aspect, determining a segmentation point corresponding to the image to be processed based on the statistically calculated sum of grayscale values ​​corresponding to each minimum segmentation unit includes:

[0019] Compare the difference between the sum of the gray values ​​corresponding to each two adjacent minimum segmentation units;

[0020] The middle pixel point between two adjacent minimum segmentation units with the smallest difference in the sum of gray values ​​is used as a partial segmentation point of the image to be processed.

[0021] In a possible implementation of the first aspect, pre-segmenting the previous frame image to obtain a plurality of minimum segmentation units corresponding to the previous frame image includes:

[0022] The previous frame image is divided into a set number of parts according to the number of rows of pixel points, and multiple minimum division units corresponding to the previous frame image are obtained, and the number of pixel rows of each minimum division unit in the multiple minimum division units is the same, or the number of pixel rows of a first minimum division unit in the multiple minimum division units is less than the number of pixel rows of other minimum division units in the multiple minimum division units except the first minimum division unit.

[0023] In a possible implementation of the first aspect, the plurality of sub-images to be processed include a first sub-image to be processed and a second sub-image to be processed, the plurality of image signal processors include a first image signal processor and a second image signal processor, and

[0024] Using multiple image signal processors to perform image preprocessing on multiple sub-images to be processed respectively to obtain multiple preprocessed sub-images, including:

[0025] Preprocessing the first sub-image to be processed by using the first image signal processor, and when it is determined that the processing progress of the first image signal processor meets a preset condition, controlling the first image signal processor to send a trigger signal to the second image signal processor to notify the second image signal processor to receive data of the second sub-image to be processed;

[0026] In the case where it is determined that the second image signal processor is used to process the second sub-image to be processed, the second sub-image to be processed is pre-processed by the second image signal processor;

[0027] When it is determined that the first image signal processor has completed preprocessing of the first sub-image to be processed, the first image signal processor is controlled to output the first preprocessed sub-image, and when it is determined that the second image signal processor has completed preprocessing of the second sub-image to be processed, the second image signal processor is controlled to output the second preprocessed sub-image.

[0028] In a possible implementation of the first aspect, the method further includes:

[0029] Use multiple pre-processed sub-images to perform target detection processing to obtain target detection results, or

[0030] The face recognition processing is performed using a plurality of pre-processed sub-images to obtain a face recognition result.

[0031] In a possible implementation of the first aspect, the method further includes:

[0032] When the time interval between the image to be processed and the last frame of image processed by the electronic device is greater than or equal to the time threshold, the image to be processed is evenly divided into multiple parts from top to bottom according to the number of pixel rows to obtain multiple sub-images with the same number of pixel rows, and the number of the multiple sub-images with the same number of pixel rows is the same as the number of image signal processors in the electronic device. That is, the image to be processed is the first frame of multiple frames of images collected within a period of time.

[0033] In a possible implementation of the first aspect, the electronic device includes a central processing unit, and

[0034] The image feature of the previous frame of the image to be processed is obtained by one of the multiple image signal processors, or

[0035] The image features of the previous frame of the image to be processed are acquired by the central processing unit.

[0036] In a second aspect, the present application provides a computer-readable storage medium having instructions stored thereon, which, when executed on an electronic device, causes the electronic device to execute the image processing method in the above-mentioned first aspect and any possible implementation of the first aspect.

[0037] In a third aspect, the present application provides a computer program product, which includes instructions, and when the instructions are executed by one or more processors, they are used to implement the image processing method in the above-mentioned first aspect and any possible implementation of the first aspect.

[0038] In a fourth aspect, the present application provides an electronic device, including:

[0039] memory for storing instructions, and

[0040] One or more processors. When the instruction is executed by the one or more processors, the processors execute the image processing method in the above-mentioned first aspect and any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1 A schematic diagram of an application scenario of autonomous driving provided in an embodiment of the present application;

[0043] Figure 2A brief signal flow diagram of an image processing method provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of the hardware structure of an autonomous driving vehicle provided in an embodiment of the present application;

[0045] Figure 4 A flowchart of an image processing method provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of a flow chart of a method for dynamic image segmentation provided in an embodiment of the present application;

[0047] Figure 6 A brief schematic diagram of image segmentation provided in an embodiment of the present application;

[0048] Figure 7 A schematic flow chart of a method for collaboratively processing multiple sub-images by multiple ISPs provided in an embodiment of the present application;

[0049] Figure 8 A schematic diagram of the state timing of an ISP provided in an embodiment of the present application;

[0050] Fig. 9 A schematic diagram of the hardware structure of an ISP provided in an embodiment of the present application;

[0051] Fig.10 A schematic diagram of a process of processing image data by a general functional module in an ISP provided in an embodiment of the present application;

[0052] Fig.11 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] Illustrative embodiments of the present application include, but are not limited to, an image processing method, a readable medium, and an electronic device.

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] Let's first combine Figure 1 , introducing an application scenario of autonomous driving to which the technical solution of this application is applicable.

[0056] like Figure 1As shown, there is an autonomous driving car 100 and a plurality of passers-by in front of the driving path of the autonomous driving car 100. The autonomous driving car 100 can collect images of the surrounding environment through a camera, perform image recognition using the collected images, and adjust the driving state according to the recognition results.

[0057] For example, in some embodiments, the camera can collect the environmental image around the autonomous driving car 100 in real time during the driving process of the autonomous driving car 100, and the environmental image can include vehicle information, pedestrian flow information, road condition information (such as water accumulation and ice on the road surface), and traffic marking information on the road (such as straight and turn markings), etc. Then the collected original image is first pre-processed, such as performing black level compensation, lens shading correction, bad pixel correction, color interpolation, noise removal, white balance correction, color correction, color and contrast enhancement, automatic exposure control, etc. on the collected original image, so as to obtain an image in YUV (a color space that uses grayscale values ​​and chromaticity to represent the color of an image) or RGB (a color space that uses red, green, and blue to represent the color of an image) format. Then, the pre-processed image is post-processed, for example, the pre-processed YUV format or RGB format image is subjected to target detection by a target detection model deployed in the autonomous driving vehicle 100, so as to identify obstacles and avoid them in time. The target detection model can be obtained by training a neural network model using a large number of preset road condition sample images.

[0058] In order to ensure the safety performance of the autonomous driving car 100, the autonomous driving car 100 usually has high requirements for the real-time performance of image processing. If a single ISP is deployed in the autonomous driving car 100 to pre-process the original image captured by the camera, when the resolution of the original image is high, for example, the resolution of the original image is 100 million pixels, and a single ISP can only support real-time processing of 30 million pixels, then it is obvious that a single ISP cannot meet the functional requirements of the autonomous driving car 100.

[0059] In order to improve the real-time performance of the autonomous driving vehicle 100 regarding image processing, in some embodiments, multiple ISPs may be deployed in the autonomous driving vehicle 100, and multiple ISPs may be used to process the original image captured by the camera. Specifically, assuming that the autonomous driving vehicle 100 divides the captured original image into a fixed number of sub-images, the number of ISPs deployed in the autonomous driving vehicle 100 is the same as the number of sub-images (i.e., the set number), and each ISP processes a corresponding sub-image. For example, in Figure 2 In the illustrated embodiment, it is assumed that the autonomous driving vehicle 100 is deployed with three ISPs, namely ISP1, ISP2, and ISP3. The original image P0 captured by the camera can be fixedly divided into three sub-images, for example, sub-image P01, sub-image P02, and sub-image P03. Then, ISP1 pre-processes sub-image P01, ISP2 pre-processes sub-image P02, and ISP3 pre-processes sub-image P03. Then, the processing results of ISP1, ISP2, and ISP3 are summarized as the pre-processing result of the original image P0, and sent to the central processor of the autonomous driving vehicle 100 for target detection.

[0060] exist Figure 2 In the illustrated embodiment, the original image is segmented in a fixed manner, that is, no matter how large the original image is, the original image is segmented into the same number of parts to obtain a fixed number of sub-images, and then each ISP processes one of the sub-images. However, for different original images, the color distribution therein may be different. Assuming that a certain area in the original image with rich colors (usually the area with rich colors has a large amount of information, which means that the area contains more features) is segmented into different sub-images, and then different ISPs are used to process different sub-images, it will affect the final image processing effect, for example, it will affect the exposure effect of the image obtained after processing, and the user experience will be poor. In particular, for autonomous driving application scenarios with high real-time requirements, if the image obtained after ISP preprocessing has a poor effect, it will affect the accuracy of subsequent target detection using the target detection model on the image obtained after preprocessing, affecting the safety of autonomous driving.

[0061] To this end, the present application provides a technical solution that can be applied to any electronic device that has high requirements for real-time image processing. Multiple ISPs are deployed in the electronic device, and how to segment the current image to be processed can be determined based on the image features of the previous frame of image adjacent to the current original image to be processed. For example, how to segment the current image to be processed is determined based on the color distribution of the previous frame of image adjacent to the current original image to be processed. Dynamic segmentation of the current original image to be processed is achieved to obtain multiple dynamically segmented sub-images. Each ISP deployed in the electronic device pre-processes one of the sub-images obtained by dynamic segmentation, such as lens correction, bad pixel correction, color interpolation, noise removal, color correction, etc. The pre-processed sub-image is then post-processed by the CPU of the electronic device.

[0062] Since the technical solution of this application determines how to segment the current original image to be processed based on the image features of the previous frame of the image adjacent to the current original image to be processed, it can avoid the area with rich color information in the current original image to be processed from being segmented into different sub-images, so that the sub-images obtained by each ISP pre-processing can have better effects, and then the image processing effect can be better when the sub-images obtained by pre-processing are post-processed. It is helpful to improve the relevant functional indicators of electronic devices regarding image processing and improve user experience.

[0063] It should be noted that the technical solution of the present application can be applied to any electronic device with image processing function, including but not limited to self-driving cars, mobile phones, tablet computers, laptop computers, desktop computers, wearable devices, head-mounted displays, portable game consoles, portable music players, reader devices, etc.

[0064] For ease of explanation, the following description will continue with the electronic device being an autonomous driving car 100 as an example.

[0065] Figure 3 According to some embodiments of the present application, a hardware structure diagram of an autonomous driving vehicle 100 to which the technical solution of the present application is applicable is illustrated.

[0066] like Figure 3 As shown, the autonomous driving car 100 may include a camera 101, a display screen 102, a system on chip (SOC) 110, a CPU 115, and a memory 114.

[0067] Specifically, the camera 101 is used to convert the optical signal into an electrical signal, generate raw image data, and send the raw image data to the system on chip 110. In some embodiments, the camera 101 is used to collect environmental images during the driving process of the autonomous driving car 100.

[0068] The display screen 102 is used to display images, videos, etc. The display screen 102 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Mini-LED, Micro-LED, Micro-OLED, quantum dot light-emitting diodes (QLED), etc. In some embodiments, the autonomous driving car 100 may include one or more display screens 102.

[0069] The system on chip 110 may include multiple ISPs (e.g. Figure 3 ISP 111 , ISP 112 , ISP 113 ), memory 114 and central processing unit (CPU) 115 are shown. The multiple ISPs, memories 114 and CPU 115 may be coupled via bus 106 .

[0070] Understandably, Figure 3 The system on chip 110 shown in is only an exemplary description. Those skilled in the art should understand that in other embodiments, some components may be added or reduced, for example, a bus control unit, an interrupt management unit, a coprocessor, etc. may be added, and some components may be split or combined, ISP may be added or reduced, etc., and the embodiments of the present application are not limited thereto.

[0071] ISP is an application-specific integrated circuit (ASIC) for image data processing, and is used to pre-process the original image captured by the camera 101 to obtain better image quality. For example, if the original image is divided into three sub-images, ISP111, ISP112, and ISP113 pre-process the three sub-images respectively, and then summarize the pre-processed results and transmit them to CPU115 for processing, so that when CPU115 uses the pre-processed results to perform further image processing, the results obtained are more accurate.

[0072] CPU115 may include one or more processing units, for example, it may include a processing module or processing circuit such as a central processing unit CPU (Central Processing Unit), a graphics processor GPU (Graphics Processing Unit), a digital signal processor DSP (Digital Signal Processor), a microprocessor MCU (Micro-programmed Control Unit), an AI (Artificial Intelligence) processor, or a programmable logic device FPGA (Field Programmable Gate Array). Among them, different processing units can be independent devices or integrated in one or more processors. In some embodiments, CPU115 can be used to perform further image processing on the preprocessing results obtained after preprocessing by ISP111, ISP112, and ISP113, such as performing target detection on the preprocessed results.

[0073] The memory 114 can be used to store data, software programs and modules, and can be a volatile memory (VolatileMemory), such as a random access memory (Random-Access Memory, RAM), a double data rate synchronous dynamic random access memory (Double Data Rate Synchronous Dynamic Random Access Memory, DDRSDRAM). In some embodiments, the memory 114 can be used to store instructions, which can implement the image processing method provided in the embodiment of the present application when executed by the CPU 115. In some embodiments, the memory 114 can also be used to cache the original image data collected by the camera 101. ISP111, ISP112, and ISP113 can read the cached original image data from the memory 114.

[0074] Understandably, Figure 3 The structure of the autonomous driving car 100 shown is only an example and does not constitute a specific limitation on the autonomous driving car 100. In other embodiments, the autonomous driving car 100 may include more or fewer modules, and some modules may be combined or split, which is not limited in the embodiments of the present application.

[0075] Next, we will continue Figure 1 As an example, the autonomous driving application scenario shown in the figure is combined with Figure 3 The hardware structure block diagram of the autonomous driving vehicle 100 is shown in FIG. Figure 4 The embodiment shown provides an image processing method, Figure 4The execution subject of each step shown in the figure may be the CPU 115 of the autonomous driving vehicle 100. Specifically, Figure 4 An image processing method shown includes the following steps:

[0076] S401: Determine the original image to be processed currently.

[0077] For example, the original image to be processed is obtained by the autonomous driving car 100 collecting the surrounding environment of the driving path in real time through the camera 101. The image includes a lot of environmental information, such as vehicle information, pedestrian flow information, road condition information, and traffic marking information on the road.

[0078] S402: Determine whether the original image to be processed is the first frame image.

[0079] If yes, it indicates that the original image to be processed is the first frame image, and the first frame image can be segmented according to the preset fixed segmentation method, and the process goes to S403; if no, it indicates that the original image to be processed is not the first frame image, and the segmentation points of the original image to be processed can be determined according to the image features of the previous frame image adjacent to the original image to be processed, and the original image to be processed can be segmented, and the process goes to S404.

[0080] For example, when the self-driving car 100 collects environmental images around the driving path in real time through the camera 101, the image data first received by each ISP is generally the data of the first frame image.

[0081] S403: dividing the original image to be processed into a plurality of sub-images according to a preset fixed division method.

[0082] Since the image data transmitted by the camera 101 to each ISP is transmitted line by line from top to bottom, each ISP also receives the data of the original image line by line, and the amount of image data cannot be predicted before obtaining a frame of image. Therefore, in order to avoid the amount of original image data exceeding the real-time processing capability of a single ISP, the first frame of image needs to be fixedly divided into N parts, and the number of N is the same as the number of ISPs deployed in the autonomous driving car 100.

[0083] For example, the original image to be processed is the first frame image, the size of the first frame image is 720*1080, that is, the first frame image has 1080 rows of pixels, and a single ISP can support real-time processing of less than 500 rows of pixels. Then the first frame image can be evenly divided into 3 parts to obtain 3 sub-images, and the size of a sub-image is 720*360, that is, a sub-image has 360 rows of pixels.

[0084] S404: Determine the segmentation points of the original image to be processed based on the image features of the adjacent previous frame image, and segment the original image to be processed into a plurality of sub-images according to the determined segmentation points.

[0085] It should be noted that, since the image data is transmitted row by row according to the pixels, after the self-driving car 100 captures a frame of image through the camera 101, each ISP also receives the data of the original image row by row, and since the self-driving car 100 has high real-time requirements for image processing, the ISP cannot segment the image after receiving all the rows of data of a frame of image. Therefore, the self-driving car 100 cannot segment the current frame of image according to the features of the current frame of image. Since the contents of several adjacent images continuously captured by the camera 101 are usually similar, the image features of the previous frame of image can be used to determine the segmentation points of the current frame of image. Compared to Figure 2 In the embodiment shown, the method of performing fixed segmentation on all original images can prevent the areas with rich color information in the original images from being segmented into different sub-images, so that the sub-images obtained by pre-processing by each ISP can have better effects, and further, when the sub-images obtained by pre-processing are post-processed, the image processing effect obtained can be better. This helps to improve the relevant functional indicators of electronic devices regarding image processing and improve user experience.

[0086] It should be noted that, since the camera 101 transmits image data to the ISP line by line, the ISP also receives the sub-image data line by line. In addition, the above-mentioned segmentation of the original image to be processed and the cropping operation on the original image are performed, but in the process of receiving data by the ISP, only the corresponding sub-image data is received or processed.

[0087] The preset dynamic segmentation method will be combined below Figure 5 A detailed introduction is given and the description is not expanded here.

[0088] S405: The multiple sub-images obtained by segmentation are collaboratively processed through multiple ISPs to obtain a pre-processed image corresponding to the original image to be processed.

[0089] In some embodiments, the original image to be processed is divided into three sub-images, which are then collaboratively processed by three ISPs of the autonomous driving vehicle to obtain a pre-processed image corresponding to the original image to be processed.

[0090] Among them, the method of multiple ISPs cooperating to process multiple sub-images will be combined below Figure 7 and Figure 8 A detailed introduction is given and the description is not expanded here.

[0091] S406: Perform further image processing on the pre-processed image to obtain a final image processing result.

[0092] In some embodiments, CPU 115 executes the executable program of the target detection model to input the pre-processed image data into the target detection model for target detection to obtain the final target detection result, thereby making the target detection result more accurate.

[0093] For example, if the original image to be processed currently includes traffic marking information on the road (such as straight-ahead, turn information, etc.), the autonomous driving vehicle 100 inputs the preprocessed image into the target detection model to detect the traffic marking information, so that the vehicle can continue to go straight, turn, etc. according to the detected traffic marking information.

[0094] In other embodiments, other processing may be performed on the pre-processed image, such as face recognition, etc., which is not limited in this application.

[0095] It can be understood that the execution order of the above steps S401 to S406 is only an illustration. In other embodiments, other execution orders may be adopted, and some steps may be split or combined, which is not limited here.

[0096] The following will be combined Figure 5 The flowchart shown in the above Figure 4 The method for determining the segmentation points of the original image to be processed based on the image features of the adjacent previous frame image involved in S404 is introduced in detail. Figure 5 The execution subject of each step in the flowchart shown can be the CPU 115 of the autonomous driving vehicle 100, or one of the multiple ISPs deployed in the autonomous driving vehicle 100. The ISP is the master ISP, and the other ISPs are slave ISPs. For example, ISP 111 is the master ISP, and ISP 112 and ISP 113 are slave ISPs. Specifically, Figure 5 The image segmentation method shown includes the following steps:

[0097] S4041: Pre-segment the previous frame image adjacent to the current original image to be processed to obtain a plurality of minimum segmentation units corresponding to the previous frame image adjacent to the original image.

[0098] In some embodiments, the previous frame image adjacent to the current original image to be processed can be divided into two fixed frames according to the number of rows. n For example, the previous frame image adjacent to the current original image to be processed includes 1080 rows of pixels, and the image is fixedly divided into 32 parts, then the number of rows of the minimum division unit of the image is M=1080 / 32, which is approximately equal to 33.

[0099] S4042: Counting the grayscale values ​​of the above-mentioned multiple minimum segmentation units.

[0100] In some embodiments, the sum of the grayscale values ​​of the plurality of minimum division units is calculated, and then a grayscale histogram is created based on the sum of the grayscale values ​​of the plurality of minimum division units, wherein the horizontal axis represents the minimum division units and the vertical axis represents the sum of the grayscale values ​​corresponding to the minimum division units.

[0101] S4043: Based on the grayscale values ​​of the above-mentioned multiple minimum segmentation units obtained by counting, determine multiple segmentation points of the original image to be processed.

[0102] In some embodiments, the difference between the sum of the grayscale values ​​of each two adjacent minimum segmentation units in the previous frame of the original image to be processed can be compared, and the middle point of the two minimum segmentation units with the smallest difference can be selected as one of the segmentation points of the original image to be processed. If there are multiple pairs of adjacent minimum segmentation units with the same grayscale value difference, the middle grayscale value of all the minimum segmentation units (for example, the average grayscale value of all the minimum segmentation units) can be determined, and then the middle point of a pair of adjacent minimum segmentation units whose grayscale value mean is closest to the middle grayscale value is selected from the multiple pairs of adjacent minimum segmentation units as the final segmentation point.

[0103] For example, for Figure 6 The image H shown includes 32 minimum segmentation units, which are x1 to x32, and the grayscale value sums corresponding to these 32 minimum segmentation units are y1 to y32. The difference between the grayscale value sums of every two adjacent minimum segmentation units of these 32 minimum segmentation units is compared, and it is determined that the difference between the grayscale value sum of x4 and the grayscale value sum of x5 is the smallest, so the middle point of x4 and x5 can be used as a segmentation point. In addition, it is determined that the difference between the sum of the grayscale values ​​of x25 and the sum of the grayscale values ​​of x26 is the same as the difference between the sum of the grayscale values ​​of x29 and the sum of the grayscale values ​​of x30. The average values ​​of the sum of the grayscale values ​​of x25 and the sum of the grayscale values ​​of x26, and the average values ​​of the sum of the grayscale values ​​of x29 and the sum of the grayscale values ​​of x30 can be calculated respectively. Then, it is determined that the average value of the sum of the grayscale values ​​of x29 and the sum of the grayscale values ​​of x30 is closest to the middle grayscale value of all the minimum segmentation units. The middle point of x29 and x30 can be used as another segmentation point.

[0104] In some embodiments, the multiple sub-images obtained after segmenting the current original image to be processed using the selected segmentation points need to be within the real-time processing capability of a single ISP. For example, a single ISP can process 500 rows of pixels in real time, and the number of rows of each segmented sub-image must be less than 500 rows. If the number of rows of the sub-image segmented using the segmentation points determined above exceeds the real-time processing capability of a single ISP, it is necessary to discard the corresponding segmentation points and redetermine the segmentation points. For example, in the sub-image whose number of rows exceeds the real-time processing capability of a single ISP, the midpoint of the two smallest segmentation units with the smallest difference in the total grayscale value is selected again as a new segmentation point, and the sub-image is segmented again.

[0105] It should be noted that how to determine the multiple segmentation points of the original image to be processed based on the image features of the previous frame of the original image may depend on the actual situation, and this application does not limit this.

[0106] S4044: and dividing the original image to be processed into a plurality of sub-images according to the determined division points.

[0107] For example, in Figure 6 In the embodiment shown, the previous frame image of the original image to be processed is used to determine that the segmentation points of the original image to be processed are segmentation point 1 and segmentation point 2, and then the original image to be processed can be segmented into three sub-images H1, H2, and H3 using segmentation point 1 and segmentation point 2.

[0108] It can be understood that the execution order of the above steps S4041 to S4044 is only an illustration. In other embodiments, other execution orders may be adopted, and some steps may be split or combined, which is not limited here.

[0109] The following will Figure 7 The flowchart shown and Figure 8 As an example, for the above Figure 4 The process of the multiple ISPs involved in S405 co-processing the multiple sub-images obtained by segmentation is introduced in detail. Figure 7 The execution subject of each step in the flowchart shown can be the main ISP among the multiple ISPs of the self-driving car 100, or the CPU 115 of the self-driving car 100. Figure 7 A method for collaboratively processing multiple sub-images by multiple ISPs provided in an embodiment of the present application includes the following steps:

[0110] S4051: Select an idle ISP (ISPi) from a plurality of idle ISPs to receive data of the sub-image Pi, and pre-process the received sub-image data.

[0111] In some embodiments, the CPU 115 of the autonomous vehicle 100 can monitor the operating status of each ISP of the autonomous vehicle 100. When only one ISP is in an idle state, the idle ISP (ISPi) can be controlled to receive the data of the sub-image Pi, and the idle ISPi can pre-process the received sub-image data. When two or more ISPs are in an idle state, one ISP can be selected from them according to certain rules to receive the data of the sub-image Pi, and the received sub-image data can be pre-processed. For example, if the previous sub-image is received and processed by ISP111, the current sub-image is received and processed by ISP112; for another example, if the previous sub-image is received and processed by ISP112, the current sub-image is received and processed by ISP113; for another example, if the previous sub-image is received and processed by ISP113, the current sub-image is received and processed by ISP111. That is, when two or more ISPs are in an idle state, multiple ISPs are controlled to poll and process the corresponding sub-images.

[0112] For example, assuming that the original image to be processed is Figure 6 The image H shown, the segmentation point corresponding to the image H is Figure 6 The segmentation points 1 and 2 shown in the figure can be used to segment the image H into sub-images H1, H2, and H3. Among the ISPs 111, 112, and 113 deployed in the autonomous driving vehicle 100, only ISP 111 is in an idle state, and the data of the sub-image H1 can be received through ISP 111, and the sub-image H1 can be pre-processed.

[0113] S4052: Determine whether the current processing progress meets the preset conditions. If yes, it indicates that the current processing progress meets the preset conditions, and can trigger ISPi to notify other idle ISPs to start preparing to receive the data of sub-image Pi+1, and enter S4053; otherwise, it indicates that the current processing progress does not meet the preset conditions, and ISPi continues to process the data of sub-image Pi.

[0114] The preset condition may be that the current processing progress of ISPi is that the "redundant row" of the sub-image Pi has been processed. For example, the sub-image Pi includes 430 rows of pixels in total, and the minimum segmentation unit of the original image to be processed is 20 rows. The preset condition may be that the current processing progress of ISPi is that the 430-20th row of the sub-image Pi has been processed, that is, the last minimum segmentation unit of the sub-image Pi has been processed.

[0115] S4053: Trigger ISPi to notify other idle ISPs to start preparing to receive data of sub-image Pi+1.

[0116] That is, when the current processing progress of ISPi meets the preset conditions, ISPi is triggered to notify other idle ISPs to start preparing to receive data of sub-image Pi+1. For example, CPU115 triggers ISPi to generate signal A (trigger signal) to notify other idle ISPs to start preparing to receive data of sub-image Pi+1.

[0117] For example, when ISP111 processes the above sub-image H1, ISP112 and ISP113 have completed corresponding data processing, and both ISP112 and ISP113 enter an idle state. If the current processing progress of ISP111 meets the preset conditions, ISP111 can send a signal A to ISP112 and ISP113 to notify ISP112 and ISP113 that they can start preparing to receive data of sub-image H2.

[0118] Among them, the status sequence of each ISP can be as follows Figure 8 As shown, when an ISP is in an idle state, the ISP will wait for a trigger signal. Once the trigger signal is received and it is determined that the ISP can receive and process the data of the current sub-image, the ISP will enter the pre-processing state and pre-process the sub-image. Until the current processing progress of the ISP reaches the "redundant row", the ISP generates a trigger signal and broadcasts it to other ISPs. After the ISP completes the processing of the last row of data of the current sub-image, the ISP completes the pre-processing of the sub-image, generates an idle signal and broadcasts it to other ISPs, and the ISP enters the idle state again.

[0119] S4054: Select one from the idle ISPs except ISPi to start receiving data of the sub-image Pi+1, and pre-process the received sub-image data.

[0120] For example, if ISP112 and ISP113 are both in idle state and both ISP112 and ISP113 have received signal A sent by ISP111, then according to the polling rule, it can be decided that ISP112 starts to receive and process the data of sub-image H2.

[0121] It should be noted that, since the camera 101 transmits image data to the ISP line by line, the ISP also receives the sub-image data line by line. Since the image data processing speed of the ISP may be lower than the speed at which the ISP receives the image data, and in order to ensure the real-time performance of the ISP image data processing, the image data corresponding to T rows of pixels can be cached, for example, the image data corresponding to 2 rows (first in, first out) of pixels can be cached, so that the ISP can read the corresponding sub-image data from the cache.

[0122] It can be understood that the execution order of the above steps S4051 to S4054 is only an illustration. In other embodiments, other execution orders may be adopted, and some steps may be split or combined, which is not limited here.

[0123] It should be understood that the ISPs deployed in the same electronic device usually have the same structure. Fig. 9 The hardware structure diagram shown in the figure is taken as an example to introduce it in detail. Figure 3 The hardware structure of one of the ISPs 111 deployed in the autonomous driving car 100 is shown.

[0124] Fig. 9 According to some embodiments of the present application, a schematic diagram of the structure of ISP111 is shown. Fig. 9 As shown, ISP111 is an application-specific integrated circuit (ASIC) for image data processing, which is used to further process the image data generated by the camera 101 to obtain better image quality.

[0125] ISP111 includes a processor 1111 , an image transmission interface 1112 , a general peripheral device 1113 , and a general function module 1115 .

[0126] Among them, the processor 1111 is used for logic control and scheduling in ISP111.

[0127] The image transmission interface 1112 is used for transmitting sub-image data.

[0128] The general peripheral devices 1113 include but are not limited to: a bus for coupling the various modules of ISP111 and its controller, a bus for coupling with other devices, such as an advanced high-performance bus (AHB), which enables the ISP to communicate with other devices (such as DSP, CPU, etc.) with high performance; a watchdog unit (WATCHDOG) for monitoring the working status of the ISP.

[0129] The general function module 1115 is used to process the sub-image input to the ISP 111, including but not limited to: bad pixel correction (BPC), black level correction (BLC), automatic white balance (AWB), gamma correction (Gamma Correction), color correction (ColorCorrection), noise reduction, edge enhancement, brightness, contrast, chromaticity adjustment, etc. When the image sensor transmits the image data in RAW format to the ISP 111, it is first processed by the function module. The process of the general function module 1115 processing the image data will be described below in conjunction with Fig.10 A detailed introduction is given and no further description is given here.

[0130] Understandably, Fig. 9 The structure of ISP111 shown is only an example. Those skilled in the art should understand that it may include more or fewer modules, and some modules may be combined or split, and the embodiments of the present application are not limited thereto.

[0131] Fig.10 A schematic diagram of a process of processing image data by a general function module 1115 is shown, and the processing process is as follows:

[0132] The RAW domain processing module 1115 a performs bad pixel correction, black level compensation and automatic white balance on the image data.

[0133] The image data processed in the RAW domain is subjected to RGB interpolation to obtain image data in the RGB domain, and then the RGB domain processing module 1115b performs gamma correction and color correction on the image data in the RGB domain.

[0134] The image data processed in the RGB domain is converted into image data in the YUV domain through color domain conversion, and then the YUV domain processing module 1115c performs noise reduction, edge enhancement, and brightness / contrast / chromaticity adjustment on the image data in the YUV domain.

[0135] Fig.11 According to some embodiments of the present application, a hardware structure diagram of an electronic device 300 is shown. The electronic device 300 can execute the following operations provided by the present application, for example: Figure 4 The image processing method shown.

[0136] like Fig.11As shown, the electronic device 300 may include one or more processors 310, system memory 302, non-volatile memory (NVM) 303, input / output (I / O) device 304, communication interface 305, and system control logic 306 for coupling the processor 310, system memory 302, non-volatile memory 303, communication interface 305, and input / output (I / O) device 304. Among them: the processor 310 may include one or more single-core or multi-core processors. In some embodiments, the processor 310 may include any combination of general-purpose processors and special-purpose processors (for example, graphics processors, application processors, baseband processors, etc.). In some embodiments, the processor 310 may be used Figure 4 The illustrated embodiment provides an image processing method.

[0137] The system memory 302 is a volatile memory, such as a random-access memory (RAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), etc. The system memory is used to temporarily store data and / or instructions. For example, in some embodiments, the system memory 302 can be used to store the executable program for implementing the image processing method.

[0138] The non-volatile memory 303 may include one or more tangible, non-temporary computer-readable media for storing data and / or instructions. In some embodiments, the non-volatile memory 303 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as a hard disk drive (HDD), a compact disc (CD), a digital versatile disc (DVD), a solid-state drive (SSD), etc. In some embodiments, the non-volatile memory 303 may also be a removable storage medium, such as a secure digital (SD) memory card, etc.

[0139] In particular, the system memory 302 and the non-volatile memory 303 may respectively include: a temporary copy and a permanent copy of the instruction 307. The instruction 307 may include: a program instruction that enables the electronic device 300 to implement the image processing method provided by each embodiment of the present application when executed by at least one of the processors 310.

[0140] The input / output (I / O) device 304 may include a user interface to enable a user to interact with the electronic device 300. For example, in some embodiments, the input / output (I / O) device 304 may include an output device such as a display for displaying an insurance management system interface in the electronic device 300, and may also include an input device such as a keyboard, a mouse, and a touch screen. Product developers may interact with the electronic device 300 through the user interface and input devices such as a keyboard, a mouse, and a touch screen.

[0141] The communication interface 305 may include a transceiver for providing a wired or wireless communication interface for the electronic device 300, and then communicating with any other suitable device through one or more networks. In some embodiments, the communication interface 305 may be integrated into other components of the electronic device 300, for example, the communication interface 305 may be integrated into the processor 310. In some embodiments, the electronic device 300 can communicate with other devices through the communication interface 305.

[0142] System control logic 306 may include any suitable interface controller to provide any suitable interface with other modules of electronic device 300. For example, in some embodiments, system control logic 306 may include one or more memory controllers to provide interfaces to system memory 302 and non-volatile memory 303.

[0143] In some embodiments, at least one of the processors 310 may be packaged together with the logic of one or more controllers for the system control logic 306 to form a system in package (SiP). In other embodiments, at least one of the processors 310 may also be integrated with the logic of one or more controllers for the system control logic 306 on the same chip to form a system-on-chip (SoC).

[0144] It can be understood that the electronic device 300 can be any electronic device that can execute the image processing method provided by the present application, including but not limited to self-driving cars, mobile phones, tablet computers, handheld computers, etc., and the embodiments of the present application are not limited thereto.

[0145] It is understood that the structure of the electronic device 300 shown in the embodiment of the present application does not constitute a specific limitation on the electronic device 300. In other embodiments of the present application, the electronic device 300 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0146] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0147] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0148] The program code can be implemented in a high-level programming language or an object-oriented programming language to communicate with the processing system, including but not limited to OpenCL, C language, C++, Java, etc. As for languages ​​such as C++ and Java, since they convert the storage, there will be some differences in the application of the image processing method in the embodiment of the present application. Those skilled in the art can make changes based on the specific high-level language without departing from the scope of the embodiment of the present application.

[0149] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, floppy disks, optical disks, optical disks, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Therefore, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0150] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0151] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0152] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0153] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. An image processing method for an electronic device having a plurality of image signal processors, characterized in that: The method comprises: Obtaining grayscale value distribution of an image to be processed and a previous frame of the image to be processed; Pre-segmenting the previous frame image to obtain a plurality of minimum segmentation units corresponding to the previous frame image; According to the gray value distribution, the sum of the gray values ​​of all pixels in each minimum segmentation unit is counted; Determining the segmentation point corresponding to the image to be processed based on the sum of the grayscale values ​​corresponding to each minimum segmentation unit obtained by counting; Using the determined segmentation points to segment the image to be processed to obtain a plurality of sub-images to be processed; The plurality of image signal processors are used to perform image preprocessing on the plurality of sub-images to be processed respectively, to obtain a plurality of preprocessed sub-images.

2. The method according to claim 1, characterized in that: The image to be processed is an image collected by the electronic device in real time.

3. The method according to claim 1, characterized in that The time interval between the image to be processed and the previous frame image is less than a time threshold.

4. The method according to claim 3, characterized in that The step of determining the segmentation point corresponding to the image to be processed based on the sum of the grayscale values ​​corresponding to each minimum segmentation unit obtained by counting, comprises: Compare the difference between the sum of the grayscale values ​​corresponding to each two adjacent minimum segmentation units; The middle pixel point between the two adjacent minimum segmentation units where the difference between the sum of the gray values ​​is the smallest is used as part of the segmentation points of the image to be processed.

5. The method according to claim 4, characterized in that The pre-segmenting the previous frame image to obtain a plurality of minimum segmentation units corresponding to the previous frame image includes: The previous frame image is divided into a set number of parts according to the number of rows of pixel points, and multiple minimum division units corresponding to the previous frame image are obtained, and the number of pixel rows of each of the multiple minimum division units is the same, or the number of pixel rows of a first minimum division unit among the multiple minimum division units is less than the number of pixel rows of other minimum division units among the multiple minimum division units except the first minimum division unit.

6. The method according to claim 5, characterized in that The plurality of sub-images to be processed include a first sub-image to be processed and a second sub-image to be processed, the plurality of image signal processors include a first image signal processor and a second image signal processor, and The using the multiple image signal processors to perform image preprocessing on the multiple sub-images to be processed respectively to obtain multiple preprocessed sub-images includes: Preprocessing the first sub-image to be processed by using the first image signal processor, and when it is determined that the processing progress of the first image signal processor meets a preset condition, controlling the first image signal processor to send a trigger signal to the second image signal processor to notify the second image signal processor to receive data of the second sub-image to be processed; In a case where it is determined that the second image signal processor is used to process the second sub-image to be processed, preprocessing the second sub-image to be processed by the second image signal processor; When it is determined that the first image signal processor has completed preprocessing of the first sub-image to be processed, the first image signal processor is controlled to output a first preprocessed sub-image; and when it is determined that the second image signal processor has completed preprocessing of the second sub-image to be processed, the second image signal processor is controlled to output a second preprocessed sub-image.

7. The method according to claim 6, characterized in that Also includes: Performing target detection processing using the multiple pre-processed sub-images to obtain a target detection result, or The multiple pre-processed sub-images are used to perform face recognition processing to obtain a face recognition result.

8. The method according to claim 7, characterized in that Also includes: When the time interval between the image to be processed and the previous frame image processed by the electronic device is greater than or equal to the time threshold, the image to be processed is evenly divided into multiple parts from top to bottom according to the number of pixel rows to obtain multiple sub-images with the same number of pixel rows, and the number of the multiple sub-images with the same number of pixel rows is the same as the number of image signal processors in the electronic device.

9. The method according to claim 8, characterized in that The electronic device includes a central processing unit, and The image feature of the previous frame of the image to be processed is obtained by one of the multiple image signal processors, or The image features of the previous frame of the image to be processed are acquired by the central processing unit.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on an electronic device, enable the electronic device to execute the image processing method according to any one of claims 1 to 9.

11. A computer program product, characterized in that The computer program product comprises instructions, which, when executed by one or more processors, are used to implement the image processing method according to any one of claims 1 to 9.

12. An electronic device, characterized in that: include: memory for storing instructions, and One or more processors, when the instructions are executed by the one or more processors, the processors perform the image processing method according to any one of claims 1-9.

Images