Image processing method, device, electronic device and computer readable medium

Abstract

The present invention provides an image processing method, device, electronic device and computer-readable medium, which relate to the technical field of image processing, including: obtaining a main camera image to be processed and a secondary camera image to be processed, and determining the main camera image and the secondary camera image as the images to be processed respectively; calculating the cost cube of the image to be processed to obtain a target cost cube, wherein the target cost cube represents the matching cost value of the pixel point in the image to be processed at each preset disparity level; determining a filtering window for each pixel point in the image to be processed, and filtering the cost map at each preset disparity level in the target cost cube through the filtering window; calculating the disparity map of the main camera image and the secondary camera image based on the target cost cube after filtering. The present application alleviates the technical problem that the existing cost aggregation algorithm is difficult to support large-size and large-disparity level calculation on mobile devices with poor performance.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to an image processing method, device, electronic device and computer-readable medium. Background Art

[0002] With the rapid development of software, hardware and algorithm industries, 3D reconstruction algorithms in the field of computer vision are gradually applied in many fields, such as intelligent robots, mobile phone camera applications, etc. Therefore, the binocular stereo matching algorithm for scene depth estimation based on the main and secondary images collected by dual cameras in 3D reconstruction has been studied by many scholars and applied by many industries. The conventional binocular stereo matching algorithm process usually includes four modules: cost cube calculation of the main and secondary images, cost aggregation, optimal disparity calculation, and disparity post-processing. As a module that enhances the reliability of the cost cube, cost aggregation plays a very important role in the calculation of the disparity map.

[0003] There are two common cost aggregation methods: local method and global method. The local method is implemented by weighted averaging (i.e. filtering) the cost values ​​of other points within a certain window radius of the current point. The local method can be regarded as using an edge-preserving smoothing filter to denoise the cost cube. However, with the increase of the disparity level d and the increase of the image size, the performance of the local method has also become a bottleneck for implementation on mobile devices. The global method usually reflects the idea of ​​cost aggregation in the form of energy terms in the optimization of the energy function, and obtains the final disparity map by optimizing the energy function. This method is difficult to use on mobile devices. Summary of the invention

[0004] In view of this, an object of the present invention is to provide an image processing method, apparatus, electronic device and computer-readable medium to alleviate the technical problem that the existing cost aggregation algorithm is difficult to support large-size and large-parallax layer calculations on mobile devices with poor performance.

[0005] In a first aspect, an embodiment of the present invention provides an image processing method, comprising: obtaining a main camera image to be processed and a secondary camera image to be processed, and respectively determining the main camera image and the secondary camera image as images to be processed; calculating the cost cube of the image to be processed to obtain a target cost cube, wherein the target cost cube represents a cost map of a pixel point in the image to be processed at each preset disparity level; determining a filtering window for each pixel point in the image to be processed, and filtering the cost map at each preset disparity level in the target cost cube through the filtering window; and calculating the disparity map of the main camera image and the secondary camera image based on the filtered target cost cube.

[0006] Furthermore, the target cost cube includes: a first cost cube corresponding to the main camera image and a second cost cube corresponding to the secondary camera image; calculating the cost cube of the image to be processed to obtain the target cost cube includes: performing a stereo correction operation on the image to be processed to transform the image to be processed to a space parallel to the optical axis; calculating the cost cube of the image to be processed after the stereo correction operation to obtain the first cost cube and the second cost cube.

[0007] Furthermore, calculating the cost cube of the image to be processed after the stereo correction operation to obtain the first cost cube and the second cost cube includes: calculating the cost cube of the image to be processed after the stereo correction operation by a target cost calculation method to obtain the first cost cube and the second cost cube, wherein the target cost calculation method includes any one of the following: SAD image matching algorithm, Census similarity algorithm.

[0008] Furthermore, determining the filtering window of each pixel in the image to be processed includes: determining the filtering window corresponding to each pixel based on the color position relationship between the pixels of the image to be processed.

[0009] Furthermore, determining the filter window corresponding to each pixel point based on the color position relationship between the pixel points of the image to be processed includes: performing a spatial domain transformation on the image to be processed to obtain a spatial domain transformation result; determining the color position relationship between each pixel point and each pixel point's neighboring pixel points based on the spatial domain transformation result, wherein the color position relationship includes: color information between pixel points and spatial information between pixel points; calculating the similarity value between each pixel point and each pixel point's neighboring pixel point based on the color position relationship, and determining the window radius of the filter window based on the similarity value; determining the filter window based on the window radius of the filter window.

[0010] Further, filtering the cost map at each preset disparity level in the target cost cube through the filtering window includes: performing mean filtering on the cost map at each preset disparity level in the target cost cube through the filtering window corresponding to each pixel point.

[0011] Further, calculating the disparity map of the main image and the secondary image based on the target cost cube after filtering processing includes: determining the disparity value of each pixel in the main image and the secondary image based on the target cost cube after filtering processing; and determining the disparity map of the main image and the secondary image based on the disparity value of each pixel.

[0012] Furthermore, the cost map in the target cost cube includes a matching cost value for each pixel in the image to be processed at each preset disparity level; determining the disparity value of each pixel in the primary image and the secondary image based on the target cost cube after filtering includes: determining the matching cost value for each pixel in the primary image and the secondary image at each preset disparity level in the target cost cube; and determining the minimum matching cost value among the determined matching cost values ​​as the disparity value of the corresponding pixel.

[0013] Furthermore, the method further comprises: optimizing the disparity map according to a preset optimization algorithm to obtain the optimized disparity map, wherein the preset optimization algorithm comprises at least one of the following: Figure 1 Consistency detection, occlusion area detection, and edge-preserving smoothing optimization.

[0014] In a second aspect, an embodiment of the present invention provides an image processing device, comprising: an acquisition unit, used for a main camera image to be processed and a secondary camera image to be processed, and respectively determining the main camera image and the secondary camera image as images to be processed; a first calculation unit, used to calculate the cost cube of the image to be processed to obtain a target cost cube, wherein the target cost cube represents a cost map of a pixel point in the image to be processed at each preset disparity level; a determination unit, used to determine a filtering window for each pixel point in the image to be processed; a filtering unit, used to filter the cost map at each preset disparity level in the target cost cube through the filtering window; and a second calculation unit, used to calculate the disparity map of the main camera image and the secondary camera image based on the filtered target cost cube.

[0015] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a storage device, a processing device, and a computer program stored on the storage device and executable on the processing device, wherein the processing device implements the steps of any one of the methods described in the first aspect when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium having a non-volatile program code executable by a processing device, wherein the program code enables the processing device to execute the steps of any one of the methods described in the first aspect.

[0017] In an embodiment of the present invention, first, a main camera image to be processed and a secondary camera image to be processed are obtained, and the main camera image and the secondary camera image are respectively determined as images to be processed, and then, the cost cube of the image to be processed is calculated to obtain a target cost cube; next, a filtering window for each pixel in the image to be processed is determined, and the cost map at each preset disparity level in the target cost cube is filtered through the filtering window; and the disparity map of the main camera image and the secondary camera image is calculated based on the target cost cube after filtering. It can be seen from the above description that a corresponding filtering window is determined for each pixel in the image to be processed, so when filtering is performed through the filtering window, the cost map at each preset disparity level can be filtered according to the filtering window. Therefore, the cost map at all preset disparity levels only needs to calculate the filtering window once. This method can improve the performance of the image processing method and increase the possibility of using large disparity levels on some platforms with poor performance, thereby alleviating the technical problem that the existing cost aggregation algorithm is difficult to support large-size large disparity level calculation on mobile devices with poor performance.

[0018] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0019] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 is a schematic diagram of an electronic device according to an embodiment of the present invention;

[0022] Figure 2 is a flow chart of an image processing method according to an embodiment of the present invention;

[0023] Figure 3 is a schematic diagram of a main image and a secondary image before correction according to an embodiment of the present invention;

[0024] Figure 4is a schematic diagram of the optical axis relationship between the main camera device and the secondary camera device before correction according to an embodiment of the present invention;

[0025] Figure 5 is a schematic diagram of a main image and a secondary image after correction according to an embodiment of the present invention;

[0026] Figure 6 is a schematic diagram of the optical axis relationship between the main camera device and the secondary camera device after correction according to an embodiment of the present invention;

[0027] Figure 7 is a schematic diagram of cost cube calculation according to an embodiment of the present invention;

[0028] Figure 8 is a schematic diagram of a cost cube structure according to an embodiment of the present invention;

[0029] Fig. 9 is a schematic diagram of a principle of mean filtering processing according to an embodiment of the present invention;

[0030] Fig.10 is a schematic diagram of the relationship between a preset parallax level and a matching cost value according to an embodiment of the present invention;

[0031] Fig.11 is a flowchart of another image processing method according to an embodiment of the present invention;

[0032] Fig.12 is a schematic diagram of an image processing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Embodiment 1:

[0035] First, refer to Figure 1 An electronic device 100 for implementing an embodiment of the present invention is described. The electronic device can be used to execute the image processing method of each embodiment of the present invention.

[0036] like Figure 1As shown, the electronic device 100 includes one or more processing devices 102, one or more storage devices 104, an input device 106, an output device 108, and an image acquisition device 110, and these components are interconnected through a bus system 112 and / or other forms of connection mechanisms (not shown). It should be noted that Figure 1 The components and structures of the electronic device 100 shown are merely exemplary and non-limiting. The electronic device may also have other components and structures as required.

[0037] The processing device 102 can be implemented in at least one hardware form of a digital signal processing device (DSP), a field programmable gate array (FPGA), a programmable logic array (PLA) and an ASIC (Application Specific Integrated Circuit). The processing device 102 can be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device 100 to perform desired functions.

[0038] The storage device 104 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile storage devices and / or non-volatile storage devices. The volatile storage device may include, for example, a random access storage device (RAM) and / or a cache storage device (cache), etc. The non-volatile storage device may include, for example, a read-only storage device (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processing device 102 may run the program instructions to implement the client functions (implemented by the processing device) and / or other desired functions in the embodiments of the present invention described below. Various applications and various data, such as various data used and / or generated by the application, may also be stored in the computer-readable storage medium.

[0039] The input device 106 may be a device used by a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.

[0040] The output device 108 may output various information (eg, images or sounds) to the outside (eg, a user), and may include one or more of a display, a speaker, and the like.

[0041] The image collector 110 is used to collect the image to be recognized, wherein the data collected by the camera is processed by the image processing method to obtain the disparity values ​​of the pixels in the image to be processed.

[0042] Embodiment 2:

[0043] According to an embodiment of the present invention, an embodiment of an image processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0044] Figure 2 is a flowchart of an image processing method according to an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0045] Step S202, obtaining a main camera image to be processed and a secondary camera image to be processed, and respectively determining the main camera image and the secondary camera image as images to be processed, that is, in the present application, the images to be processed include the main camera image to be processed and the secondary camera image to be processed.

[0046] In the present application, the electronic device described in the above-mentioned embodiment 1 may be a mobile terminal, which includes multiple camera devices, for example, a main camera device and a secondary camera device. Then, the main camera image to be processed is the image taken by the main camera device, and the secondary camera image to be processed is the image taken by the secondary camera device.

[0047] It should be noted that a mobile device generally includes a main camera device and at least one secondary camera device. This application does not specifically limit the number of main camera devices and secondary camera devices.

[0048] Step S204, calculating the cost cube of the image to be processed to obtain a target cost cube, wherein the target cost cube represents a cost map of pixels in the image to be processed at each preset disparity level, and the target cost cube contains a matching cost value of each pixel in the image to be processed at each preset disparity level in the cost map at each preset disparity level.

[0049] The calculation of the cost cube mainly calculates the similarity between the pixel point of the primary image at (x, y) and the pixel point of the secondary image at (x+d, y) (ie, the matching cost value in this application), where d is the number of preset parallax levels.

[0050] In the present application, after obtaining the main image and the secondary image, the cost calculation can be performed on the main image and the secondary image respectively. For example, the similarity between the pixel point (x, y) in the main image and the pixel point (x, y) to the pixel point (x+d, y) in the secondary image can be calculated respectively, so as to obtain the target cost cube.

[0051] Step S206 , determining a filtering window for each pixel in the image to be processed, and performing filtering processing on the cost map at each preset disparity level in the target cost cube through the filtering window.

[0052] In the present application, a filtering window is determined for each pixel point (x, y) of the image to be processed, and the image to be filtered at each preset disparity level in the target cost cube (i.e., the cost map at each preset disparity level) is filtered through the filtering window.

[0053] Step S208 : calculating the disparity map of the primary image and the secondary image based on the filtered target cost cube.

[0054] In the present application, disparity calculation can be performed on the main camera image to be processed and the auxiliary camera image to be processed to obtain corresponding disparity maps (ie, left and right disparity maps), and then a final disparity map can be determined based on the left and right disparity maps.

[0055] From the above description, it can be seen that a corresponding filtering window is determined for each pixel in the image to be processed. Then, when filtering is performed through the filtering window, the cost map under each preset disparity level can be filtered according to the filtering window. Therefore, the cost map under all preset disparity levels only needs to calculate the filtering window once. This approach can improve the performance of the image processing method and increase the possibility of using large disparity levels on some platforms with poor performance, thereby alleviating the technical problem that the existing cost aggregation algorithm is difficult to support large-size and large disparity level calculations on mobile devices with poor performance.

[0056] In an optional embodiment, in the present application, the target cost cube includes: a first cost cube corresponding to the primary image and a second cost cube corresponding to the secondary image. Based on this, step S204, calculating the cost cube of the image to be processed, and obtaining the target cost cube includes the following process:

[0057] First, a stereoscopic correction operation is performed on the image to be processed to transform the image to be processed into a space in which the optical axis is parallel.

[0058] It can be seen from the above description that in the present application, firstly, an image to be processed is obtained. For example, it includes a main image to be processed and a secondary image to be processed. Figure 3The figure shows a schematic diagram of a main image to be processed and a secondary image to be processed. Figure 4 It can be seen that the optical axes of the main camera and the secondary camera are not in a balanced state. At this time, it is necessary to perform a stereo correction operation on the image to be processed. For example, the image to be processed can be rotated by a rotation matrix so that the image to be processed is transformed into a space parallel to the optical axis. Figure 6 As shown in FIG. 1 , the optical axes of the main camera device and the secondary camera device are not in a balanced state, and as shown in FIG. Figure 5 As shown in Figure 3 The image to be processed is shown in the effect diagram after the stereo correction operation is performed.

[0059] Secondly, the cost cube of the image to be processed after the stereo rectification operation is calculated to obtain the first cost cube and the second cost cube.

[0060] In the present application, after performing a stereoscopic correction operation on an image to be processed, a cost cube of the image to be processed after the stereoscopic correction operation can be calculated to obtain a first cost cube and a second cost cube, wherein the calculation process of the first cost cube and the second cost cube is described as follows:

[0061] Specifically, in the present application, the cost cube of the image to be processed after the stereo correction operation can be calculated by a target cost calculation method to obtain a first cost cube and a second cost cube, wherein the target cost calculation method includes any one of the following: SAD (Sum of Absolute Differences) image matching algorithm, Census similarity algorithm. It should be noted that in the present application, in addition to the above method, other calculation methods can also be used, and the present application does not specifically limit this.

[0062] like Figure 7 As shown in the figure, the left image is the main image, denoted as mainImg; the right image is the auxiliary image, denoted as auxImg. Figure 7 As shown, the pixel points of the shaded part in the main image mainImg are recorded as (x, y), and the pixel points of the shaded part in the auxiliary image auxImg are recorded as (x, y) to (x+d, y), where d is the number of preset parallax levels.

[0063] When calculating the target cost cube, we can calculate the similarity between the pixel point (x, y) on the left and the dmax pixels on the right (i.e., (x, y) to (x+d, y)) to get a graph like Figure 8 As shown in the figure, the cost cube with a length of W, a width of H, and a height of dmax is recorded as the target cost cube. Figure 8In the example, C(x, y, d) is the similarity (ie, the matching cost in this application).

[0064] In the present application, after the target cost cube is calculated in the manner described above, the filtering window of each pixel in the image to be processed can be determined.

[0065] In the present application, for each pixel in the image to be processed, the filter window corresponding to each pixel can be determined according to the color position relationship between the pixel points.

[0066] The color position relationship can be understood as: the color relationship between the current pixel and its adjacent pixels, and the position relationship between the current pixel and its adjacent pixels. If the image to be processed is an RGB image, then the color relationship can be understood as the color value of each pixel. If the image to be processed is a grayscale image, then the color relationship can be understood as the grayscale value of each pixel.

[0067] In an optional implementation, determining the filter window corresponding to each pixel point based on the color position relationship between the pixel points of the image to be processed includes the following process:

[0068] Step S1, performing spatial domain transformation on the image to be processed to obtain a spatial domain transformation result;

[0069] Step S2, determining the color position relationship between each pixel and its neighboring pixel points based on the spatial domain transformation result, wherein the color position relationship includes: color information between the pixel points and spatial information between the pixel points;

[0070] Step S3, calculating the similarity value between each pixel and its neighboring pixel points according to the color position relationship, and determining the window radius of the filtering window according to the similarity value;

[0071] Step S4: determining the filtering window based on the window radius of the filtering window.

[0072] In the present application, the image to be processed may be transformed in the spatial domain, thereby transforming the image to be processed into the Domain Transform domain.

[0073] Specifically, in this application, the formula Perform spatial domain transformation on the image to be processed to obtain a spatial domain transformation result. In this formula, k represents the color channel of the image to be processed, and k can be 1 (grayscale image) or 3 (RGB) or other. k (x) represents the gradient map of the image I to be processed in channel k, σ sRepresents the spatial distance parameter, which is used to control the spatial distance between pixels that affect each other. For example, it is used to control the spatial distance between pixels at which they no longer affect each other. Similarly, σ r It represents the color distance parameter, which is used to characterize the color parameter of mutual influence between pixels, and represents how large the color difference between pixels is when they no longer influence each other; c represents the value range of color channel k, and u belongs to the data in the spatial domain.

[0074] In the present application, after the image to be processed is transformed in the spatial domain, the neighboring pixels of each pixel can be determined in the spatial domain transformation result according to the controllable parameters, thereby determining the color information and spatial information between each pixel and the neighboring pixels.

[0075] It should be noted that for a pixel, if the color of its neighboring pixel is too different from that of the pixel, the influence weight of the pixel on it will be very small or even 0; if a neighboring pixel is far away from it, the influence weight of the pixel on it will also be very small or even 0; then at this time, the neighboring pixel of each pixel can be determined by controllable parameters.

[0076] After the color position relationship is determined, the spatial similarity value and / or color similarity value between each pixel and the neighboring pixel can be calculated according to the color position relationship, so as to determine the window radius of the filter window corresponding to each pixel according to the spatial similarity value or the color similarity value. For example, the window radius of the filter window can be directly determined according to the spatial similarity value or the color similarity value.

[0077] It should be noted that, in the present application, the window radius can also be understood as the radius of influence of the current pixel by the neighboring pixel. From the above description, it can be seen that if the color of its neighboring pixel is too different from that of the neighboring pixel, the influence weight of the point on it will be very small, or even 0; if a neighboring pixel is far away from it, the influence weight of this point on it will also be very small, or even 0. Therefore, in the present application, the domain pixel of the current pixel can be determined by an adjustable parameter, so as to determine the window radius according to the phase velocity value between the current pixel and the neighboring pixel.

[0078] In the present application, after determining the filtering window of each pixel in the image to be processed in the manner described above, the cost map at each preset disparity level in the target cost cube can be mean filtered through the filtering window corresponding to each pixel.

[0079] Specifically, in this application, after calculating the radius of the filtering window, each preset disparity level is traversed, and the cost map under each preset disparity level is subjected to window mean filtering. That is to say, in this application, for each pixel point (x, y) in the image to be processed, each cost map in each preset disparity level is filtered using the same filtering window, and the specific process is as follows:

[0080] For a cost map I(x,y) to be filtered at a preset disparity level, if a 3x 3 filter window is used to perform mean filtering on the image, the 3x 3 window can be decomposed into Fig. 9 The structure shown.

[0081] Based on this, the above mean filtering process can be described as: for a cost map I(x, y) to be filtered, it can be changed to average in the column direction first, and then average in the row direction. It should be noted that in this application, the window radius of the filter window calculated in the above steps is Fig. 9 The distance between the upper and lower limits in the column direction, or the left and right limits in the row direction and the current pixel.

[0082] From the above description, it can be seen that the cost map I(x, y) to be filtered at each preset disparity level will share a filtering boundary lookup table (ie, filtering window) at the same time, so the boundary can be directly searched during filtering to implement variable window mean filtering.

[0083] In this application, for the cost cube slices (i.e., cost maps) at d preset disparity levels, the complexity of variable window filtering for each cost cube slice is consistent with the complexity of ordinary mean filtering, and the complexity is much smaller than the cost aggregation solution based on the edge-preserving smoothing filter of the local method. Therefore, the use of this method can greatly improve the performance of the algorithm and increase the possibility of using large-size stereo pairs and large disparity levels on some platforms with poor performance.

[0084] In the present application, after filtering the cost map at each preset disparity level in the target cost cube in the manner described above, the disparity map of the primary camera image and the secondary camera image can be calculated based on the target cost cube after filtering, which specifically includes the following process:

[0085] First, the disparity value of each pixel in the primary image and the secondary image is determined based on the target cost cube after filtering.

[0086] Specifically, in the present application, the matching cost value (for example, the similarity described in the above embodiment) of each pixel in the image to be processed at each preset disparity level can be determined in the target cost cube. Fig.10The figure shows the matching cost value of any pixel at each preset disparity level. At this time, the minimum matching cost value can be determined from the determined matching cost values ​​as the disparity value of the corresponding pixel. Fig.10 The smallest matching cost value Winner d* is selected from the multiple matching cost values ​​shown as the disparity value of the current pixel point in the image to be processed.

[0087] Then, a disparity map of the main image and the auxiliary image is determined based on the disparity value of each pixel point.

[0088] In the present application, after obtaining the disparity map of the image to be processed, the disparity map usually has occlusion areas and noise areas that need to be optimized. The general process is consistency detection of the left and right disparity maps, interpolation of the occlusion areas, and edge-preserving smoothing optimization to obtain the final disparity map. Specifically, in the present application, the disparity map can be optimized according to a preset optimization algorithm to obtain the optimized disparity map, wherein the preset optimization algorithm includes at least one of the following: Figure 1 Consistency detection, occlusion area detection, and edge-preserving smoothing optimization.

[0089] From the above description, it can be seen that in the present application, a method of image processing is provided, which is a stereo matching cost aggregation performance optimization solution with edge-preserving and smoothing characteristics. For a fixed image size and all preset disparity levels d, it is only necessary to calculate the filter window radius once, and then it is only necessary to complete the local cost aggregation method based on the integral image for mean filtering. The entire cost aggregation process is equivalent to mean filtering the images under d preset disparity levels. Since the window radius of the filter window is calculated based on the color position relationship between the pixels of the color image, the present application has edge-preserving and smoothing characteristics. At the same time, the method of calculating the filter radius once and performing mean filtering on d preset disparity levels can greatly improve the algorithm performance, and increase the possibility of using large-size stereo image pairs and large disparity levels on some platforms with poor performance.

[0090] Fig.11 is a flowchart of another image processing method according to an embodiment of the present invention. Fig.11 As shown, the method includes the following process:

[0091] Input a main camera RGB image (hereinafter referred to as the left view) and a secondary camera RGB image (hereinafter referred to as the right view). After the stereo correction operation, calculate the cost cubes of the left view and the right view to obtain the left cost cube and the right cost cube respectively. Then calculate the window radius of the filter window under the left and right views respectively. The main idea is to determine the window radius of each pixel based on the space and color information between the pixels of the RGB image and the adjustable parameters; then traverse each disparity level for the left and right cost cubes, and perform mean filtering based on the window radius of the filter window calculated in the previous step. Finally, the disparity map can be calculated using methods such as WTA, and then the disparity map can be optimized using common post-processing methods to complete the entire stereo matching process.

[0092] From the above description, it can be seen that the method provided by the present application can optimize the performance to be close to the cost aggregation performance based on mean filtering while ensuring the edge-preserving smoothing characteristics. Although the cost aggregation method is fast, it does not have the edge-preserving smoothing characteristics, and will lose the quality of the cost cube in the discontinuous area of ​​the parallax. For example, the cost aggregation method based on the guided filter needs to perform an edge-preserving smoothing filter for each disparity level, including the weight calculation and weighted average of the pixel points in the window. Although it has the edge-preserving smoothing characteristics, the performance is still slow, and it is difficult to support large-size and large-disparity level calculations on mobile devices with poor performance. In the present application, a corresponding filtering window is determined for each pixel point in the image to be processed. Then, when filtering through the filtering window, the cost map under each preset disparity level can be filtered according to the filtering window. Therefore, the cost map under all preset disparity levels only needs to calculate the filtering window once. The use of this method can improve the performance of the image processing method, increase the possibility of using large disparity levels on some platforms with poor performance, thereby alleviating the technical problem that the existing cost aggregation algorithm is difficult to support large-size and large-disparity level calculations on mobile devices with poor performance.

[0093] Embodiment 3:

[0094] The embodiment of the present invention further provides an image processing device, which is mainly used to execute the image processing method provided by the above content of the embodiment of the present invention. The image processing device provided by the embodiment of the present invention is specifically introduced below.

[0095] Fig.12 is a schematic diagram of an image processing device according to an embodiment of the present invention. Fig.12 As shown, the image processing device mainly includes:

[0096] An acquisition unit 10 is used for obtaining a main image to be processed and a secondary image to be processed, and determining the main image and the secondary image as images to be processed respectively;

[0097] A first calculation unit 20 is used to calculate the cost cube of the image to be processed to obtain a target cost cube, wherein the target cost cube represents a cost map of a pixel point in the image to be processed at each preset disparity level;

[0098] A determination unit 30, configured to determine a filter window for each pixel in the image to be processed;

[0099] A filtering unit 40, configured to perform filtering processing on the cost map at each preset disparity level in the target cost cube through the filtering window;

[0100] The second calculation unit 50 is configured to calculate a disparity map of the primary image and the secondary image based on the filtered target cost cube.

[0101] In the present application, a corresponding filtering window is determined for each pixel in the image to be processed. When filtering is performed through the filtering window, the cost map under each preset disparity level can be filtered according to the filtering window. Therefore, the cost map under all preset disparity levels only needs to calculate the filtering window once. This approach can improve the performance of the image processing method and increase the possibility of using large disparity levels on some platforms with poor performance, thereby alleviating the technical problem that the existing cost aggregation algorithm is difficult to support large-size and large disparity level calculations on mobile devices with poor performance.

[0102] Optionally, the target cost cube includes: a first cost cube corresponding to the main camera image and a second cost cube corresponding to the secondary camera image; the first calculation unit is used to: perform a stereo correction operation on the image to be processed to transform the image to be processed into a space parallel to the optical axis; calculate the cost cube of the image to be processed after the stereo correction operation to obtain the first cost cube and the second cost cube.

[0103] Optionally, the first calculation unit is also used to: calculate the cost cube of the image to be processed after the stereo correction operation by a target cost calculation method to obtain the first cost cube and the second cost cube, wherein the target cost calculation method includes any one of the following: SAD image matching algorithm, Census similarity algorithm.

[0104] Optionally, the determining unit is used to determine the filtering window corresponding to each pixel point based on the color position relationship between the pixel points of the image to be processed.

[0105] Optionally, the determination unit is also used to: perform a spatial domain transformation on the image to be processed to obtain a spatial domain transformation result; determine the color position relationship between each pixel point and each pixel point's neighboring pixel points based on the spatial domain transformation result, wherein the color position relationship includes: color information between pixel points, spatial information between pixel points; calculate the similarity value between each pixel point and each pixel point's neighboring pixel points based on the color position relationship, and determine the window radius of the filter window based on the similarity value; determine the filter window based on the window radius of the filter window.

[0106] Optionally, the filtering unit is used to: perform mean filtering processing on the cost map at each preset disparity level in the target cost cube through a filtering window corresponding to each pixel point.

[0107] Optionally, the second calculation unit is used to: determine the disparity value of each pixel in the primary image and the secondary image based on the target cost cube after filtering; and determine the disparity map of the primary image and the secondary image based on the disparity value of each pixel.

[0108] Optionally, the cost map in the target cost cube includes the matching cost value of each pixel point in the image to be processed at each preset disparity level; the second calculation unit is also used to: determine the matching cost value of each pixel point in the primary image and the secondary image at each preset disparity level in the target cost cube; and determine the minimum matching cost value among the determined matching cost values ​​as the disparity value of the corresponding pixel point.

[0109] Optionally, the device is further configured to: optimize the disparity map according to a preset optimization algorithm to obtain the optimized disparity map, wherein the preset optimization algorithm includes at least one of the following: Figure 1 Consistency detection, occlusion area detection, and edge-preserving smoothing optimization.

[0110] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for image processing, characterized in that: include: Acquire a main camera image to be processed and a secondary camera image to be processed, and respectively determine the main camera image and the secondary camera image as images to be processed; Calculating a cost cube of the image to be processed to obtain a target cost cube, wherein the target cost cube represents a cost map of a pixel point in the image to be processed at each preset disparity level; Determine a filtering window for each pixel in the image to be processed based on a color position relationship between pixels of the image to be processed; wherein the color position relationship includes: color information between pixels and spatial information between pixels; Performing filtering processing on the cost map at each preset disparity level in the target cost cube through the filtering window; The disparity map of the primary image and the secondary image is calculated based on the filtered target cost cube.

2. The method according to claim 1, characterized in that The target cost cube includes: a first cost cube corresponding to the primary image and a second cost cube corresponding to the secondary image; Calculating the cost cube of the image to be processed to obtain a target cost cube includes: Performing a stereoscopic correction operation on the image to be processed to transform the image to be processed into a space in which the optical axis is parallel; A cost cube of the image to be processed after the stereo rectification operation is calculated to obtain the first cost cube and the second cost cube.

3. The method according to claim 2, characterized in that Calculating the cost cube of the image to be processed after the stereo rectification operation to obtain the first cost cube and the second cost cube comprises: The cost cube of the image to be processed after the stereo correction operation is calculated by a target cost calculation method to obtain the first cost cube and the second cost cube, wherein the target cost calculation method includes any one of the following: SAD image matching algorithm, Census similarity algorithm.

4. The method according to claim 1, characterized in that: Determining the filter window corresponding to each pixel point based on the color position relationship between the pixel points of the image to be processed includes: Performing spatial domain transformation on the image to be processed to obtain a spatial domain transformation result; Determine the color position relationship between each pixel and its neighboring pixels based on the spatial domain transformation result; Calculating a similarity value between each pixel and its neighboring pixel points according to the color position relationship, and determining a window radius of the filtering window according to the similarity value; The filtering window is determined based on a window radius of the filtering window.

5. The method according to claim 1, characterized in that Filtering the cost map at each preset disparity level in the target cost cube through the filtering window includes: The cost map at each preset disparity level in the target cost cube is subjected to mean filtering processing through the filtering window corresponding to each pixel point.

6. The method according to any one of claims 1 to 5, characterized in that Calculating the disparity map of the primary image and the secondary image based on the target cost cube after filtering includes: Determine the disparity value of each pixel in the primary image and the secondary image based on the target cost cube after filtering; A disparity map of the primary image and the secondary image is determined based on the disparity value of each pixel.

7. The method according to claim 6, characterized in that The cost map in the target cost cube includes a matching cost value for each pixel in the image to be processed at each preset disparity level; Determining the disparity value of each pixel in the primary image and the secondary image based on the target cost cube after filtering includes: Determining, in the target cost cube, a matching cost value for each pixel in the primary image and the secondary image at each preset disparity level; The minimum matching cost value among the determined matching cost values ​​is determined as the disparity value of the corresponding pixel point.

8. The method according to any one of claims 1 to 5, characterized in that The method further comprises: The disparity map is optimized according to a preset optimization algorithm to obtain the optimized disparity map, wherein the preset optimization algorithm includes at least one of the following: disparity map consistency detection, occlusion area detection, and edge-preserving smoothing optimization.

9. An image processing device, characterized in that: include: an acquisition unit, used for obtaining a main image to be processed and a secondary image to be processed, and determining the main image and the secondary image as images to be processed respectively; A first calculation unit, configured to calculate a cost cube of the image to be processed to obtain a target cost cube, wherein the target cost cube represents a cost map of a pixel point in the image to be processed at each preset disparity level; A determination unit, configured to determine a filter window for each pixel in the image to be processed based on a color position relationship between pixels of the image to be processed; wherein the color position relationship includes: color information between pixels and spatial information between pixels; A filtering unit, used for filtering the cost map at each preset disparity level in the target cost cube through the filtering window; The second calculation unit is used to calculate the disparity map of the primary camera image and the secondary camera image based on the target cost cube after filtering.

10. An electronic device comprising a storage device, a processing device, and a computer program stored in the storage device and executable on the processing device, characterized in that: When the processing device executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer readable medium having non-volatile program code executable by a processing device, characterized in that: The program code enables the processing device to execute the steps of the method according to any one of claims 1 to 8.

Images