Smart device and image processing method

Abstract

The present application discloses an intelligent device and an image processing method, which are used to reduce the computational complexity of the stereo matching algorithm of the binocular view and greatly improve the matching efficiency and accuracy. The intelligent device includes an image acquisition device for acquiring images, and also includes: a processor, which is used to: determine the gradient map of the binocular source image; and perform image compression on the binocular source image to obtain the compressed binocular image; filter the compressed binocular image, and then obtain the gradient map of the filtered binocular image, perform window-based stereo matching on the gradient map, and obtain an initial disparity image; based on the gradient map of the binocular source image and the initial disparity image, determine the edge disparity of the initial disparity image, and use the edge disparity to determine a sparse disparity image, and perform image recognition based on the sparse disparity image.

Description

Technical Field

[0001] The present application relates to the field of electrical technology, and in particular to smart devices and image processing methods. Background Art

[0002] In recent years, depth cameras have been increasingly used in smart home, industrial automation, security and other fields. They have the characteristics of strong recognition ability, high detection accuracy, good reliability, strong anti-interference ability, real-time and high efficiency. At present, there are three main types of depth cameras: time of flight (TOF), binocular red, green and blue (RGB) and structured light. Among them, binocular RGB camera and binocular structured light depth camera are mainly based on binocular stereo vision technology.

[0003] Binocular stereo vision is a mechanism that simulates the human eye's recognition of the three-dimensional information of the scene. It obtains the two-dimensional image of the scene from two angles, and then obtains the disparity image by establishing the matching relationship between the images. Then, according to the triangulation principle, the depth information of the original image is obtained. Stereo matching is the core of binocular stereo vision technology. Stereo matching methods are mainly divided into two categories: local matching algorithm and global matching algorithm. The local matching algorithm uses the neighborhood information for matching. When matching, it only considers the grayscale information or position relationship between the pixels in the window. The calculation complexity is low, but the matching accuracy is poor, especially in low-texture and discontinuous disparity areas. It is easy to produce mismatches. The global matching algorithm uses the information of the entire image, obtains the cost space based on local matching, constructs an energy function with data terms and smoothing terms, and converts the matching problem into the optimization problem of the energy function. Compared with the local matching algorithm, it has higher matching accuracy, higher calculation complexity, and longer time consumption.

[0004] The disparity images obtained by binocular stereo matching are mainly divided into two types, dense disparity maps and sparse disparity maps, which are called edge-based disparity maps and surface-based disparity maps in the industry. Each pixel in the dense disparity map has a corresponding disparity value and is rich in information. The sparse disparity image, on the other hand, has corresponding disparity values ​​only for feature points, and invalid disparity at non-feature points. It is often used for detection and recognition of specific scenes. Although dense disparity images have rich three-dimensional information, they are relatively time-consuming to calculate and are prone to erroneous disparity. The feature point disparity in sparse disparity images is usually relatively accurate, the calculation time is relatively short, but the disparity image is relatively sparse.

[0005] Depth images can be further acquired based on the relationship between parallax and depth. However, traditional stereo matching methods are often highly complex and have poor real-time performance. It is difficult to transplant the algorithm to a low-cost computing platform, which limits the commercialization of this technology. Summary of the invention

[0006] The embodiments of the present application provide an intelligent device and an image processing method to reduce the computational complexity of a stereo matching algorithm for binocular views and significantly improve matching efficiency and accuracy.

[0007] An intelligent device provided in an embodiment of the present application includes an image acquisition device for acquiring an image, and further includes:

[0008] Processor for:

[0009] Determine a gradient map of a binocular source image; and, perform image compression on the binocular source image to obtain a compressed binocular image;

[0010] Filtering the compressed binocular image; and obtaining a gradient map of the filtered binocular image, performing window-based stereo matching on the gradient map, and obtaining an initial disparity image;

[0011] Based on the gradient map of the binocular source image and the initial disparity image, the edge disparity of the initial disparity image is determined, and a sparse disparity image is determined using the edge disparity, and image recognition is performed based on the sparse disparity image.

[0012] The embodiment of the present application collects images through the image acquisition device in the smart device, and executes through the processor: determining the gradient map of the binocular source image; and compressing the binocular source image to obtain a compressed binocular image; filtering the compressed binocular image; and obtaining the gradient map of the filtered binocular image, performing window-based stereo matching on the gradient map, and obtaining an initial disparity image; based on the gradient map of the binocular source image and the initial disparity image, determining the edge disparity of the initial disparity image, and using the edge disparity to determine a sparse disparity image, and performing image recognition based on the sparse disparity image, thereby reducing the computational complexity of the stereo matching algorithm of the binocular view and greatly improving the matching efficiency and accuracy.

[0013] Optionally, the processor is further configured to:

[0014] Post-processing the initial disparity image, wherein the post-processing includes one or a combination of the following processing: removing noise points, removing mismatched points, and smoothing the disparity image;

[0015] Determining the edge disparity of the initial disparity image based on the gradient map of the binocular source image and the initial disparity image specifically includes: determining the edge disparity of the post-processed initial disparity image based on the gradient map of the binocular source image and the post-processed initial disparity image.

[0016] Optionally, the left view in the binocular source view is a comparison image imgL, and the right view is a reference image imgR. The binocular source image is compressed N times using the following formula to obtain a compressed comparison image imgpresL and a compressed reference image imgpresR:

[0017] imgpresL(nx,ny)=(imgL(x,y)+imgL(x,y+1)…imgL(x,y+N-1)) / N

[0018] imgpresR(nx,ny)=(imgR(x,y)+imgR(x,y+1)…imgR(x,y+N-1)) / N

[0019] Among them, nx=1,2,…,width;ny=1,2,…,height / N,height and width represent the height and width of the comparison image imgL and the reference image imgR respectively.

[0020] Optionally, determining the edge disparity of the initial disparity image based on the gradient map of the binocular source image and the initial disparity specifically includes:

[0021] Copy any one row of parallax in the initial parallax image into N rows of parallax, so that the initial parallax image is expanded N times in the height direction;

[0022] For the gradient image imGradR of the reference image in the binocular source image, a gradient threshold T is set, and for the pixel position whose pixel value in the gradient imGradR is less than T, the disparity value of the same pixel position is assigned as an invalid disparity in the disparity image dispIm;

[0023] Determine the pixel positions whose pixel values ​​in the gradient image imGradR are greater than the gradient threshold T as the image edge areas in the initial disparity image;

[0024] For the disparity value Disp(x,y) of any pixel in the image edge area in the initial disparity image, the disparity value Disp(x,y) of the pixel is determined based on the previous pixel disparity value disp(x-1,y), the current pixel disparity value disp(x,y) and the next pixel disparity value disp(x+1,y).

[0025] Optionally, when the disparity value Disp(x, y) at any pixel in the image edge region in the initial disparity image is not zero, it is determined that the disparity at the pixel position is filled with the original disparity value;

[0026] When the disparity value Disp(x,y) of any pixel in the image edge area of ​​the initial disparity image is zero, compare the disparity value disp(x-1,y) of the previous pixel and the disparity value disp(x+1,y) of the next pixel. If one of the disparities is zero, use the non-zero disparity value as Disp(x,y); if both are non-zero, use the smaller value as Disp(x,y).

[0027] An image processing method provided in an embodiment of the present application includes:

[0028] Determine a gradient map of a binocular source image; and, perform image compression on the binocular source image to obtain a compressed binocular image;

[0029] Filtering the compressed binocular image; and obtaining a gradient map of the filtered binocular image, performing window-based stereo matching on the gradient map, and obtaining an initial disparity image;

[0030] Based on the gradient map of the binocular source image and the initial disparity image, the edge disparity of the initial disparity image is determined, and a sparse disparity image is determined using the edge disparity, and image recognition is performed based on the sparse disparity image.

[0031] Optionally, the method further comprises:

[0032] Post-processing the initial disparity image, wherein the post-processing includes one or a combination of the following processing: removing noise points, removing mismatched points, and smoothing the disparity image;

[0033] Determining the edge disparity of the initial disparity image based on the gradient map of the binocular source image and the initial disparity image specifically includes: determining the edge disparity of the post-processed initial disparity image based on the gradient map of the binocular source image and the post-processed initial disparity image.

[0034] Optionally, the left view in the binocular source view is a comparison image imgL, and the right view is a reference image imgR. The binocular source image is compressed N times using the following formula to obtain a compressed comparison image imgpresL and a compressed reference image imgpresR:

[0035] imgpresL(nx,ny)=(imgL(x,y)+imgL(x,y+1)…imgL(x,y+N-1)) / N

[0036] imgpresR(nx,ny)=(imgR(x,y)+imgR(x,y+1)…imgR(x,y+N-1)) / N

[0037] Among them, nx=1,2,…,width;ny=1,2,…,height / N,height and width represent the height and width of the comparison image imgL and the reference image imgR respectively.

[0038] Optionally, determining the edge disparity of the initial disparity image based on the gradient map of the binocular source image and the initial disparity specifically includes:

[0039] Copy any one row of parallax in the initial parallax image into N rows of parallax, so that the initial parallax image is expanded N times in the height direction;

[0040] For the gradient image imGradR of the reference image in the binocular source image, a gradient threshold T is set, and for the pixel position whose pixel value in the gradient imGradR is less than T, the disparity value of the same pixel position is assigned as an invalid disparity in the disparity image dispIm;

[0041] Determine the pixel positions whose pixel values ​​in the gradient image imGradR are greater than the gradient threshold T as the image edge areas in the initial disparity image;

[0042] For the disparity value Disp(x,y) of any pixel in the image edge area in the initial disparity image, the disparity value Disp(x,y) of the pixel is determined based on the previous pixel disparity value disp(x-1,y), the current pixel disparity value disp(x,y) and the next pixel disparity value disp(x+1,y).

[0043] Another embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are used to enable the computer to execute any of the above methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 A schematic diagram of an image correction error image provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of a stereo matching process based on image compression and expansion provided in an embodiment of the present application;

[0047] Figure 3 A schematic diagram of a 1*3 gradient operator provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of a 3*3 gradient operator provided in an embodiment of the present application;

[0049] Figure 5 A schematic diagram of a Gaussian filter template provided in an embodiment of the present application;

[0050] Figure 6A schematic diagram of stereo matching based on a fixed window provided in an embodiment of the present application;

[0051] Figure 7 A schematic diagram of the structure of the smart device provided in the embodiment of the present application;

[0052] Figure 8 A flowchart of an image processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] The technical solution provided in the embodiment of the present application compresses the reference image (such as the right view) and the comparison image (such as the left view) along the image height direction, and then uses the compressed images to perform stereo matching of the binocular views. When using a small matching window, the technical solution provided in the embodiment of the present application can reduce the computational complexity of the matching algorithm and greatly improve the matching efficiency and accuracy.

[0055] The embodiment of the present application effectively avoids erroneous matching of complex texture areas through the matching of compressed images, reduces the influence of strict limit constraints on image matching of left and right views, obtains sparse disparity images with higher accuracy, and facilitates effective ranging or detection of objects in the later stage.

[0056] The methods and devices described in the embodiments of the present application are based on the same application concept. Since the methods and devices solve problems based on similar principles, the implementation of the devices and methods can refer to each other, and the repeated parts will not be repeated.

[0057] The following is a detailed description of each embodiment of the present application in conjunction with the accompanying drawings. It should be noted that the display order of the embodiments of the present application only represents the order of the embodiments, and does not represent the advantages and disadvantages of the technical solutions provided by the embodiments.

[0058] The binocular image matching method is based on the binocular correction process. After the image correction, the epipolar lines of the two images are exactly on the same horizontal line, that is, any point in the left image and its matching point in the right image must be on the same line. However, the traditional correction algorithm still has large parallel equiposition errors, such as Figure 1In addition, a large matching window in the binocular matching process will consume too many hardware resources, especially the field programmable gate array (FPGA), which will face the problem of insufficient random access memory (RAM) and memory resources.

[0059] The edge-based stereo matching method of binocular images (including left view and right view) proposed in the embodiment of the present application obtains edge-based sparse disparity images, and performs image compression-based stereo matching on the entire image (i.e., the entire image, that is, both the left and right disparity images need to be compressed) to obtain a more robust disparity value, and then uses the disparity corresponding to the effective pixel points to filter the disparity image to enhance the matching accuracy of the left view and the right view, and then extracts the edge-based disparity image based on image expansion and gradient. Among them, the sparse disparity map generation method in the industry is also called the edge-based matching method, and the corresponding dense disparity map generation method is called the surface-based matching method.

[0060] The smart device provided in the embodiment of the present application includes modules such as an image acquisition device for acquiring images, and also includes a processor. The processor adopts a binocular stereo matching method and uses a method based on image compression and expansion to perform stereo matching, and finally obtains a sparse parallax image. The main process executed by the processor is as follows:

[0061] Step 1: compress the input binocular source image (including the left view and the right view) to obtain a compressed binocular image; and extract the gradient map of the uncompressed image (i.e., the reference image); wherein, the gradient image can be obtained by traversing the entire image using a gradient operator (two operators are given in the embodiment of the present application), and the edge information of the image can be further obtained by gradient extraction.

[0062] Step 2: preprocessing the compressed binocular image (including the left view and the right view), that is, performing Gaussian filtering, and after Gaussian filtering, extracting the gradient map of the binocular image (including the left view and the right view) after Gaussian filtering;

[0063] Step 3: performing window-based stereo matching on the gradient map corresponding to the reference image processed in step 2 and the gradient map corresponding to the comparison image processed in step 2 to obtain an initial disparity image;

[0064] Each element in the disparity image stores a disparity value obtained through a matching algorithm, and the disparity value can be further calculated to obtain a distance value.

[0065] Step 4: post-processing the initial disparity image, that is, removing noise points, mismatched points, and smoothing the disparity image;

[0066] Step 5: Based on the gradient image of the full image (i.e., the uncompressed image (i.e., the reference image)), the edge disparity of the initial disparity image obtained after the post-processing is extracted, and the full image disparity is restored using the edge disparity, i.e., the expansion, to obtain a sparse disparity image. The sparse disparity image here mainly refers to further extracting the initial disparity image, and only retaining the disparity value at the boundary position of the object.

[0067] The process of stereo matching method based on the above process can be found in Figure 2 , where the specific content of each step is as follows, where the left view is taken as the comparison image (referred to as the comparison image), the right view is taken as the reference image (referred to as the reference image), and the final disparity image obtained is based on the right view:

[0068] Step 1: Compression and gradient extraction of binocular source images:

[0069] When traditional binocular stereo matching algorithms match binocular source images, the matching window size is often limited in hardware implementation and cannot be expanded arbitrarily, resulting in too many mismatched disparities in the disparity map and a reduction in effective disparity.

[0070] Therefore, the technical solution provided in the embodiment of the present application proposes to pre-compress the comparison image and the reference image. During the compression process, the image can be compressed N times, and N is generally not too large. The specific value can be determined according to actual needs, and the embodiment of the present application does not limit it. The so-called compression, for example, merges N adjacent upper and lower rows of pixels, and uses the merged pixels as compressed pixels.

[0071] There are many methods for image compression technology, such as using difference, mean, etc. for fusion. Preferably, mean fusion is used. Specifically, assuming that the image size is height (height)*width (width), the comparison image (imgL) and the reference image (imgR) are compressed row by row according to the following formula (1) to obtain the compressed comparison image (imgpresL) and the compressed reference image (imgpresR).

[0072] Formula (1):

[0073] imgpresL(nx,ny)=(imgL(x,y)+imgL(x,y+1)…imgL(x,y+N-1)) / N

[0074] imgpresR(nx,ny)=(imgR(x,y)+imgR(x,y+1)…imgR(x,y+N-1)) / N

[0075] Among them, nx=1,2,…,width;ny=1,2,…,height / N. When height / N cannot be divided evenly, the remainder rows are not compressed and are directly copied to the compressed image.

[0076] Optionally, the value of imgpresL(nx,ny) can also be calculated using the median or maximum value of imgL(x,y), ...imgL(x,y+N-1), and the same applies to imgpresR(nx,ny).

[0077] For example, when implementing the above compression function in FPGA hardware design, take N=2 as an example: First, select N RAMs to implement the data cache structure. For RAM1, RAM2..., RAMm, where the size of m is mainly determined by the size of the image, the write signal is performed at the first system clock clk1, and the read signal is performed at the second system clock clk2. Writing data means writing data to RAM1, RAM2..., RAMm in turn during the selected row valid cycle, and only writing data to one RAM per valid cycle; reading data means reading data from RAM1, RAM2..., RAMm at the same time during the selected valid cycle; for RAMs that perform read and write operations simultaneously in the same valid cycle, the write operation is performed first and then the read operation is performed to ensure that data will not be lost. In each cycle, two RAM ports will simultaneously output the same column data of two adjacent rows. After these two data are sent to the calculation module, the average value is taken as the image data after downsampling.

[0078] In addition, before the image is compressed, the uncompressed reference image imgR is subjected to gradient processing, and the gradient processing can use a 1*3 gradient template (see Figure 3 ) or a 3*3 gradient template (see Figure 4 ), the templates are all horizontal gradient templates, that is, only the horizontal gradient of the image is calculated to obtain the gradient map imGradR of the uncompressed reference image, in preparation for the subsequent extraction of edge disparity.

[0079] Step 2: Preprocessing of binocular source images:

[0080] The preprocessing described in the embodiment of the present application also uses Gaussian filtering to filter and smooth the compressed comparison image imgpresL and the compressed reference image imgpresR respectively to remove noise points in the image. In addition, in order to further increase the robustness of the later stereo matching, after Gaussian filtering, gradient processing is performed on the comparison image and the reference image respectively, and the pixel grayscale information in the comparison image and the reference image after Gaussian filtering is processed into pixel gradient information.

[0081] Among them, Gaussian filtering can use a 3*3 filter template (see Figure 5), Gaussian filtering is performed on each pixel in the comparison image and the reference image to obtain a filtered comparison image, denoted as imgausL, and a filtered reference image, denoted as imgausR. Then, gradient processing is performed on the filtered comparison image imgausL and the reference image imgausR, respectively. The gradient processing can use a 1*3 gradient template or a 3*3 gradient template, and gradient processing is performed on each pixel in the filtered comparison image imgausL and the reference image imgausR, respectively, to obtain a gradient image based on the comparison image and a gradient image based on the reference image (the following step three performs stereo matching based on the gradient image obtained in step two), completing the preprocessing step of the compressed binocular source image.

[0082] Step 3: Window-based stereo matching:

[0083] The principle of the stereo matching method based on a fixed window is that for each to-be-matched point in the reference image, a fixed window is taken with the point (assuming it is (x, y)) as the center. Similarly, for each candidate matching point in the comparison image (the candidate matching point is generally within the disparity range, that is, the x-coordinate range is [x+dmin, x+dmax], and the y-coordinate is the same as the y-coordinate of the to-be-matched point, where dmin and dmax are preset values, respectively. The specific values ​​can be determined according to actual needs and are not limited in this application), a fixed window of the same size is taken with each candidate matching point as the center, and grayscale similarity or grayscale difference calculation is performed with the fixed window centered on (x, y) in the reference image.

[0084] For example, commonly used grayscale similarity calculation functions include: zero mean grayscale correlation coefficient function (ZCC), and normalized grayscale cross-correlation function (NCC), and their formulas are as follows:

[0085]

[0086]

[0087] For example, commonly used grayscale difference functions include: the sum of absolute values ​​of pixel differences (SAD), and the sum of squares of pixel differences (SSD), and their formulas are as follows:

[0088]

[0089]

[0090] In the above formula, W represents the window, I1 represents the reference image, I2 represents the comparison image, d represents the disparity (the element in the disparity map is the disparity), avgW1 represents the pixel average in the reference image window, and avgW2 represents the pixel average in the comparison image window.

[0091] For the grayscale similarity function, the disparity d corresponding to the maximum function value is selected as the disparity value of the point to be matched (x, y). For the grayscale value difference function, the disparity d corresponding to the minimum function value is selected as the disparity value of the point to be matched (x, y).

[0092] The schematic diagram of stereo matching based on fixed window is as follows Figure 6 As shown, for the to-be-matched point in the reference image (the center point in the rectangular box), the disparity value finally obtained is the disparity corresponding to the minimum SSD. Figure 6 The small rectangular box in is the window, and the scanning line is a row of pixels determined in the comparison image that is at the same height as the point to be matched (x, y) in the reference image. The scanning search range is determined by the camera parameters.

[0093] The matching cost is calculated using a fixed window stereo matching method to obtain the initial disparity image.

[0094] Step 4: Parallax post-processing:

[0095] For the initial disparity image obtained, the disparity of the mismatched pixels can be removed through left-right consistency detection (at this time, two disparity images need to be obtained, namely, the disparity image based on the left view and the disparity image based on the right view), and then the valid disparity in the disparity image is smoothed and filtered through median filtering or mean filtering, and the invalid disparity (disparity is zero, that is, invalid disparity) is not processed to obtain the post-processed disparity image dispIm.

[0096] Step 5: Extract the edge disparity of the initial disparity image and restore the full image disparity:

[0097] After the disparity post-processing, an image of size (height / N*width) is obtained, which needs to be expanded to obtain a disparity map of the same size as the original image. However, during the image compression process, a lot of detail information is compressed vertically, and after stereo matching, a lot of detail information is lost. Therefore, the embodiment of the present application uses a row of disparity information to fill the adjacent N rows of disparity, and then determines the disparity position on the edge (the gradient is greater than the set threshold, which is considered to be an edge), and compares the adjacent disparity sizes before and after the position to determine the final filled disparity value. The specific operation steps are as follows:

[0098] Step 1: Copy one row of disparity into N rows of disparity, so that the disparity map is expanded N times along the height direction. The rows that cannot be divided evenly during the compression process are not expanded, thereby obtaining a disparity map of size (height*width).

[0099] Step 2: For the gradient processed image imGradR, set the gradient threshold T. If there are pixels with pixel values ​​less than T in the image imGradR, the disparity value of the pixel at the same position in the disparity image dispIm is assigned as invalid disparity.

[0100] Step 3: If the pixel value in the image imGradR is greater than the gradient threshold T, the pixel value is considered to be the edge area of ​​the image. The disparity value Disp(x,y) at the edge ultimately needs to be determined based on the previous pixel disparity value disp(x,y-1), the current pixel disparity value disp(x,y), and the next pixel disparity value disp(x,y+1). When the disparity value at the edge is not zero, the disparity at the edge is filled with the original value. When the disparity disp(x,y) at the edge is zero, compare the previous disparity and the next disparity. If one of the two is zero, use the non-zero disparity to fill. If both are not zero, use the smaller value of the two to fill. The formula is as follows:

[0101]

[0102] Among them, A=min(disp(x-1,y),disp(x+1,y)), B=max(disp(x-1,y),disp(x+1,y)),

[0103] disp(x,y) is the disparity value at the edge.

[0104] When disp(x,y) is not zero at the boundary, it means that the disparity value is valid and the original value is directly filled.

[0105] When disp(x,y) is zero at the boundary, first compare the disparity values ​​of the previous adjacent pixel and the next adjacent pixel at the boundary, obtain min(disp(x-1,y),disp(x+1,y)), and use the smaller disparity value to fill. The disparity is inversely proportional to the actual distance. Using a small disparity value to fill is more reliable, making the post-processing cost caused by the error very small.

[0106] Furthermore, when the disparity value of min(disp(x-1,y), disp(x+1,y)) is zero, the larger disparity value among the adjacent pixels is used to fill the gap, so as to ensure that more valid disparity values ​​are obtained.

[0107] According to the above steps, all the pixels in imGradR are traversed, and finally an edge-based stereo matching disparity image, that is, the sparse disparity image, is obtained, which is used for later detection and recognition based on the sparse disparity image. For example, detection and recognition of the following scenes: Home scenes: detection and recognition of human hands, human bodies, faces, etc.; Robot navigation: detection and recognition of obstacles. With accurate depth values, many depth-based detection and recognition applications can be deployed. The method proposed in the embodiment of the present application significantly reduces the area of ​​​​error matching, and the effective disparity at the edge is richer, which is conducive to later detection and recognition work.

[0108] In summary, see Figure 7 , an intelligent device provided in an embodiment of the present application includes an image acquisition device 11 for acquiring an image, and further includes:

[0109] The processor 12 is configured to:

[0110] Determine a gradient map of a binocular source image (including a left view and a right view); and perform image compression on the binocular source image to obtain a compressed binocular image;

[0111] Filtering the compressed binocular image (including the left view and the right view); and obtaining a gradient map of the filtered binocular image, performing window-based stereo matching on the gradient map, and obtaining an initial disparity image;

[0112] Based on the gradient map of the binocular source image and the initial disparity image, the edge disparity of the initial disparity image is determined, and a sparse disparity image is determined using the edge disparity, and image recognition is performed based on the sparse disparity image.

[0113] Optionally, the processor 12 is further configured to:

[0114] Post-processing the initial disparity image, wherein the post-processing includes one or a combination of the following processing: removing noise points, removing mismatched points, and smoothing the disparity image;

[0115] Determining the edge disparity of the initial disparity image based on the gradient map of the binocular source image and the initial disparity image specifically includes: determining the edge disparity of the post-processed initial disparity image based on the gradient map of the binocular source image and the post-processed initial disparity image.

[0116] Optionally, the left view in the binocular source view is a comparison image imgL, and the right view is a reference image imgR. The binocular source image is compressed N times using the following formula to obtain a compressed comparison image imgpresL and a compressed reference image imgpresR:

[0117] imgpresL(nx,ny)=(imgL(x,y)+imgL(x,y+1)…imgL(x,y+N-1)) / N

[0118] imgpresR(nx,ny)=(imgR(x,y)+imgR(x,y+1)…imgR(x,y+N-1)) / N

[0119] Among them, nx=1,2,…,width;ny=1,2,…,height / N,height and width represent the height and width of the comparison image imgL and the reference image imgR respectively.

[0120] Optionally, determining the edge disparity of the initial disparity image based on the gradient map of the binocular source image and the initial disparity specifically includes:

[0121] Copy any one row of parallax in the initial parallax image into N rows of parallax, so that the initial parallax image is expanded N times in the height direction;

[0122] For the gradient image imGradR of the reference image in the binocular source image, a gradient threshold T is set, and for the pixel position whose pixel value in the gradient imGradR is less than T, the disparity value of the same pixel position is assigned as an invalid disparity in the disparity image dispIm;

[0123] Determine the pixel positions whose pixel values ​​in the gradient image imGradR are greater than the gradient threshold T as the image edge areas in the initial disparity image;

[0124] For the disparity value Disp(x,y) of any pixel in the image edge area in the initial disparity image, the disparity value Disp(x,y) of the pixel is determined based on the previous pixel disparity value disp(x-1,y), the current pixel disparity value disp(x,y) and the next pixel disparity value disp(x+1,y).

[0125] Optionally, when the disparity value Disp(x, y) at any pixel in the image edge region in the initial disparity image is not zero, it is determined that the disparity at the pixel position is filled with the original disparity value;

[0126] When the disparity value Disp(x,y) of any pixel in the image edge area of ​​the initial disparity image is zero, compare the disparity value disp(x-1,y) of the previous pixel and the disparity value disp(x+1,y) of the next pixel. If one of the disparities is zero, use the non-zero disparity value as Disp(x,y); if both are non-zero, use the smaller value as Disp(x,y).

[0127] An image processing device provided in an embodiment of the present application includes any of the above-mentioned smart devices.

[0128] Optionally, the image processing device may be, for example, a camera or other image-related device.

[0129] See also Figure 8 , an image processing method provided by an embodiment of the present application includes:

[0130] S101, determining a gradient map of a binocular source image; and performing image compression on the binocular source image to obtain a compressed binocular image;

[0131] This step is, for example, the processing process of step 1 above.

[0132] S102, filtering the compressed binocular image; and obtaining a gradient map of the filtered binocular image, performing window-based stereo matching on the gradient map, and obtaining an initial disparity image;

[0133] This step is, for example, the processing process of the above-mentioned steps 2 and 3.

[0134] S103 , determining an edge disparity of the initial disparity image based on the gradient map of the binocular source image and the initial disparity image, determining a sparse disparity image using the edge disparity, and performing image recognition based on the sparse disparity image.

[0135] This step is, for example, the processing process of step five above.

[0136] Optionally, the method further comprises:

[0137] Post-processing the initial disparity image, wherein the post-processing includes one or a combination of the following processing: removing noise points, removing mismatched points, and smoothing the disparity image;

[0138] Determining the edge disparity of the initial disparity image based on the gradient map of the binocular source image and the initial disparity image specifically includes: determining the edge disparity of the post-processed initial disparity image based on the gradient map of the binocular source image and the post-processed initial disparity image.

[0139] That is, the method may also include the processing process of step four above.

[0140] Optionally, the left view in the binocular source view is a comparison image imgL, and the right view is a reference image imgR. The binocular source image is compressed N times using the following formula to obtain a compressed comparison image imgpresL and a compressed reference image imgpresR:

[0141] imgpresL(nx,ny)=(imgL(x,y)+imgL(x,y+1)…imgL(x,y+N-1)) / N

[0142] imgpresR(nx,ny)=(imgR(x,y)+imgR(x,y+1)…imgR(x,y+N-1)) / N

[0143] Among them, nx=1,2,…,width;ny=1,2,…,height / N,height and width represent the height and width of the comparison image imgL and the reference image imgR respectively.

[0144] Optionally, determining the edge disparity of the initial disparity image based on the gradient map of the binocular source image and the initial disparity specifically includes:

[0145] Copy any one row of parallax in the initial parallax image into N rows of parallax, so that the initial parallax image is expanded N times in the height direction;

[0146] For the gradient image imGradR of the reference image in the binocular source image, a gradient threshold T is set, and for the pixel position whose pixel value in the gradient imGradR is less than T, the disparity value of the same pixel position is assigned as an invalid disparity in the disparity image dispIm;

[0147] Determine the pixel positions whose pixel values ​​in the gradient image imGradR are greater than the gradient threshold T as the image edge areas in the initial disparity image;

[0148] For the disparity value Disp(x,y) of any pixel in the image edge area in the initial disparity image, the disparity value Disp(x,y) of the pixel is determined based on the previous pixel disparity value disp(x-1,y), the current pixel disparity value disp(x,y) and the next pixel disparity value disp(x+1,y).

[0149] Optionally, when the disparity value Disp(x, y) at any pixel in the image edge region in the initial disparity image is not zero, it is determined that the disparity at the pixel position is filled with the original disparity value;

[0150] When the disparity value Disp(x,y) of any pixel in the image edge area of ​​the initial disparity image is zero, compare the disparity value disp(x-1,y) of the previous pixel and the disparity value disp(x+1,y) of the next pixel. If one of the disparities is zero, use the non-zero disparity value as Disp(x,y); if both are non-zero, use the smaller value as Disp(x,y).

[0151] The method provided in the embodiment of the present application is a method flow executed in the processor.

[0152] It should be noted that the division of the functional units of each device in the embodiments of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation. In addition, the functional units in the various embodiments of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units.

[0153] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0154] The smart device provided in the embodiment of the present application may be a smart home device, such as a smart refrigerator, etc. It may also be a desktop computer, a portable computer, a smart phone, a tablet computer, a personal digital assistant (PDA), etc. The smart device may include a central processing unit (CPU), a memory, an input / output device, etc. The input device may include a keyboard, a mouse, a touch screen, etc. The output device may include a display device, such as a liquid crystal display (LCD), a cathode ray tube (CRT), etc.

[0155] The memory may include a read-only memory (ROM) and a random access memory (RAM), and provides the processor with program instructions and data stored in the memory. In an embodiment of the present application, the memory may be used to store the program of any of the methods provided in the embodiments of the present application.

[0156] The processor calls the program instructions stored in the memory, and the processor is used to execute any of the methods provided in the embodiments of the present application according to the obtained program instructions.

[0157] An embodiment of the present application provides a computer storage medium for storing computer program instructions used by the apparatus provided in the above-mentioned embodiment of the present application, which includes a program for executing any of the methods provided in the above-mentioned embodiment of the present application.

[0158] The computer storage medium can be any available medium or data storage device that can be accessed by a computer, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO)), optical storage (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drives (SSDs)), etc.

[0159] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0160] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0161] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0163] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An intelligent device, comprising an image acquisition device for acquiring images, characterized in that: Also includes: Processor for: Determine a gradient map of a binocular source image; and, perform image compression on the binocular source image to obtain a compressed binocular image; Filtering the compressed binocular image; and obtaining a gradient map of the filtered binocular image, performing window-based stereo matching on the gradient map, and obtaining an initial disparity image; Based on the gradient map of the binocular source image and the initial disparity image, the edge disparity of the initial disparity image is determined, and a sparse disparity image is determined using the edge disparity, and image recognition is performed based on the sparse disparity image.

2. The smart device according to claim 1, characterized in that: The processor is further configured to: Post-processing the initial disparity image, wherein the post-processing includes one or a combination of the following processing: removing noise points, removing mismatched points, and smoothing the disparity image; Determining the edge disparity of the initial disparity image based on the gradient map of the binocular source image and the initial disparity image specifically includes: determining the edge disparity of the post-processed initial disparity image based on the gradient map of the binocular source image and the post-processed initial disparity image.

3. The smart device according to claim 1, characterized in that: The left view in the binocular source view is the comparison image imgL, and the right view is the reference image imgR. The binocular source image is compressed N times using the following formula to obtain the compressed comparison image imgpresL and the compressed reference image imgpresR: imgpresL(nx,ny)=(imgL(x,y)+imgL(x,y+1)…imgL(x,y+N-1)) / N imgpresR(nx,ny)=(imgR(x,y)+imgR(x,y+1)…imgR(x,y+N-1)) / N Among them, nx=1,2,…,width;ny=1,2,…,height / N,height and width represent the height and width of the comparison image imgL and the reference image imgR respectively.

4. The smart device according to claim 3, characterized in that: Determining the edge disparity of the initial disparity image based on the gradient map of the binocular source image and the initial disparity specifically includes: Copy any one row of parallax in the initial parallax image into N rows of parallax, so that the initial parallax image is expanded N times in the height direction; For the gradient image imGradR of the reference image in the binocular source image, a gradient threshold T is set, and for the pixel position whose pixel value in the gradient imGradR is less than T, the disparity value of the same pixel position is assigned as an invalid disparity in the disparity image dispIm; Determine the pixel positions whose pixel values ​​in the gradient image imGradR are greater than the gradient threshold T as the image edge areas in the initial disparity image; For the disparity value Disp(x,y) of any pixel in the image edge area in the initial disparity image, the disparity value Disp(x,y) of the pixel is determined based on the previous pixel disparity value disp(x-1,y), the current pixel disparity value disp(x,y) and the next pixel disparity value disp(x+1,y).

5. The smart device according to claim 4, characterized in that: When the disparity value Disp(x, y) at any pixel in the image edge region in the initial disparity image is not zero, it is determined that the disparity at the pixel position is filled with the original disparity value; When the disparity value Disp(x,y) of any pixel in the image edge area of ​​the initial disparity image is zero, compare the disparity value disp(x-1,y) of the previous pixel and the disparity value disp(x+1,y) of the next pixel. If one of the disparities is zero, use the non-zero disparity value as Disp(x,y); if both are non-zero, use the smaller value as Disp(x,y).

6. An image processing method, characterized in that: include: Determine a gradient map of a binocular source image; and, perform image compression on the binocular source image to obtain a compressed binocular image; Filter the compressed binocular image; Furthermore, a gradient map of the filtered binocular image is obtained, and a window-based stereo matching is performed on the gradient map to obtain an initial disparity image; Based on the gradient map of the binocular source image and the initial disparity image, the edge disparity of the initial disparity image is determined, and a sparse disparity image is determined using the edge disparity, and image recognition is performed based on the sparse disparity image.

7. The method according to claim 6, characterized in that The method further includes: Post-processing the initial disparity image, wherein the post-processing includes one or a combination of the following processing: removing noise points, removing mismatched points, and smoothing the disparity image; Determining the edge disparity of the initial disparity image based on the gradient map of the binocular source image and the initial disparity image specifically includes: determining the edge disparity of the post-processed initial disparity image based on the gradient map of the binocular source image and the post-processed initial disparity image.

8. The method according to claim 6, characterized in that The left view in the binocular source view is the comparison image imgL, and the right view is the reference image imgR. The binocular source image is compressed N times using the following formula to obtain the compressed comparison image imgpresL and the compressed reference image imgpresR: imgpresL(nx,ny)=(imgL(x,y)+imgL(x,y+1)…imgL(x,y+N-1)) / N imgpresR(nx,ny)=(imgR(x,y)+imgR(x,y+1)…imgR(x,y+N-1)) / N Among them, nx=1,2,…,width;ny=1,2,…,height / N,height and width represent the height and width of the comparison image imgL and the reference image imgR respectively.

9. The method according to claim 8, characterized in that Determining the edge disparity of the initial disparity image based on the gradient map of the binocular source image and the initial disparity specifically includes: Copy any one row of parallax in the initial parallax image into N rows of parallax, so that the initial parallax image is expanded N times in the height direction; For the gradient image imGradR of the reference image in the binocular source image, a gradient threshold T is set, and for the pixel position whose pixel value in the gradient imGradR is less than T, the disparity value of the same pixel position is assigned as an invalid disparity in the disparity image dispIm; Determine the pixel positions whose pixel values ​​in the gradient image imGradR are greater than the gradient threshold T as the image edge areas in the initial disparity image; For the disparity value Disp(x,y) of any pixel in the image edge area in the initial disparity image, the disparity value Disp(x,y) of the pixel is determined based on the previous pixel disparity value disp(x-1,y), the current pixel disparity value disp(x,y) and the next pixel disparity value disp(x+1,y).

10. A computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, and the computer executable instructions are used to enable the computer to execute any one of the methods of claims 6 to 9.

Images