Sub-pixel interpolation method and device for binocular stereoscopic vision

Abstract

The invention provides a sub-pixel interpolation method and device for binocular stereoscopic vision. The method comprises the following steps of: calculating an aggregation cost value of each pixel in a left eye image and a right eye image which are acquired at the same time in a preset parallax range; determining the optimal aggregation cost value of each pixel; determining an integer-level parallax value of each pixel in the left-eye image and the right-eye image according to the optimal aggregation cost value of each pixel; performing sub-pixel interpolation according to the optimal aggregation cost value of each pixel to obtain a decimal-level parallax value of each pixel; and determining a final parallax value of each pixel according to the integer-level parallax value and the decimal-level parallax value of each pixel. According to the method, the sub-pixel interpolation is carried out on the pixels, so that the distribution condition of the SGM algorithm in a spatial plane under a binocular visual angle can be remarkably refined, the finally obtained parallax value is enabled to present a smoother and more accurate characteristic, and the performance of a binocular vision system in an application scene is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of binocular stereo vision, and in particular to a sub-pixel interpolation method and device for binocular stereo vision. Background Art

[0002] Binocular stereo vision is a computer vision technology that estimates the distance and three-dimensional shape of an object by calculating disparity values. This technology is widely used in drones, autonomous driving, target detection and other fields. The Semi-Global matching (SGM) algorithm is widely used in binocular stereo vision because of its low power consumption and high disparity value accuracy.

[0003] In binocular stereo vision, for a certain pixel in an image, the disparity is the column coordinate of the pixel on the right image minus the column coordinate of the pixel on the left image, in pixels. The disparity of pixels is the basis of stereo matching algorithms. By calculating the disparity of each pixel, a disparity map can be generated. The disparity map is a two-dimensional image with the same size as the original image, and each position stores the disparity value of the pixel at that position. The disparity map can be further converted into a depth map for applications such as three-dimensional reconstruction. That is, the disparity of pixels is mainly used to generate disparity maps and depth maps, providing basic data for applications such as three-dimensional reconstruction.

[0004] However, in the traditional semi-global stereo matching algorithm, the calculated pixel disparity value is an integer value. The integer-level pixel accuracy cannot meet the needs in actual situations, which leads to low performance of the binocular vision system in application scenarios.

[0005] Therefore, the prior art has defects and needs to be improved and developed. Summary of the invention

[0006] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a sub-pixel interpolation method and device for binocular stereo vision is provided, aiming to solve the problem that the pixel disparity value calculated by the semi-global matching algorithm in the prior art is an integer value, and the integer-level pixel accuracy makes the performance of the binocular vision system in the application scenario low.

[0007] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0008] A sub-pixel interpolation method for binocular stereo vision, wherein the method comprises:

[0009] Calculate the aggregate cost value of each pixel in the left eye image and the right eye image collected simultaneously within a preset parallax range;

[0010] determining an optimal aggregate cost value for each pixel based on the aggregate cost value for each pixel;

[0011] Determine an integer-level disparity value of each pixel in the left-eye image and the right-eye image according to the optimal aggregation cost value of each pixel;

[0012] Performing sub-pixel interpolation according to the optimal aggregation cost value of each pixel to obtain a decimal disparity value of each pixel;

[0013] A final disparity value of each pixel is determined according to the integer-level disparity value and the fractional-level disparity value of each pixel.

[0014] In one embodiment of the present application, calculating the aggregation cost value of each pixel in the left-eye image and the right-eye image collected simultaneously within a preset disparity range includes:

[0015] For each pixel in the left-eye image and the right-eye image, calculating the matching cost at different parallaxes within a preset parallax range to obtain initial cost values ​​at different parallaxes;

[0016] The initial cost value is regionally optimized to obtain an aggregated cost value of each pixel under different parallaxes.

[0017] In one embodiment of the present application, regional optimization is performed on the initial cost value to obtain an aggregate cost value of each pixel under different parallaxes, including:

[0018] Divide every four initial cost values ​​into a region, and select the smallest initial cost value as the optimized cost value of the current region;

[0019] The optimized cost values ​​are aggregated in different preset directions to obtain an aggregated cost value of each pixel under different parallaxes.

[0020] In one embodiment of the present application, determining the optimal aggregation cost value of each pixel based on the aggregation cost value of each pixel includes:

[0021] Find the minimum aggregate cost value of each pixel among the aggregate cost values ​​of each pixel through the selector;

[0022] The minimum aggregation cost value is taken as the optimal aggregation cost value.

[0023] In one embodiment of the present application, sub-pixel interpolation is performed according to the optimal aggregation cost value of each pixel to obtain a decimal disparity value of each pixel, including:

[0024] Determine adjacent disparity values ​​of an optimal disparity for each pixel, wherein the adjacent disparity values ​​include a left disparity value and a right disparity value;

[0025] Obtaining an aggregation cost value corresponding to the left disparity value and an aggregation cost value corresponding to the right disparity value;

[0026] Calculating a first difference between an aggregation cost value corresponding to the left disparity value and an optimal aggregation cost value, and calculating a second difference between an aggregation cost value corresponding to the right disparity value and the optimal aggregation cost value;

[0027] Using the second difference as a dividend and the first difference as a divisor, inputting the two values ​​into a divider to obtain an output value of the divider;

[0028] The decimal disparity value of each pixel is obtained according to the output value of the divider and a preset cosine value lookup table.

[0029] In one embodiment of the present application, the second difference is used as a dividend, the first difference is used as a divisor, and the result is input into a divider to obtain a divider output value, including:

[0030] Using the second difference as the dividend and the first difference as the divisor, inputting the two into a radix-4 divider, initializing the dividend and the divisor by binary division, and performing iteration;

[0031] In the iteration process, a quotient is obtained by using a flying conversion method and a shift instead of a subtraction, and a divider output value is obtained based on the quotient obtained in the iteration process.

[0032] In one embodiment of the present application, determining a final disparity value of each pixel according to the integer-level disparity value and the decimal-level disparity value of each pixel includes:

[0033] The integer-level disparity value and the decimal-level disparity value of each pixel are added together to obtain a final disparity value of each pixel.

[0034] The present application also provides a sub-pixel interpolation device for binocular stereo vision, wherein the device comprises:

[0035] A cost value calculation module, used to calculate the aggregate cost value of each pixel in the left eye image and the right eye image collected simultaneously within a preset parallax range;

[0036] an optimization module, configured to determine an optimal aggregation cost value for each pixel based on the aggregation cost value for each pixel;

[0037] An integer disparity determination module, configured to determine an integer disparity value of each pixel in the left-eye image and the right-eye image according to the optimal aggregation cost value of each pixel;

[0038] A sub-pixel interpolation module, used for performing sub-pixel interpolation according to the optimal aggregation cost value of each pixel to obtain a decimal disparity value of each pixel;

[0039] The final disparity determination module is used to determine the final disparity value of each pixel according to the integer-level disparity value and the decimal-level disparity value of each pixel.

[0040] The present application also provides a terminal, which includes: a memory, a processor, and a sub-pixel interpolation program for binocular stereoscopic vision stored in the memory and executable on the processor, wherein the sub-pixel interpolation program for binocular stereoscopic vision implements the steps of the sub-pixel interpolation method for binocular stereoscopic vision as described above when executed by the processor.

[0041] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the sub-pixel interpolation method for binocular stereo vision as described above.

[0042] The present invention provides a sub-pixel interpolation method and device for binocular stereo vision, the method comprising: calculating the aggregation cost value of each pixel in the left-eye image and the right-eye image collected simultaneously within a preset disparity range; determining the optimal aggregation cost value of each pixel based on the aggregation cost value of each pixel; determining the integer-level disparity value of each pixel in the left-eye image and the right-eye image according to the optimal aggregation cost value of each pixel; performing sub-pixel interpolation according to the optimal aggregation cost value of each pixel to obtain the decimal-level disparity value of each pixel; determining the final disparity value of each pixel according to the integer-level disparity value and the decimal-level disparity value of each pixel. The present application can significantly refine the distribution of the spatial plane of the SGM algorithm under the binocular perspective by performing sub-pixel interpolation on pixels, thereby making the final disparity value present a smoother and more accurate characteristic, thereby improving the performance of the binocular vision system in the application scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of a preferred embodiment of the sub-pixel interpolation method for binocular stereo vision in the present invention;

[0044] Figure 2 It is a logical architecture diagram of sub-pixel interpolation parallax calculation in the present invention;

[0045] Figure 3 It is a functional principle block diagram of a preferred embodiment of a sub-pixel interpolation device for binocular stereo vision in the present invention;

[0046] Figure 4 It is a functional principle block diagram of a preferred embodiment of the terminal in the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0048] When calculating the disparity of pixels, the RTSGM algorithm reduces resource consumption by downsampling images with a smaller disparity range, but the algorithm's operating frequency is relatively low, only 14.5Hz, which makes the algorithm not well suited for high-speed or real-time processing applications. The SGM processor based on pipeline and regional optimization methods balances resource consumption and processing speed and can meet the needs of actual situations, but the disparity value of this processor only contains integers. In actual situations, integer-level pixel accuracy cannot meet the requirements, and integer-level pixel accuracy will lead to reduced accuracy of the SGM algorithm in long-distance applications. Therefore, the key challenge of SGM is how to balance speed and resource allocation while improving the accuracy of disparity values.

[0049] The present invention combines a resource-optimized cosine lookup table with an SRT divider, adopts a pixel-level pipeline architecture, and uses the maximum error between two adjacent pixels as a metric for function fitting to accurately perform sub-pixel interpolation operations on target pixels. Compared with existing methods, the sub-pixel interpolation technology of the present invention can significantly refine the distribution of the spatial plane of the SGM algorithm under binocular perspective, thereby making the final object surface disparity value present a smoother and more accurate characteristic, effectively improving the performance of the binocular vision system in application scenarios such as three-dimensional reconstruction.

[0050] See also Figure 1 , Figure 1 Flowchart of the sub-pixel interpolation method for binocular stereo vision in the present invention. Figure 1 As shown, the sub-pixel interpolation method for binocular stereo vision described in the embodiment of the present invention includes:

[0051] Step S100: calculating the aggregation cost value of each pixel in the left eye image and the right eye image collected simultaneously within a preset parallax range.

[0052] This application introduces sub-pixel interpolation to improve the accuracy of the result. Specifically, in the traditional SGM disparity calculation, the binocular perspective space is divided into n (n = maximum disparity value) planes, but the actual scene is continuous and gradually changing. Therefore, in order to make the object surface disparity smoother and the disparity accuracy higher, the present invention will perform sub-pixel interpolation on it.

[0053] Specifically, the prototype of the classic sub-pixel interpolation function is:

[0054] d final =d+f(m d-1,m d , m d+1 );

[0055] Among them, d final represents the final disparity, d represents the integer disparity, f(m d-1 , m d , m d+1 ) indicates sub-pixel interpolation, m d-1 represents the aggregation cost of a pixel when the disparity value is d-1, m d represents the aggregation cost of a pixel when the disparity value is d, m d+1 Indicates the aggregation cost of a pixel when the disparity value is d+1.

[0056] Since it is difficult to model three independent input variables, we transform the three independent variables into two:

[0057] left m =m d-1 -m d ;

[0058] right m =m d+1 -m d ;

[0059] Among them, left m is the first difference between the aggregation cost value corresponding to the left disparity value and the optimal aggregation cost value, right m is the second difference between the aggregation cost value corresponding to the right disparity value and the optimal aggregation cost value.

[0060] The mathematical model using polar coordinate angles and the principle of symmetry is:

[0061]

[0062] In order to make the system more robust, it is important to consider the worst-case error. Therefore, the present invention uses an interpolation formula that uses the maximum error as a function fitting metric, rather than a formula that uses the sum of errors as a function fitting metric. The sub-pixel interpolation formula used in the present invention is as follows:

[0063]

[0064] In the embodiment of the present application, the step S100 specifically includes:

[0065] Step S110, for each pixel in the left-eye image and the right-eye image, calculating the matching cost at different parallaxes within a preset parallax range to obtain initial cost values ​​at different parallaxes;

[0066] Step S120: performing regional optimization on the initial cost value to obtain an aggregated cost value of each pixel under different parallaxes.

[0067] Specifically, see Figure 2 , the preset different directions of the present application are preferably 0 degree and 135 degree directions. The present application inputs cost values ​​in the 0 degree and 135 degree directions respectively. In order to reduce resource consumption, the cost value used in the present invention is the cost value after regional optimization.

[0068] In the embodiment of the present application, the step S120 specifically includes:

[0069] Step S121, divide every four initial cost values ​​into one region, and select the smallest initial cost value as the optimized cost value of the current region;

[0070] Step S122: Aggregate the optimized cost values ​​in different preset directions to obtain aggregated cost values ​​of each pixel under different parallaxes.

[0071] Specifically, regional optimization means dividing every 4 initial cost values ​​into a group, selecting the smallest cost value in each group to represent the cost value of the group, so a region consisting of 4 cost values ​​is optimized into 1 cost value and a position mark, and the position mark refers to a region consisting of 4 cost values. For example, the position marks of the four values ​​(10, 2, 39, 7) are (0, 1, 2, 3) respectively, the smallest cost value is 2, and its position mark is 1. When the parallax range is 128, the optimized cost values ​​of the 32 regions in both directions are aggregated to obtain the aggregated cost value of each pixel.

[0072] like Figure 1 As shown, the sub-pixel interpolation method for binocular stereo vision described in the embodiment of the present invention includes:

[0073] Step S200: Determine the optimal aggregation cost value of each pixel based on the aggregation cost value of each pixel.

[0074] In one embodiment of the present application, step S200 specifically includes:

[0075] Step S210: Find the minimum aggregation cost value of each pixel among the aggregation cost values ​​of each pixel through a selector;

[0076] Step S220: taking the minimum aggregation cost value as the optimal aggregation cost value.

[0077] Specifically, five groups of selectors are used to find the minimum aggregation cost value, which is the optimal aggregation cost value, denoted as disp w1 , whose position is marked as disp w2.

[0078] like Figure 1 As shown, the sub-pixel interpolation method for binocular stereo vision described in this embodiment also includes:

[0079] Step S300: Determine an integer-level disparity value of each pixel in the left-eye image and the right-eye image according to the optimal aggregation cost value of each pixel.

[0080] Specifically, it is determined whether the current region is on an edge. The region edge refers to the edge of the aggregated cost value. For example, when the parallax range d is 128, the optimized cost values ​​of the 32 regions in both directions are aggregated to obtain the aggregated cost value m of each pixel. d When d is equal to 0 or 31, it is the edge. If it is not the edge area, it is disp w1 ×4+disp w2 The integer disparity value is calculated.

[0081] like Figure 1 As shown, the sub-pixel interpolation method for binocular stereo vision described in this embodiment also includes:

[0082] Step S400: performing sub-pixel interpolation according to the optimal aggregation cost value of each pixel to obtain a decimal-level disparity value of each pixel.

[0083] In the embodiment of the present application, the step S400 specifically includes:

[0084] Step S410, determining adjacent disparity values ​​of the optimal disparity of each pixel, wherein the adjacent disparity values ​​include a left disparity value and a right disparity value;

[0085] Step S420, obtaining an aggregation cost value corresponding to the left disparity value and an aggregation cost value corresponding to the right disparity value;

[0086] Step S430: calculating a first difference between the aggregation cost value corresponding to the left disparity value and the optimal aggregation cost value, and calculating a second difference between the aggregation cost value corresponding to the right disparity value and the optimal aggregation cost value;

[0087] Step S440, taking the second difference as the dividend and the first difference as the divisor, inputting the two values ​​into a divider to obtain an output value of the divider;

[0088] Step S450: Obtain a decimal-level disparity value of each pixel according to the divider output value and a preset cosine value lookup table.

[0089] This application calculates the difference between the minimum aggregation cost value and its neighboring aggregation cost value. Here, the neighboring refers to the adjacent disparity value. For example, if the disparity value of the optimal aggregation cost value is d=10, then the first difference left m is the aggregation cost when the parallax value is 9, the second parallax right m It is the aggregation cost when the parallax value is 11.

[0090] The present application searches the preset cosine value lookup table according to the output value of the divider to obtain the corresponding decimal parallax value, that is, the cosine value lookup table after resource optimization is combined, and the angle of the cosine value lookup table is divided into 6 blocks, thereby realizing resource optimization of the lookup table and improving the search efficiency. The cosine value lookup table includes: the correspondence between the input value and the cosine value, which can be stored as a list or in the form of an operation formula. The operation formula of the cosine value lookup table is the sub-pixel interpolation formula of the present application. In part.

[0091] In the embodiment of the present application, the step S440 specifically includes:

[0092] Step S441, taking the second difference as the dividend and the first difference as the divisor, inputting them into a radix-4 divider, initializing the dividend and the divisor by binary division, and performing iteration;

[0093] Step S442: During the iteration process, a quotient is obtained by using a flying conversion method and a shift instead of a subtraction, and a divider output value is obtained based on the quotient obtained during the iteration process.

[0094] Specifically, right m As the dividend x,left m As the divisor d, input the base 4 divider. In the divider, the dividend x and divisor d are first initialized by binary division. The following form is adjusted by shifting:

[0095] X=s00.001x 1 x 2 ......, D = s00.1d 1 d 2 ......;

[0096] The initialized dividend is recorded as X, and the initialized divisor is recorded as D. Among them, s is the sign bit, x n Represents the nth digit of the dividend x, d n Represents the nth digit of the divisor d, for example, x=011, x 0 =0,x 1=1……. Then the iteration begins. The partial remainder W of the first iteration is X. The partial remainder W and the first 4 decimal places of the divisor D are input into the quotient selection function lookup table to determine the quotient q selected for this iteration. Then the partial remainder W of the second iteration is calculated according to the following formula:

[0097] W j+1 =4×W j -D×q j+1 ;

[0098] Where j represents the jth iteration, W j is the partial remainder from the jth iteration, q j+1 is the quotient value corresponding to the j+1th iteration, and D is the initialized divisor. Subsequent iterations are performed in the above manner, and 7 quotient values ​​q are obtained after 7 iterations. In order to simplify the quotient data set, a quotient data set represented by redundant numbers is used, but negative values ​​will appear in the quotient values. Therefore, an adder is finally used to subtract the positive and negative parts of the quotient to obtain the final quotient. In order to reduce the delay of bit transfer and the inefficiency of correction when the remainder is negative, the present application adopts On-the-Fly Conversion and shifting instead of subtraction to obtain the final quotient. Each iteration will simultaneously generate two quotient values ​​q and q-ulp (ulp is unit of last position), and these two quotient values ​​will be stored in two registers (positive quotient register Q and negative quotient register Q) respectively. m ). The specific formula is as follows:

[0099]

[0100] Among them, Q[j+1] represents the number in the positive quotient register after the j+1th iteration, Q m [j+1] represents the value in the negative quotient register after iteration j+1. For a given floating-point format, the ulp of a particular real value is the distance between the two nearest floating-point numbers to the left and right of the real value.

[0101] Finally, the corresponding quotient register is selected according to the sign of the remainder. If the remainder is negative, the quotient register is a negative quotient register; otherwise, it is a positive quotient register. The decimal point of the quotient is adjusted according to the shift result during initialization to obtain the output value of the divider.

[0102] This application adopts a radix 4 divider, which is implemented based on the SRT (Sweeney, Robertson, and Tocher Algorithm) algorithm, and has the characteristics of accurate calculation and short delay. There are various ways to implement the division operation. In addition to the radix 4 divider, the restoring or non-restoring remainder method or approximate calculation method can also be used.

[0103] When performing sub-pixel interpolation, the present application ensures high accuracy and calculation speed of the algorithm by using a radix-4 divider based on a flying conversion method.

[0104] like Figure 1 As shown, the sub-pixel interpolation method for binocular stereo vision described in this embodiment also includes:

[0105] Step S500: determining a final disparity value of each pixel according to the integer-level disparity value and the decimal-level disparity value of each pixel.

[0106] In the embodiment of the present application, the step S500 is specifically: adding the integer-level disparity value and the decimal-level disparity value of each pixel to obtain a final disparity value of each pixel.

[0107] The final disparity value of the present application can be composed of 7 integer bits and 9 decimal bits, but the number of integer bits and decimal bits can be flexibly adjusted according to actual needs. The divider type can also be changed, such as changing the fixed-point divider to a floating-point divider. Correspondingly, the final disparity value representation will also be changed from a fixed 16-bit 7-bit integer plus a 9-bit decimal form to a floating-point form. In addition, the interpolation formula used in the present application can also be replaced by other interpolation formulas, such as linear interpolation, etc., to meet the interpolation requirements in different scenarios.

[0108] The sub-pixel interpolation formula used in the present invention has been screened through a large number of tests. By using the maximum error between two adjacent pixels as the metric for function fitting, the sub-pixel interpolation operation can be accurately performed on the target pixel. The sub-pixel interpolation technology of the present invention can significantly refine the distribution of the spatial plane of the SGM algorithm under the binocular perspective, thereby making the final parallax value of the object surface present a smoother and more accurate characteristic, effectively improving the performance of the binocular vision system in application scenarios such as three-dimensional reconstruction.

[0109] In one embodiment, if Figure 3 As shown, based on the above sub-pixel interpolation method for binocular stereo vision, the present invention also provides a sub-pixel interpolation device for binocular stereo vision, including:

[0110] The cost value calculation module 100 is used to calculate the aggregate cost value of each pixel in the left eye image and the right eye image collected simultaneously within a preset parallax range;

[0111] An optimization module 200, configured to determine an optimal aggregation cost value for each pixel based on the aggregation cost value for each pixel;

[0112] An integer disparity determination module 300 is used to determine an integer disparity value of each pixel in the left-eye image and the right-eye image according to the optimal aggregation cost value of each pixel;

[0113] A sub-pixel interpolation module 400, configured to perform sub-pixel interpolation according to the optimal aggregation cost value of each pixel to obtain a decimal disparity value of each pixel;

[0114] The final disparity determination module 500 is used to determine the final disparity value of each pixel according to the integer-level disparity value and the decimal-level disparity value of each pixel.

[0115] Figure 4 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal may include:

[0116] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0117] When the processor 502 executes the program, the sub-pixel interpolation method for binocular stereo vision provided in the above embodiment is implemented.

[0118] Furthermore, the terminal further includes:

[0119] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0120] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0121] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0122] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0123] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0124] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0125] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the sub-pixel interpolation method for binocular stereo vision as described above is implemented.

[0126] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0127] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0128] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes additional implementations, in which the order shown or discussed may not be followed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0129] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can read instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary and then storing it in a computer memory.

[0130] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0131] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0132] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above-mentioned embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above-mentioned embodiments within the scope of the present application.

[0133] In summary, the present invention discloses a sub-pixel interpolation method and device for binocular stereo vision, the method comprising: calculating the aggregation cost value of each pixel in the left-eye image and the right-eye image collected simultaneously within a preset disparity range; determining the optimal aggregation cost value of each pixel based on the aggregation cost value of each pixel; determining the integer-level disparity value of each pixel in the left-eye image and the right-eye image according to the optimal aggregation cost value of each pixel; performing sub-pixel interpolation according to the optimal aggregation cost value of each pixel to obtain the decimal-level disparity value of each pixel; determining the final disparity value of each pixel according to the integer-level disparity value and the decimal-level disparity value of each pixel. The present application can significantly refine the distribution of the spatial plane of the SGM algorithm under the binocular perspective by performing sub-pixel interpolation on pixels, thereby making the final disparity value present a smoother and more accurate characteristic, thereby improving the performance of the binocular vision system in the application scenario.

[0134] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A sub-pixel interpolation method for binocular stereo vision, characterized in that: The method comprises: Calculate the aggregate cost value of each pixel in the left eye image and the right eye image collected simultaneously within a preset parallax range; determining an optimal aggregate cost value for each pixel based on the aggregate cost value for each pixel; Determine an integer-level disparity value of each pixel in the left-eye image and the right-eye image according to the optimal aggregation cost value of each pixel; Performing sub-pixel interpolation according to the optimal aggregation cost value of each pixel to obtain a decimal disparity value of each pixel; A final disparity value of each pixel is determined according to the integer-level disparity value and the fractional-level disparity value of each pixel.

2. The sub-pixel interpolation method for binocular stereo vision according to claim 1, characterized in that: The aggregate cost value of each pixel in the left and right images collected simultaneously is calculated within the preset disparity range, including: For each pixel in the left-eye image and the right-eye image, calculating the matching cost at different parallaxes within a preset parallax range to obtain initial cost values ​​at different parallaxes; The initial cost value is regionally optimized to obtain an aggregated cost value of each pixel under different parallaxes.

3. The sub-pixel interpolation method for binocular stereo vision according to claim 2, characterized in that: The initial cost value is regionally optimized to obtain an aggregate cost value of each pixel under different parallaxes, including: Divide every four initial cost values ​​into a region, and select the smallest initial cost value as the optimized cost value of the current region; The optimized cost values ​​are aggregated in different preset directions to obtain an aggregated cost value of each pixel under different parallaxes.

4. The sub-pixel interpolation method for binocular stereo vision according to claim 3, characterized in that: Determining an optimal aggregate cost value for each pixel based on the aggregate cost value for each pixel includes: Find the minimum aggregate cost value of each pixel among the aggregate cost values ​​of each pixel through the selector; The minimum aggregation cost value is taken as the optimal aggregation cost value.

5. The sub-pixel interpolation method for binocular stereo vision according to claim 3, characterized in that: Sub-pixel interpolation is performed according to the optimal aggregation cost value of each pixel to obtain a decimal disparity value of each pixel, including: Determine adjacent disparity values ​​of an optimal disparity for each pixel, wherein the adjacent disparity values ​​include a left disparity value and a right disparity value; Obtaining an aggregation cost value corresponding to the left disparity value and an aggregation cost value corresponding to the right disparity value; Calculating a first difference between an aggregation cost value corresponding to the left disparity value and an optimal aggregation cost value, and calculating a second difference between an aggregation cost value corresponding to the right disparity value and the optimal aggregation cost value; Using the second difference as a dividend and the first difference as a divisor, inputting the two values ​​into a divider to obtain an output value of the divider; The decimal disparity value of each pixel is obtained according to the output value of the divider and a preset cosine value lookup table.

6. The sub-pixel interpolation method for binocular stereo vision according to claim 5, characterized in that: The second difference is used as a dividend and the first difference is used as a divisor, and the two are input into a divider to obtain a divider output value, including: Using the second difference as the dividend and the first difference as the divisor, inputting the two into a radix-4 divider, initializing the dividend and the divisor by binary division, and performing iteration; In the iteration process, a quotient is obtained by using a flying conversion method and a shift instead of a subtraction, and a divider output value is obtained based on the quotient obtained in the iteration process.

7. The sub-pixel interpolation method for binocular stereo vision according to claim 1, characterized in that: Determining a final disparity value of each pixel according to the integer-level disparity value and the fractional-level disparity value of each pixel comprises: The integer-level disparity value and the decimal-level disparity value of each pixel are added together to obtain a final disparity value of each pixel.

8. A sub-pixel interpolation device for binocular stereo vision, characterized in that: The device comprises: A cost value calculation module, used to calculate the aggregate cost value of each pixel in the left eye image and the right eye image collected simultaneously within a preset parallax range; an optimization module, configured to determine an optimal aggregation cost value for each pixel based on the aggregation cost value for each pixel; An integer disparity determination module, configured to determine an integer disparity value of each pixel in the left-eye image and the right-eye image according to the optimal aggregation cost value of each pixel; A sub-pixel interpolation module, used for performing sub-pixel interpolation according to the optimal aggregation cost value of each pixel to obtain a decimal disparity value of each pixel; The final disparity determination module is used to determine the final disparity value of each pixel according to the integer-level disparity value and the decimal-level disparity value of each pixel.

9. A terminal, characterized in that: include: A memory, a processor, and a sub-pixel interpolation program for binocular stereoscopic vision stored in the memory and executable on the processor, wherein the sub-pixel interpolation program for binocular stereoscopic vision, when executed by the processor, implements the steps of the sub-pixel interpolation method for binocular stereoscopic vision as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the sub-pixel interpolation method for binocular stereo vision as described in any one of claims 1 to 7.

Images