A parallax correction method and device, electronic equipment, storage medium and product

Abstract

This application discloses a disparity correction method, apparatus, electronic device, storage medium, and product. The method includes: determining an initial disparity map between a left and right image acquired by a binocular camera based on a binocular matching disparity network; determining matching pixel pairs between the left and right images based on the initial disparity map; for each matching pixel pair, traversing all disparities within the disparity neighborhood centered on the initial disparity corresponding to the matching pixel pair, and calculating the aggregation cost of the target pixel within a preset-sized pixel window in its target image based on the current disparity; determining the target disparity corresponding to the target pixel based on the aggregation cost corresponding to each disparity in the disparity neighborhood, and correcting the corresponding initial disparity in the initial disparity map based on the target disparity to generate a target disparity map. This solution not only effectively ensures the robustness of deep learning-based matching disparity but also improves the accuracy of binocular visual disparity.

Description

Technical Field

[0001] This application relates to the field of binocular vision technology, and in particular to a parallax correction method, device, electronic device, storage medium, and product. Background Technology

[0002] Stereo matching is a crucial part of binocular vision, and its accuracy directly impacts the precision of the binocular visual point cloud. Currently, there are two main methods for calculating binocular disparity: deep learning-based binocular matching (such as RAFT-Stereo) and traditional binocular matching (such as SGM and SGBM). Deep learning-based methods offer robustness and generate smooth disparity maps, but they are less effective under varying lighting conditions and are less efficient at sub-pixel matching. Traditional binocular matching methods are less robust, more susceptible to environmental influences, and have lower disparity accuracy, but they are more efficient at sub-pixel matching. Summary of the Invention

[0003] This application provides a parallax correction method, device, electronic device, storage medium, and product, which not only effectively ensures the robustness of deep learning-based parallax matching but also improves the accuracy of binocular visual parallax.

[0004] According to one aspect of this application, a parallax correction method is provided, the method comprising: The initial disparity map between the left and right images acquired by the binocular camera is determined based on a binocular matching disparity network. Based on the initial disparity map, determine each pair of matching pixel points between the left image and the right image; For each pair of matching pixels, traverse each disparity in the disparity neighborhood centered on the initial disparity corresponding to the pair of matching pixels, and calculate the aggregation cost of the target pixel within a preset-sized pixel window in the target image based on the current disparity; wherein, the target pixel is a pixel in the pair of matching pixels that belongs to the left image or the right image; The target disparity corresponding to the target pixel is determined based on the aggregation cost corresponding to each disparity in the disparity neighborhood, and the initial disparity in the initial disparity map is corrected based on the target disparity to generate a target disparity map.

[0005] According to one aspect of this application, a parallax correction device is provided, the device comprising: The initial disparity map determination module is used to determine the initial disparity map between the left and right images acquired by the binocular camera based on the binocular matching disparity network. The matching pixel pair determination module is used to determine each set of matching pixel pairs between the left image and the right image based on the initial disparity map; The aggregation cost determination module is used to, for each pair of matching pixel points, traverse each disparity in the disparity neighborhood centered on the initial disparity corresponding to the matching pixel point pair, and calculate the aggregation cost of the target pixel point in the target image of a preset size pixel window based on the current disparity; wherein, the target pixel point is the pixel point in the matching pixel point pair that belongs to the left image or the right image; A target disparity determination module is used to determine the target disparity corresponding to the target pixel based on the aggregation cost corresponding to each disparity in the disparity neighborhood, and to correct the initial disparity in the initial disparity map based on the target disparity to generate a target disparity map. According to another aspect of this application, an electronic device is provided, comprising: At least one processor; and A memory that is communicatively connected to at least one processor; wherein, The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the parallax correction method of any embodiment of the present application.

[0006] According to another aspect of this application, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the parallax correction method of any embodiment of this application.

[0007] According to another aspect of this application, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the parallax correction method of any embodiment of this application.

[0008] The technical solution of this application embodiment, in response to the triggering of a binocular visual disparity determination event, determines an initial disparity map between the left and right images captured by the binocular camera based on a binocular matching disparity network; determines each pair of matching pixel points between the left and right images according to the initial disparity map; for each pair of matching pixel points, traverses each disparity in the disparity neighborhood centered on the initial disparity corresponding to the matching pixel point pair, and calculates the aggregation cost of the target pixel point within a preset-sized pixel window in its target image based on the current disparity; wherein, the target pixel point is a pixel point in the matching pixel point pair belonging to the left or right image; determines the target disparity corresponding to the target pixel point based on the aggregation cost corresponding to each disparity in the disparity neighborhood, and corrects the initial disparity corresponding to the initial disparity map based on the target disparity to generate a target disparity map. This solution corrects the initial disparity map obtained based on deep learning matching based on a binocular matching method, which not only effectively ensures the robustness of deep learning matching disparity but also improves the accuracy of binocular visual disparity.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart of a parallax correction method provided in an embodiment of this application; Figure 2 This application provides an embodiment of a target pixel-centered method. Pixel window diagram; Figure 3 This is a schematic diagram of the structure of a parallax correction device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0013] It should be noted that the terms "first," "second," "third," "fourth," "actual," "preset," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0014] Figure 1 This is a flowchart illustrating a disparity correction method provided in an embodiment of this application. This embodiment is applicable to correcting binocular visual disparity obtained based on deep learning matching. The method can be executed by a disparity correction device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes: S110. Determine the initial disparity map between the left and right images acquired by the binocular camera based on a binocular matching disparity network.

[0015] In this embodiment, in response to a binocular visual disparity determination event being triggered, left and right images captured by the binocular cameras are acquired. The left image is captured by the left camera in the binocular camera system, and the right image is captured by the right camera. After stereo correction of the left and right images, they are input into a pre-constructed binocular matching disparity network. A disparity map between the left and right images is determined based on the output of the binocular matching disparity network. The disparity map contains either the initial disparity between each pixel in the left image and its corresponding pixel in the right image, or the initial disparity between each pixel in the right image and its corresponding pixel in the left image.

[0016] S120. Determine each set of matching pixel pairs between the left image and the right image based on the initial disparity map.

[0017] In this embodiment of the application, each pixel in the left image is traversed. Determine pixel points from the initial disparity map Corresponding initial disparity And based on the initial disparity Determine the pixel points in the right image that correspond to those in the left image. Matching pixels ,in, Understandably, following the above method, pixels in the right image that match each pixel in the left image can be determined, and the two matching pixels between the left and right images are called matching pixel pairs. Optionally, the initial disparity corresponding to each pixel in the right image can be determined from the initial disparity map, and pixels in the left image that match each pixel in the right image can be determined based on the initial disparity, thereby determining each set of matching pixel pairs between the left and right images.

[0018] S130. For each pair of matching pixels, traverse each disparity in the disparity neighborhood centered on the initial disparity corresponding to the pair of matching pixels, and calculate the aggregation cost of the target pixel in the target image of a preset size pixel window based on the current disparity; wherein, the target pixel is the pixel in the pair of matching pixels that belongs to the left image or the right image.

[0019] In this embodiment of the application, for each pair of matched pixels, the initial disparity corresponding to the pair of matched pixels is determined. parallax neighborhood centered on ,in, Then, it iterates through each disparity in the disparity neighborhood and calculates the aggregation cost of the target pixel within a preset-sized pixel window in its respective target image based on the current disparity. For example, if the disparity neighborhood is... Then the disparity neighborhood The parallaxes within are respectively , , Then based on parallax Parallax and parallax The aggregation cost of a target pixel within a preset-sized pixel window in its target image is calculated. The target pixel is either a pixel belonging to the left or right image in a matching pixel pair. That is, the target pixel can be a pixel belonging to the left image in a matching pixel pair, in which case the target image to which the target pixel belongs is the left image; conversely, the target pixel can be a pixel belonging to the right image in a matching pixel pair, in which case the target image to which the target pixel belongs is the right image. In this embodiment, the aggregation cost of the target pixel within a preset-sized pixel window in its target image can be calculated based on the SGM cost aggregation calculation principle, using the current disparity. For example, if a matching pixel pair consists of a first pixel in the left image and a second pixel in the right image, then either the first pixel or the second pixel can be used as the target pixel.

[0020] Optionally, calculating the aggregation cost of the target pixel within a preset-sized pixel window in the target image based on the current disparity includes: calculating the matching cost between the matching pixel pairs; wherein the matching cost reflects the similarity between the matching pixel pairs; in the target image to which the target pixel belongs, for each neighboring pixel within a preset-sized pixel window centered on the target pixel (excluding the target pixel), calculating the path cost of the neighboring pixel based on the matching cost and the current disparity; and using the sum of the path costs corresponding to each neighboring pixel as the aggregation cost of the target pixel within the preset-sized pixel window in the target image.

[0021] In this embodiment, a matching cost is calculated between matching pixel pairs. The matching cost can be the grayscale value matching degree within a preset-sized pixel window centered on the corresponding pixel in the matching pixel pair in the left and right images. The matching cost reflects the similarity between matching pixel pairs; a smaller matching cost indicates a higher similarity between the matching pixel pairs. Optionally, calculating the matching cost between the matching pixel pairs includes: calculating the pixel grayscale value difference between each pair of pixels within a preset-sized pixel window centered on the corresponding pixel in the matching pixel pair in the left and right images respectively; calculating the matching cost between the matching pixel pairs based on each pixel grayscale value difference; wherein the matching cost is the sum of the absolute values ​​of each pixel grayscale value difference.

[0022] In this embodiment of the application, a preset-size pixel window (e.g., centered on the first pixel in the matching pixel pair) is determined in the left image. The left and right images each contain a pixel window of a predetermined size, centered on the second pixel of a matching pixel pair. For ease of description, the pixel window centered on the first pixel of the matching pixel pair in the left image is referred to as the first pixel window, and the pixel window centered on the second pixel of the matching pixel pair in the right image is referred to as the second pixel window. Pairs of pixels are determined within the first and second pixel windows, where a pair of pixels consists of corresponding pixels at the same position in the first and second pixel windows. The pixel grayscale differences between each pair of pixels are determined, and the matching cost between matching pixel pairs is calculated based on these differences; the matching cost is the sum of the absolute values ​​of the pixel grayscale differences.

[0023] In the target image containing the target pixel, for each neighboring pixel other than the target pixel within a preset-sized pixel window centered on the target pixel, the path cost of the neighboring pixels is calculated based on the matching cost and the current disparity. For example, Figure 2 This application provides an embodiment of a target pixel-centered method. A pixel window diagram, such as Figure 2 As shown, the white pixel is the target pixel, and the eight green pixels surrounding the target pixel are its neighboring pixels. This is relevant to the target image where the target pixel is located. Centered Excluding the target pixel within the pixel window Each neighboring pixel outside the target pixel, based on the target pixel The corresponding matching cost and the path cost of the neighboring pixels for the current disparity calculation.

[0024] Optionally, calculating the path cost of the neighboring pixels based on the matching cost and the current disparity includes: determining the left-side neighboring disparity and the right-side neighboring disparity of the current disparity; determining matching neighboring pixels corresponding to the neighboring pixels from another image based on the current disparity, the left-side neighboring disparity, and the right-side neighboring disparity, and forming neighboring pixel pairs based on the neighboring pixels and the matching neighboring pixels; wherein, the other image is the image other than the target image in the left image and the right image; calculating the matching cost between the corresponding neighboring pixel pairs based on the current disparity, the left-side neighboring disparity, and the right-side neighboring disparity; and determining the path cost of the neighboring pixels based on the matching cost corresponding to the matching cost, the current disparity, the left-side neighboring disparity, and the right-side neighboring disparity, respectively.

[0025] In this embodiment of the application, taking the pixel belonging to the left image in the matched pixel pair as the target pixel as an example, and taking the current disparity as the disparity neighborhood. Parallax within For example, we will illustrate how to calculate the path cost of neighboring pixels. Determine the current disparity. left adjacent parallax and adjacent parallax on the right Based on the current parallax Left adjacent parallax and adjacent parallax on the right From the right image, identify the matching neighboring pixels that correspond to the neighboring pixels in the left image, and construct neighboring pixel pairs based on the neighboring pixels and the matching neighboring pixels. It is understandable that for different disparities (current disparity)... Left adjacent parallax or right adjacent parallax The matching neighboring pixels determined from the right image are different from those corresponding to the neighboring pixels in the left image; therefore, the neighboring pixel pairs determined based on different disparities are different. For different disparities (current disparity)... Left adjacent parallax or right adjacent parallax For the corresponding neighboring pixel pair, determine the matching cost of the neighboring pixel pair. The matching cost of the neighboring pixel pair is determined in the same way as the matching cost between the matching pixel pairs mentioned above, and will not be repeated here.

[0026] Based on the matching cost between matching pixel pairs and the current disparity Left adjacent parallax or right adjacent parallax The path cost of neighboring pixels is determined by calculating the corresponding matching cost. For example, the path cost of neighboring pixels can be calculated using the following formula:

[0027] in, For target pixel , This is the current disparity value. To match the matching cost between pixel pairs, express Figure 2 Any neighboring pixel in the green square shown, The neighboring pixels of the target pixel At the current parallax The matching cost, The neighboring pixels of the target pixel At the current parallax left adjacent parallax The matching cost, The neighboring pixels of the target pixel At the current parallax right adjacent parallax The matching cost, This is a penalty parameter, which can be set based on experience.

[0028] In this embodiment, the sum of the path costs corresponding to each neighboring pixel of the target pixel is used as the aggregate cost of the target pixel within a preset-sized pixel window in the target image. It can be understood that, for each pair of matching pixels, the aggregate cost corresponding to each disparity within the disparity neighborhood centered on the initial disparity of the matching pixel pair can be determined in the manner described above.

[0029] S140. Determine the target disparity corresponding to the target pixel based on the aggregation cost corresponding to each disparity in the disparity neighborhood, and correct the initial disparity in the initial disparity map based on the target disparity to generate a target disparity map.

[0030] Optionally, determining the target disparity corresponding to the target pixel based on the aggregated cost corresponding to each disparity within the disparity neighborhood includes: determining the minimum aggregated cost from the aggregated costs corresponding to each disparity within the disparity neighborhood, and using the disparity corresponding to the minimum aggregated cost as the target disparity corresponding to the target pixel. For example, from the disparity neighborhood... The disparity corresponding to the minimum aggregate cost among all disparities is selected as the target disparity for the target pixel. (That is, optimal disparity). Then based on the target disparity Correct the initial disparity in the initial disparity map to generate the target disparity map.

[0031] Optionally, determining the target disparity corresponding to the target pixel based on the aggregate cost corresponding to each disparity within the disparity neighborhood includes: determining the minimum aggregate cost from the aggregate costs corresponding to each disparity within the disparity neighborhood, and using the disparity corresponding to the minimum aggregate cost as a candidate disparity; determining the left and right neighbor candidate disparities of the candidate disparity; constructing a first candidate point based on the candidate disparity and its corresponding aggregate cost, constructing a second candidate point based on the left neighbor candidate disparity and its corresponding aggregate cost, and constructing a third candidate point based on the right neighbor candidate disparity and its corresponding aggregate cost; performing curve fitting based on the first, second, and third candidate points to generate an aggregate cost-disparity curve, and using the sub-pixel disparity corresponding to the minimum aggregate cost point in the aggregate cost-disparity curve as the target disparity corresponding to the target pixel. The advantage of this configuration is that it allows for sub-pixel correction of disparities matched based on deep learning, further improving the accuracy of binocular vision disparity.

[0032] In this embodiment, the minimum aggregate cost is determined from the aggregate costs corresponding to each disparity within the disparity neighborhood, and the disparity corresponding to the minimum aggregate cost is selected as a candidate disparity. Determine candidate parallax left neighbor candidate parallax and right neighbor candidate parallax Here, the left neighbor candidate disparity can be understood as the disparity to the left of the candidate disparity, and the right neighbor candidate disparity can be understood as the disparity to the right of the candidate disparity. Based on candidate disparities... and the aggregation cost corresponding to candidate disparities Constitute the first candidate point Based on left neighbor candidate disparity and the aggregation cost corresponding to the left neighbor candidate disparity Constitutes the second candidate point and based on right neighbor candidate disparity and the aggregation cost corresponding to the right neighbor candidate disparity Constitutes a third candidate point Based on the first candidate point Second candidate point and the third candidate point Perform curve fitting, such as quadratic parabola fitting, to generate a convergence cost-disparity curve. Determine the minimum convergence cost point in the convergence cost-disparity curve, and use the disparity corresponding to the minimum convergence cost point (i.e., sub-pixel disparity) as the target disparity corresponding to the target pixel. Based on the target disparity, correct the corresponding initial disparity in the initial disparity map to generate a target disparity map.

[0033] The technical solution of this application embodiment, in response to the triggering of a binocular visual disparity determination event, determines an initial disparity map between the left and right images captured by the binocular camera based on a binocular matching disparity network; determines each pair of matching pixel points between the left and right images according to the initial disparity map; for each pair of matching pixel points, traverses each disparity in the disparity neighborhood centered on the initial disparity corresponding to the matching pixel point pair, and calculates the aggregation cost of the target pixel point within a preset-sized pixel window in its target image based on the current disparity; wherein, the target pixel point is a pixel point in the matching pixel point pair belonging to the left or right image; determines the target disparity corresponding to the target pixel point based on the aggregation cost corresponding to each disparity in the disparity neighborhood, and corrects the initial disparity corresponding to the initial disparity map based on the target disparity to generate a target disparity map. This solution corrects the initial disparity map obtained based on deep learning matching based on a binocular matching method, which not only effectively ensures the robustness of deep learning matching disparity but also improves the accuracy of binocular visual disparity.

[0034] Figure 3 This is a schematic diagram of a parallax correction device provided in an embodiment of this application. This device can execute the parallax correction method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the method. Figure 3 As shown, the device includes: The initial disparity map determination module 310 is used to determine the initial disparity map between the left and right images acquired by the binocular camera based on the binocular matching disparity network. The matching pixel pair determination module 320 is used to determine each set of matching pixel pairs between the left image and the right image based on the initial disparity map; The aggregation cost determination module 330 is used to, for each pair of matching pixel points, traverse each disparity in the disparity neighborhood centered on the initial disparity corresponding to the matching pixel point pair, and calculate the aggregation cost of the target pixel point in the target image of a preset size pixel window based on the current disparity; wherein, the target pixel point is the pixel point in the matching pixel point pair that belongs to the left image or the right image; The target disparity determination module 340 is used to determine the target disparity corresponding to the target pixel based on the aggregation cost corresponding to each disparity in the disparity neighborhood, and to correct the initial disparity corresponding to the initial disparity map based on the target disparity to generate a target disparity map.

[0035] Optionally, the aggregation cost determination module includes: A matching cost calculation unit is used to calculate the matching cost between the matching pixel pairs; wherein the matching cost is used to reflect the similarity between the matching pixel pairs; The path cost calculation unit is used to calculate the path cost of each neighboring pixel in the target image to which the target pixel belongs, based on the matching cost and the current disparity, for each neighboring pixel other than the target pixel within a preset-size pixel window centered on the target pixel. The aggregation cost determination unit is used to take the sum of the path costs corresponding to each of the neighboring pixels as the aggregation cost of the target pixel within the preset-size pixel window in the target image.

[0036] Optional, a matching cost calculation unit, used for: Calculate the difference in pixel grayscale values ​​between each pair of pixels in the left and right images, respectively, within a preset-sized pixel window centered on the corresponding pixel in the matching pixel pair. The matching cost between the matching pixel pairs is calculated based on the difference in grayscale values ​​of each pixel; wherein the matching cost is the sum of the absolute values ​​of the differences in grayscale values ​​of each pixel.

[0037] Optional, path cost calculation unit, used for: Determine the left and right adjacent disparities of the current disparity; Based on the current disparity, the left adjacent disparity, and the right adjacent disparity, a matching neighboring pixel corresponding to the neighboring pixel is determined from another image, and a neighboring pixel pair is formed based on the neighboring pixel and the matching neighboring pixel; wherein, the other image is the image other than the target image in the left image and the right image; Calculate the matching cost between the corresponding neighboring pixel pairs based on the current disparity, the left adjacent disparity, and the right adjacent disparity, respectively. The path cost of the neighboring pixel is determined based on the matching cost, the current disparity, the matching cost corresponding to the left neighboring disparity and the right neighboring disparity, respectively.

[0038] Optional, a target parallax determination module is used for: The minimum aggregation cost is determined from the aggregation costs corresponding to each disparity in the disparity neighborhood, and the disparity corresponding to the minimum aggregation cost is taken as the target disparity corresponding to the target pixel.

[0039] Optional, a target parallax determination module is used for: The minimum aggregation cost is determined from the aggregation costs corresponding to each disparity in the disparity neighborhood, and the disparity corresponding to the minimum aggregation cost is taken as a candidate disparity. Determine the left and right neighbor candidate disparities of the candidate disparity; A first candidate point is formed based on the candidate disparity and the corresponding aggregation cost; a second candidate point is formed based on the left neighbor candidate disparity and the corresponding aggregation cost; and a third candidate point is formed based on the right neighbor candidate disparity and the corresponding aggregation cost. Curve fitting is performed based on the first candidate point, the second candidate point, and the third candidate point to generate a converged cost-disparity curve, and the sub-pixel disparity corresponding to the minimum converged cost point in the converged cost-disparity curve is taken as the target disparity corresponding to the target pixel.

[0040] The parallax correction device provided in this application embodiment can execute a parallax correction method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of executing the method.

[0041] Figure 4 A schematic diagram of an electronic device 10, which can be used to implement embodiments of this application, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0042] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0043] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of monitors, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0044] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as parallax correction methods.

[0045] In some embodiments, the parallax correction method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the parallax correction method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the parallax correction method by any other suitable means (e.g., by means of firmware).

[0046] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0047] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable parallax correction device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0048] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0049] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0050] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0051] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0052] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the parallax correction method as provided in any embodiment of this application.

[0053] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or it can be connected to an external computer (e.g., via the Internet using an Internet service provider). It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired information of the technical solution of this application can be achieved, and this is not limited herein.

[0054] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A parallax correction method, characterized in that, The method includes: The initial disparity map between the left and right images acquired by the binocular camera is determined based on a binocular matching disparity network. Based on the initial disparity map, determine each pair of matching pixel points between the left image and the right image; For each pair of matching pixels, traverse each disparity in the disparity neighborhood centered on the initial disparity corresponding to the pair of matching pixels, and calculate the aggregation cost of the target pixel within a preset-sized pixel window in the target image based on the current disparity; wherein, the target pixel is a pixel in the pair of matching pixels that belongs to the left image or the right image; The target disparity corresponding to the target pixel is determined based on the aggregation cost corresponding to each disparity in the disparity neighborhood, and the initial disparity in the initial disparity map is corrected based on the target disparity to generate a target disparity map.

2. The method according to claim 1, characterized in that, Based on the current disparity, the aggregation cost of the target pixel within a preset-sized pixel window in its target image is calculated, including: Calculate the matching cost between the matching pixel pairs; wherein the matching cost is used to reflect the similarity between the matching pixel pairs; In the target image to which the target pixel belongs, for each neighboring pixel other than the target pixel within a preset-size pixel window centered on the target pixel, the path cost of the neighboring pixel is calculated based on the matching cost and the current disparity. The sum of the path costs corresponding to each of the neighboring pixels is used as the aggregate cost of the target pixel within the preset-size pixel window in the target image.

3. The method according to claim 2, characterized in that, Calculating the matching cost between the matching pixel pairs includes: Calculate the difference in pixel grayscale values ​​between each pair of pixels in the left and right images, respectively, within a preset-sized pixel window centered on the corresponding pixel in the matching pixel pair. The matching cost between the matching pixel pairs is calculated based on the difference in grayscale values ​​of each pixel; wherein the matching cost is the sum of the absolute values ​​of the differences in grayscale values ​​of each pixel.

4. The method according to claim 2, characterized in that, Calculating the path cost of the neighboring pixels based on the matching cost and the current disparity includes: Determine the left and right adjacent disparities of the current disparity; Based on the current disparity, the left adjacent disparity, and the right adjacent disparity, a matching neighboring pixel corresponding to the neighboring pixel is determined from another image, and a neighboring pixel pair is formed based on the neighboring pixel and the matching neighboring pixel; wherein, the other image is the image other than the target image in the left image and the right image; Calculate the matching cost between the corresponding neighboring pixel pairs based on the current disparity, the left adjacent disparity, and the right adjacent disparity, respectively. The path cost of the neighboring pixel is determined based on the matching cost, the current disparity, the matching cost corresponding to the left neighboring disparity and the right neighboring disparity, respectively.

5. The method according to claim 1, characterized in that, Determining the target disparity corresponding to the target pixel based on the aggregated cost corresponding to each disparity in the disparity neighborhood includes: The minimum aggregation cost is determined from the aggregation costs corresponding to each disparity in the disparity neighborhood, and the disparity corresponding to the minimum aggregation cost is taken as the target disparity corresponding to the target pixel.

6. The method according to claim 1, characterized in that, Determining the target disparity corresponding to the target pixel based on the aggregated cost corresponding to each disparity in the disparity neighborhood includes: The minimum aggregation cost is determined from the aggregation costs corresponding to each disparity in the disparity neighborhood, and the disparity corresponding to the minimum aggregation cost is taken as a candidate disparity. Determine the left and right neighbor candidate disparities of the candidate disparity; A first candidate point is formed based on the candidate disparity and the corresponding aggregation cost; a second candidate point is formed based on the left neighbor candidate disparity and the corresponding aggregation cost; and a third candidate point is formed based on the right neighbor candidate disparity and the corresponding aggregation cost. Curve fitting is performed based on the first candidate point, the second candidate point, and the third candidate point to generate a converged cost-disparity curve, and the sub-pixel disparity corresponding to the minimum converged cost point in the converged cost-disparity curve is taken as the target disparity corresponding to the target pixel.

7. A parallax correction device, characterized in that, include: The initial disparity map determination module is used to determine the initial disparity map between the left and right images acquired by the binocular camera based on the binocular matching disparity network. The matching pixel pair determination module is used to determine each set of matching pixel pairs between the left image and the right image based on the initial disparity map; The aggregation cost determination module is used to, for each pair of matching pixel points, traverse each disparity in the disparity neighborhood centered on the initial disparity corresponding to the matching pixel point pair, and calculate the aggregation cost of the target pixel point in the target image of a preset size pixel window based on the current disparity; wherein, the target pixel point is the pixel point in the matching pixel point pair that belongs to the left image or the right image; The target disparity determination module is used to determine the target disparity corresponding to the target pixel based on the aggregation cost corresponding to each disparity in the disparity neighborhood, and to correct the initial disparity corresponding to the initial disparity map based on the target disparity to generate a target disparity map.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the parallax correction method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the parallax correction method according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the parallax correction method according to any one of claims 1-6.

Images