Parallax determination method and electronic device

Abstract

The application discloses a parallax determination method and an electronic device. A first image and a second image are obtained according to a relative imaging position relationship determination. A target region of the first image is determined, and a region outside the target region is a non-target region. First cost calculation is performed on each first pixel point in the non-target region of the first image and a corresponding pixel point of the second image to obtain a first cost data set corresponding to the first pixel point, and the first cost calculation at least includes first cost aggregation processing. Second cost calculation is performed on each second pixel point in the target region of the first image and a corresponding pixel point of the second image to obtain a second cost data set corresponding to the second pixel point, and the second cost calculation includes second cost aggregation processing, wherein the number of aggregation directions of the second cost aggregation processing is less than the number of aggregation directions of the first cost aggregation processing. A parallax map is obtained based on the first cost data set and the second cost data set.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to a parallax determination method and electronic device. Background Technology

[0002] Stereo matching is a key part of stereo vision research. Its goal is to match corresponding pixels in two or more viewpoints and calculate disparity.

[0003] SGM (Semi-Global Matching) is a semi-global matching algorithm for calculating disparity in binocular vision. It mainly includes matching cost calculation, cost aggregation, disparity calculation, and disparity optimization. However, in the cost aggregation stage, cost fusion in 4 or 8 directions is typically required for each pixel, which reduces the efficiency of disparity calculation. Summary of the Invention

[0004] In view of this, this application provides a parallax determination method and an electronic device, the specific solution of which is as follows:

[0005] A method for determining parallax includes:

[0006] Acquire a first image and a second image whose relative imaging positional relationship is determined;

[0007] Determine the target region of the first image, and define the region outside the target region as a non-target region;

[0008] A first cost calculation is performed on each first pixel in the non-target region of the first image and the corresponding pixel in the second image to obtain a first cost dataset corresponding to the first pixel. The first cost calculation includes at least a first cost aggregation process.

[0009] A second cost calculation is performed on each second pixel in the target region of the first image and the corresponding pixel in the second image to obtain a second cost dataset corresponding to the second pixel. The second cost calculation includes a second cost aggregation process, wherein the number of aggregation directions in the second cost aggregation process is less than the number of aggregation directions in the first cost aggregation process.

[0010] A disparity map is obtained based on the first cost dataset and the second cost dataset.

[0011] Furthermore, determining the target region of the first image includes:

[0012] In response to a user-specified operation, determine a first region in the first image, or, based on the image data of the first image, identify a first region in the first image where the target imaging subject is located, wherein the target imaging subject is located within the first region;

[0013] The target region in the first image is determined based on the first region, and the target region includes at least the target imaging subject within the first region.

[0014] Furthermore, determining the target region of the first image includes:

[0015] Obtain the first region of the first image;

[0016] Image extraction is performed on the first region to determine the target imaging subject in the first region, and the target imaging subject is determined as the target region of the first image.

[0017] Furthermore, identifying the first region where the target imaging subject is located in the first image based on the image data of the first image includes:

[0018] A segmentation model based on a deep neural network mechanism is used to segment the image data of the first image to obtain the first region where the target imaging subject is located in the first image.

[0019] Furthermore,

[0020] Each first pixel corresponds to a first cost dataset;

[0021] All second-pixel points correspond to the same second cost dataset.

[0022] Furthermore, the first cost calculation includes:

[0023] Each first pixel is allocated a separate independent cost storage space to store the first cost dataset corresponding to each first pixel. Each first pixel corresponds to a separate independent cost storage space.

[0024] Furthermore, the second cost calculation includes:

[0025] Allocate a shared cost storage space for all second pixels within the target area to store the second cost dataset, and all second pixels within the target area share the second cost dataset.

[0026] Furthermore, the sharing of the second cost dataset includes:

[0027] Obtain the third cost dataset corresponding to each second pixel, wherein the third cost dataset includes multiple second-generation values;

[0028] The second-generation values ​​at corresponding positions in each third cost dataset are shared to obtain shared second-generation values. The shared second-generation values ​​at multiple positions constitute the second cost dataset.

[0029] Furthermore, the sharing process for the second-generation value at each corresponding position includes:

[0030] The second-generation values ​​at each corresponding position are smoothed.

[0031] An electronic device, comprising:

[0032] An image acquisition device for obtaining a first image and a second image whose relative imaging positional relationship is determined;

[0033] The processor is configured to: determine a target region of the first image, wherein regions outside the target region are non-target regions; perform a first cost calculation based on each first pixel in the non-target region of the first image and a corresponding pixel in the second image to obtain a first cost dataset corresponding to the first pixel, wherein the first cost calculation includes at least a first cost aggregation process; perform a second cost calculation based on each second pixel in the target region of the first image and a corresponding pixel in the second image to obtain a second cost dataset corresponding to the second pixel, wherein the second cost calculation includes a second cost aggregation process, wherein the number of aggregation directions in the second cost aggregation process is less than the number of aggregation directions in the first cost aggregation process; and obtain a disparity map based on the first cost dataset and the second cost dataset.

[0034] As can be seen from the above technical solutions, the disparity determination method and electronic device disclosed in this application acquire a first image and a second image with a relative imaging positional relationship; determine the target region of the first image, and the region outside the target region is the non-target region; perform a first cost calculation based on each first pixel in the non-target region of the first image and the corresponding pixel in the second image to obtain a first cost dataset corresponding to the first pixel, wherein the first cost calculation includes at least a first cost aggregation process; perform a second cost calculation based on each second pixel in the target region of the first image and the corresponding pixel in the second image to obtain a second cost dataset corresponding to the second pixel, wherein the second cost calculation includes a second cost aggregation process, wherein the number of aggregation directions in the second cost aggregation process is less than the number of aggregation directions in the first cost aggregation process; and obtain a disparity map based on the first cost dataset and the second cost dataset. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1This is a flowchart of a disparity determination method disclosed in an embodiment of this application;

[0037] Figure 2 This is a flowchart of a disparity determination method disclosed in an embodiment of this application;

[0038] Figure 3 This is a schematic diagram of an image obtained by model segmentation according to an embodiment of this application;

[0039] Figure 4 This is a flowchart of a disparity determination method disclosed in an embodiment of this application;

[0040] Figure 5 This is a schematic diagram illustrating the effect of a parallax result that does not distinguish between target and non-target regions.

[0041] Figure 6 This is a schematic diagram illustrating the effect of disparity in distinguishing between target and non-target regions as disclosed in an embodiment of this application.

[0042] Figure 7 This is a flowchart of a disparity determination method disclosed in an embodiment of this application;

[0043] Figure 8 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. Detailed Implementation

[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] This application discloses a method for determining disparity, the flowchart of which is shown below. Figure 1 As shown, it includes:

[0046] Step S11: Obtain the first image and the second image whose relative imaging position relationship is determined;

[0047] Step S12: Determine the target region of the first image; the region outside the target region is the non-target region.

[0048] Step S13: Perform a first cost calculation on each first pixel in the non-target region of the first image and the corresponding pixel in the second image to obtain the first cost dataset corresponding to the first pixel. The first cost calculation includes at least a first cost aggregation process.

[0049] Step S14: Perform a second cost calculation based on each second pixel in the target region of the first image and the corresponding pixel in the second image to obtain the second cost dataset corresponding to the second pixel. The second cost calculation includes a second cost aggregation process, and the number of aggregation directions in the second cost aggregation process is less than the number of aggregation directions in the first cost aggregation process.

[0050] Step S15: Obtain the disparity map based on the first cost dataset and the second cost dataset.

[0051] In some embodiments, the first image and the second image can be a binocular image set acquired by a binocular camera module. The principle of binocular vision measurement is based on the principle that there is parallax when human eyes see objects. In computer vision, by obtaining two images at different positions on the same baseline, the shape and distance of objects can be estimated just like human eyes. That is, by using the different positions of the target object in the two images, the disparity map of the images can be determined.

[0052] Currently, the commonly used binocular vision measurement calculates the disparity information of each pixel in the image from the images acquired by the left and right cameras, and then obtains the three-dimensional information of the object in the actual space. This requires cost fusion of each pixel in the image in 4 or 8 directions, which leads to a decrease in the efficiency of disparity calculation.

[0053] Of course, the first and second images are not limited to a set of binocular images acquired by a binocular camera module. They can also be two separate images for the same field of view, and the corresponding positional relationship of pixels in the two images can be determined by calibration and alignment operations.

[0054] In this scheme, target regions and non-target regions are distinguished. Cost fusion is performed only for each pixel in the non-target region in 4 or 8 directions, while for pixels in the target region, the number of fusion directions is less than 4 or 8 directions when performing cost fusion, thereby improving the efficiency of disparity calculation.

[0055] Specifically, when acquiring multiple images, it is first determined whether the acquired multiple images include two images with a defined relative imaging position relationship. If the first image and the second image are determined to be two images with a defined relative imaging position relationship from the acquired multiple images, then the first image and the second image are determined to be an image group, and disparity calculation needs to be performed on the first image and the second image.

[0056] The image set requiring disparity calculation is a group of images that have undergone epipolar correction. These images are all the same size, e.g., width W and height H. The two epipolar-corrected images share the same reference plane and exhibit disparity only between pixels in the row direction.

[0057] After determining the image group that needs to be disparity calculated, the target region is determined in the first image of the image group. All regions in the first image except the target region are non-target regions. Correspondingly, there are regions in the second image that correspond to the target region and non-target region in the first image.

[0058] The target region can be determined based on the image's focal point, the image's focal range, user specification, or image recognition.

[0059] Cost calculations are performed on pixels in both the target and non-target regions of the image.

[0060] Pixels in the non-target region of the first image are identified as first pixels, and pixels in the target region of the first image are identified as second pixels.

[0061] For pixels in non-target regions, a first cost calculation is performed on each first pixel in the non-target region of the first image and the corresponding pixel in the second image to obtain a first cost dataset corresponding to the first pixel. The first cost calculation includes at least a first cost aggregation process.

[0062] In the first cost calculation, the aggregation direction of the first cost aggregation process can be 4 or 8 directions to ensure the disparity accuracy of non-target areas.

[0063] Since the purpose of cost aggregation is to enable the aggregated cost to more accurately reflect the correlation between pixels, avoid sensitivity to noise, and enhance the robustness of disparity results, and since the target region is selected and usually has the characteristic of continuous disparity proximity, the aggregation direction of cost aggregation can be reduced in the target region, thereby improving the efficiency of disparity calculation while ensuring disparity accuracy.

[0064] For each pixel in the target region, a second cost calculation is performed on each second pixel in the target region of the first image and the corresponding pixel in the second image to obtain the second cost dataset corresponding to the second pixel. The second cost calculation includes at least a second cost aggregation process.

[0065] In this process, the number of aggregation directions in the second cost aggregation process is less than the number of aggregation directions in the first cost aggregation process. Specifically, if the first cost aggregation process has 4 aggregation directions, then the second cost aggregation process can have 2 aggregation directions, 1 aggregation direction, or 0 aggregation directions, i.e., no aggregation is performed. If the first cost aggregation process has 8 aggregation directions, then the second cost aggregation process can have 4 aggregation directions, 2 aggregation directions, 1 aggregation direction, or 0 aggregation directions, i.e., no aggregation is performed.

[0066] As long as the aggregation direction of the second cost aggregation process is less than that of the first cost aggregation process, the calculation time can be shortened when performing cost aggregation calculation in the target region, thereby improving the efficiency of disparity calculation, without affecting the accuracy of disparity calculation.

[0067] After the cost calculation is completed, a disparity map is obtained based on the first cost dataset and the second cost dataset. For example, disparity optimization is performed to remove erroneous disparities, and appropriate smoothing and sub-pixel precision optimization are performed to improve the quality of the disparity map.

[0068] This embodiment discloses a disparity determination method, which involves acquiring a first image and a second image with a determined relative imaging position relationship; determining a target region in the first image, and defining regions outside the target region as non-target regions; performing a first cost calculation on each first pixel in the non-target region of the first image and its corresponding pixel in the second image to obtain a first cost dataset corresponding to the first pixel, wherein the first cost calculation includes at least a first cost aggregation process; performing a second cost calculation on each second pixel in the target region of the first image and its corresponding pixel in the second image to obtain a second cost dataset corresponding to the second pixel, wherein the second cost calculation includes a second cost aggregation process, wherein the number of aggregation directions in the second cost aggregation process is less than the number of aggregation directions in the first cost aggregation process; and obtaining a disparity map based on the first cost dataset and the second cost dataset. This scheme divides the images in the image group for which the disparity map needs to be determined into target and non-target regions, performs cost aggregation processing on each pixel in the non-target region, and performs cost aggregation processing on the target region with fewer aggregation directions than on the non-target region, thereby reducing the time required for cost aggregation processing and improving the efficiency of disparity calculation.

[0069] This embodiment discloses a method for determining disparity, the flowchart of which is shown below. Figure 2 As shown, it includes:

[0070] Step S21: Obtain the first image and the second image whose relative imaging position relationship is determined;

[0071] Step S22: Obtain the first region of the first image;

[0072] Step S23: Extract the image from the first region, determine the target imaging subject in the first region, and define the target imaging subject as the target region of the first image;

[0073] Step S24: Perform a first cost calculation on each first pixel in the non-target region of the first image and the corresponding pixel in the second image to obtain the first cost dataset corresponding to the first pixel. The first cost calculation includes at least a first cost aggregation process.

[0074] Step S25: Perform a second cost calculation based on each second pixel in the target region of the first image and the corresponding pixel in the second image to obtain the second cost dataset corresponding to the second pixel. The second cost calculation includes a second cost aggregation process. The number of aggregation directions in the second cost aggregation process is less than the number of aggregation directions in the first cost aggregation process.

[0075] Step S26: Obtain the disparity map based on the first cost dataset and the second cost dataset.

[0076] When determining the target region of the first image, the first region in the first image is first determined, and then the target region is determined from the first region.

[0077] Determining the first region can be achieved by responding to a user's specified operation and determining the first region in the first image.

[0078] This means that the user selects an area, which is then designated as the first area. The user can specify an area on the first image through touch or gesture, or by pressing a button.

[0079] Alternatively, determining the first region can also be: based on the image data of the first image, identifying the first region where the target imaging subject in the first image is located, and the target imaging subject is located within the first region.

[0080] That is, image recognition is performed on the first image to identify the target imaging subject in the first image and determine the area where it is located as the first region.

[0081] For example: take a picture of a person to obtain a first image and a second image, identify the person from the first image, determine the area where the person is located, and define this area as the first area.

[0082] Among them, image recognition of the first image can identify the target imaging subject based on the foreground and background. For example, when taking a picture of a person, if the person in the image is in the foreground and all other objects in the image are in the background, then the target imaging subject can be identified.

[0083] Image recognition can also be performed based on object type to determine the target imaging subject. For example, when taking a picture of a bird, if the type of the target imaging subject can be determined during the photo taking, then the area where the target imaging subject is located can be determined by recognizing it according to the pre-determined type during image recognition.

[0084] Image recognition can also be performed based on color distribution or brightness distribution to identify the target imaging subject.

[0085] Specifically, a segmentation model based on a deep neural network mechanism can be used to segment the image data of the first image to obtain the first region where the target imaging subject is located in the first image.

[0086] The first image is segmented based on a segmentation model to obtain the first region where the target imaging subject is located, such as... Figure 3 The image shown is the result obtained through the segmentation model. The white part is the first region obtained based on the segmentation model, which is the region where the target imaging subject is located.

[0087] Segmenting the first image based on the segmentation model does not increase the computational burden on the system. Furthermore, the use of the segmentation model significantly improves the disparity accuracy and efficiency of binocular depth estimation.

[0088] After determining the first region where the target imaging subject is located, it is necessary to refine the first region in order to determine the target region from the first region.

[0089] The first region can be further refined by performing image segmentation on the first region to extract the target imaging subject from the first region. The extracted target imaging subject is determined as the target region, while other regions in the first region other than the target region and other regions outside the first region are determined as non-target regions.

[0090] Furthermore, after identifying the target region and non-target region from the first image, local structural features can be detected in the first and second images, such as grayscale processing or Census transformation, to obtain images with grayscale values ​​of 0 and 1. Then, subsequent disparity calculation processing can be performed based on the obtained images.

[0091] The Census transform is a type of nonparametric image transform that can effectively detect local structural features in an image, such as edges and corners.

[0092] The Census transform typically works as follows: A rectangular window is defined within the image region. This window is used to traverse the entire image, selecting the center pixel as the reference pixel. The grayscale value of each pixel within the window is compared to the reference pixel's grayscale value. Pixels with grayscale values ​​less than or equal to the reference pixel's value are marked as 0, and those greater are marked as 1. The pixels are then concatenated bit-by-bit to obtain the transformed result, which is a binary code stream composed of 0s and 1s. In essence, the Census transform encodes the grayscale values ​​of image pixels into a binary code stream to obtain the relative magnitudes of neighboring pixels' grayscale values ​​to the center pixel's grayscale value.

[0093] This embodiment discloses a disparity determination method, which involves acquiring a first image and a second image with a determined relative imaging position relationship; obtaining a first region of the first image; segmenting the first region to determine the target imaging subject within the first region; defining the target imaging subject as the target region of the first image; performing a first cost calculation on each first pixel in the non-target region of the first image and its corresponding pixel in the second image to obtain a first cost dataset corresponding to the first pixel; the first cost calculation includes at least a first cost aggregation process; performing a second cost calculation on each second pixel in the target region of the first image and its corresponding pixel in the second image to obtain a second cost dataset corresponding to the second pixel; the second cost calculation includes a second cost aggregation process, wherein the number of aggregation directions in the second cost aggregation process is less than the number of aggregation directions in the first cost aggregation process; and obtaining a disparity map based on the first cost dataset and the second cost dataset. In this scheme, the target imaging subject in the first region is determined through image segmentation, thereby defining it as the target region. This reduces the number of aggregation directions when performing cost aggregation processing on the pixels in the target region, thus reducing the time required for cost aggregation processing and improving the efficiency of disparity calculation.

[0094] This embodiment discloses a method for determining disparity, the flowchart of which is shown below. Figure 4 As shown, it includes:

[0095] Step S41: Obtain the first image and the second image whose relative imaging position relationship is determined;

[0096] Step S42: Determine the target region of the first image; the region outside the target region is the non-target region.

[0097] Step S43: Perform a first cost calculation on each first pixel in the non-target region of the first image and the corresponding pixel in the second image to obtain a first cost dataset corresponding to the first pixel. The first cost calculation includes at least a first cost aggregation process, and each first pixel corresponds to a first cost dataset.

[0098] Step S44: Perform a second cost calculation based on each second pixel in the target region of the first image and the corresponding pixel in the second image to obtain the second cost dataset corresponding to the second pixel. The second cost calculation includes a second cost aggregation process. The number of aggregation directions in the second cost aggregation process is less than the number of aggregation directions in the first cost aggregation process. All second pixels correspond to the same second cost dataset.

[0099] Step S45: Obtain the disparity map based on the first cost dataset and the second cost dataset.

[0100] Each first pixel corresponds to a first cost dataset. That is, when calculating the cost for non-target regions, each pixel is calculated separately, so that a corresponding first cost dataset is obtained for each first pixel.

[0101] The number of first pixels in the non-target region determines the number of first cost datasets. Furthermore, when calculating the cost of each first pixel in the non-target region, a larger number of aggregation directions are used, such as 4-direction cost aggregation or 8-direction cost aggregation, to ensure that the disparity between each first pixel and its corresponding pixel in the second image can be accurately determined in the non-target region.

[0102] All second pixels correspond to the same second cost dataset. That is, when calculating the cost of the target region, whether only one pixel is calculated or all pixels in the target region are calculated, the dataset corresponding to each second pixel in the target region is the same second cost dataset.

[0103] That is, all second pixels in the target region share the same second cost dataset; and when calculating the second cost dataset, a smaller number of aggregation directions are used for cost aggregation, which is at least less than the number of aggregation directions used in the first cost calculation. This reduces the amount of data processing during the second cost calculation, reduces the time required for cost aggregation processing, and improves the efficiency of disparity calculation.

[0104] Specifically, in disparity calculation, if the target region and non-target region are not distinguished, and each pixel in the image group corresponds to a separate cost dataset, and the number of aggregation directions for each pixel during cost calculation is also the same, then the final disparity result will look like the image below. Figure 5 As shown;

[0105] If, during disparity calculation, the scheme disclosed in this embodiment is adopted, distinguishing between target and non-target regions, with all pixels in the target region corresponding to the same second cost dataset, while each pixel in the non-target region corresponds to a separate first cost dataset, and the number of aggregation directions for cost calculation of pixels in the target region is less than the number of aggregation directions for cost calculation of pixels in the non-target region, then the resulting disparity result is illustrated as follows. Figure 6 As stated above.

[0106] Figure 6 Compared to Figure 5 The disparity results significantly optimize the disparity accuracy and smoothness in salient regions, and the implementation method also improves the efficiency of disparity calculation.

[0107] Furthermore, the first cost calculation can be as follows:

[0108] Each first pixel is allocated a separate independent cost storage space to store the first cost dataset corresponding to each first pixel. Each first pixel corresponds to a separate independent cost storage space.

[0109] The cost storage space stores the cost dataset, which is actually the difference between each pixel in the first image and the corresponding pixel in the second image. Only subtle differences between the two images can reflect the change in depth.

[0110] Each first pixel in the non-target region corresponds to an independent first cost dataset. When storing these datasets, an independent cost storage space is allocated for each first pixel to store the first cost dataset of the corresponding pixel.

[0111] For example, if there are n first pixels in the non-target region, then when calculating the cost, n first cost datasets will be obtained. To store the n first cost datasets, n independent cost storage spaces are needed. Each independent cost storage space is used to store one first cost dataset, so as to ensure that each first cost dataset is stored in a different independent cost storage space.

[0112] Specifically, for a specific pixel in the first image, its corresponding point in the second image can be determined. Then, multiple pixels within a certain pixel range corresponding to the corresponding point are determined, and the differences between these pixels and the specific pixel in the first image are determined and stored. The stored data is the cost dataset corresponding to the specific pixel, and the location where the cost dataset is stored is the cost storage space.

[0113] For example: if it is determined that there are n first pixels in the non-target region, calculate the difference between each first pixel and the 128 pixels in the corresponding search range on the second image, that is, calculate the difference between each first pixel and the 128 pixels. This difference is actually the cost value. Then, for each first pixel, 128 cost values ​​can be obtained. These 128 cost values ​​form the first cost dataset and are stored in the independent cost storage space corresponding to the first pixel. The storage of the first cost datasets of different first pixels does not interfere with each other. n first pixels correspond to n first cost datasets, that is, n independent cost storage spaces.

[0114] The process involves cost aggregation, yielding 128 cost values. The minimum cost value is then identified and aggregated again to obtain the aggregated value. This process is repeated until the minimum cost value stabilizes. This cycle typically requires 3-4 cost aggregations. Because of the iterative data updates, cost storage space needs to be allocated, and each first pixel requires individual iterative updates.

[0115] This embodiment discloses a disparity determination method, which involves acquiring a first image and a second image with a determined relative imaging position relationship; determining a target region in the first image, and defining regions outside the target region as non-target regions; performing a first cost calculation on each first pixel in the non-target region of the first image and its corresponding pixel in the second image to obtain a first cost dataset corresponding to the first pixel, wherein the first cost calculation includes at least a first cost aggregation process, and each first pixel corresponds to a first cost dataset; performing a second cost calculation on each second pixel in the target region of the first image and its corresponding pixel in the second image to obtain a second cost dataset corresponding to the second pixel, wherein the second cost calculation includes a second cost aggregation process, wherein the number of aggregation directions in the second cost aggregation process is less than the number of aggregation directions in the first cost aggregation process, and all second pixels correspond to the same second cost dataset; and obtaining a disparity map based on the first cost dataset and the second cost dataset. In this scheme, each first pixel corresponds to a first cost dataset, and all second pixels correspond to the same second cost dataset. That is, cost aggregation is performed on each pixel in the non-target region and different datasets are obtained, while the same dataset is used after cost aggregation on the pixels in the target region. This reduces the space occupied by the dataset of pixels in the target region, reduces the time required for cost aggregation, and improves the efficiency of disparity calculation.

[0116] This embodiment discloses a method for determining disparity, the flowchart of which is shown below. Figure 7 As shown, it includes:

[0117] Step S71: Obtain the first image and the second image whose relative imaging position relationship is determined;

[0118] Step S72: Determine the target region of the first image; the region outside the target region is the non-target region.

[0119] Step S73: Perform a first cost calculation on each first pixel in the non-target region of the first image and the corresponding pixel in the second image to obtain the first cost dataset corresponding to the first pixel. The first cost calculation includes at least a first cost aggregation process.

[0120] Step S74: Perform a second cost calculation based on each second pixel in the target region of the first image and the corresponding pixel in the second image. The second cost calculation includes allocating a shared cost storage space for the second pixels in the target region to store the second cost dataset. All second pixels in the target region share the second cost dataset.

[0121] Step S75: Obtain the second cost dataset corresponding to the second pixel;

[0122] Step S76: Obtain the disparity map based on the first cost dataset and the second cost dataset.

[0123] For the calculation of the first cost of each first pixel in the non-target region of the image, an independent cost storage space is allocated for each first pixel to store the first cost dataset corresponding to each first pixel. This ensures that each first pixel in the non-target region has a first cost dataset, and each first cost dataset has a corresponding independent cost storage space for independent storage, so as to ensure the accuracy of the cost calculation of each first pixel in the non-target region.

[0124] As for the target region, since it has the characteristic of continuous proximity of disparity, the target region can be treated as a local whole for cost calculation and optimization. That is, all second pixels correspond to the same second cost dataset, and there is no need to perform cost aggregation during the calculation process.

[0125] When all second pixels correspond to the same second cost dataset, a shared cost storage space can be directly allocated for all second pixels in the target region to store the second cost dataset.

[0126] Since the second cost datasets of each second pixel in the target region are the same, there is no need to allocate a corresponding cost storage space for each pixel in the target region separately. Instead, a shared cost storage space is allocated for all second pixels in the target region to store the second cost dataset corresponding to each second pixel, thereby saving memory space. Furthermore, only one cost dataset needs to be stored in this shared cost storage space, so there is no need to store a large number of cost datasets.

[0127] For example, if there are m second pixels in the target region, and each second pixel corresponds to a second cost dataset, and all the second cost datasets corresponding to the second pixels in the target region are the same, then all the second pixels share the same second cost dataset. When storing the second cost dataset, only one shared cost storage space is needed to store the second cost datasets corresponding to all the second pixels. Since the second cost datasets corresponding to the m second pixels are all the same, there is no need to set up m cost storage spaces to store them. Only one is needed to store the dataset.

[0128] Specifically, sharing the second cost dataset can include:

[0129] Obtain a third cost dataset containing multiple second cost values ​​corresponding to each second pixel. Perform sharing processing on the second-generation values ​​at corresponding positions in each third cost dataset to obtain shared second-generation values. The shared second-generation values ​​at multiple positions constitute the second cost dataset.

[0130] In other words, the second cost dataset is determined by multiple second-generation values ​​for each second pixel. During the calculation process, the second cost value is first calculated for each pixel in the target region, thereby obtaining the third cost dataset corresponding to each second pixel. Each third cost dataset includes multiple second-generation values ​​corresponding to its corresponding pixel. Then, the second-generation values ​​at corresponding positions in all third cost datasets are shared to obtain shared second-generation values. The second cost dataset is composed of the shared second-generation values ​​at each position.

[0131] For example, if there are m second pixels in the target region, calculate the difference between each second pixel and the corresponding pixel in the second image within a range of 128 pixels. That is, calculate the difference between each second pixel and the 128 pixels. This difference is the second-generation value. Then, 128 second-generation values ​​can be obtained for each second pixel. The 128 second-generation values ​​corresponding to each second pixel form the third cost dataset for that second pixel. Then, the m second-generation values ​​corresponding to the positions of the m second pixels are shared to obtain 128 shared second-generation values. These 128 shared second-generation values ​​can form the second cost dataset, which serves as the shared cost dataset for all second pixels and is stored in the shared cost storage space.

[0132] The second cost dataset is obtained based on the m second-generation values ​​corresponding to the positions of the m second pixels. That is, each second pixel corresponds to a pixel range of 128 pixels, i.e., each second pixel corresponds to position 1, position 2, position 3, ..., position 128. Then, the m second pixels correspond to m third cost datasets. Based on the cost values ​​of the m positions 1 in the m third cost datasets, a shared second-generation value for position 1 is determined. Based on the cost values ​​of the m positions 2 in the m third cost datasets, a shared second-generation value for position 2 is determined. ..., until a shared second-generation value for position 128 is determined based on the cost values ​​of the m positions 128 in the m third cost datasets. The m shared second-generation values ​​constitute the second cost dataset.

[0133] Alternatively, the second cost dataset can be obtained in other ways. For example, the cost can be calculated for only one second pixel in the target region to obtain the corresponding cost dataset. This cost dataset can then be directly identified as the second cost dataset and stored in the shared cost storage space as the cost dataset for each second pixel in the target region. Another method is to select k second pixels in the target region for cost calculation to obtain k corresponding cost datasets. Then, the cost values ​​in the k cost datasets are shared to obtain a shared dataset. This shared dataset can then be identified as the second cost dataset and stored in the shared cost storage space as the cost dataset for each second pixel in the target region.

[0134] Of course, other methods can also be used to obtain the second cost dataset, which will not be elaborated here.

[0135] Specifically, the second-generation value of each corresponding position can be shared by performing smoothing processing on the second-generation value of each corresponding position, such as mean processing, weighted average processing, etc.

[0136] The mean value processing involves directly summing the values ​​of each second generation corresponding to each position, then taking the average value, and using this average value as the shared second generation value to complete the sharing process.

[0137] Weighted averaging can be achieved by assigning different weights to pixels at different locations, summing the weights based on these weights, and then averaging the sum. This average value is used as the shared second-generation value to complete the sharing process. For example, pixels in the center of the target region have a higher weight than pixels in the edge regions. Alternatively, weights can be assigned based on importance or other rules to achieve the same sharing effect. Furthermore, the second cost calculation can be performed by dividing the target region into several sub-regions and treating each sub-region as a whole for cost calculation and optimization. This means all pixels in a sub-region correspond to the same cost dataset, while pixels in different sub-regions can have different cost datasets. This reduces computational complexity and improves the efficiency of disparity determination compared to cost calculations in non-target regions.

[0138] This embodiment discloses a disparity determination method, which involves acquiring a first image and a second image with a determined relative imaging position relationship; determining a target region in the first image, and defining regions outside the target region as non-target regions; performing a first cost calculation on each first pixel in the non-target region of the first image and its corresponding pixel in the second image to obtain a first cost dataset corresponding to the first pixel, wherein the first cost calculation includes at least a first cost aggregation process; performing a second cost calculation on each second pixel in the target region of the first image and its corresponding pixel in the second image to obtain a second cost dataset corresponding to the second pixel, wherein the second cost calculation includes allocating shared cost storage space for the second pixels in the target region to store the second cost dataset, wherein all second pixels in the target region share the second cost dataset, and the second cost calculation includes a second cost aggregation process, wherein the number of aggregation directions in the second cost aggregation process is less than the number of aggregation directions in the first cost aggregation process; and obtaining a disparity map based on the first cost dataset and the second cost dataset. This scheme divides the image groups in which the disparity map needs to be determined into target regions and non-target regions. Cost aggregation is performed on each pixel in the non-target region, while the number of aggregation directions in the target region is less than that in the non-target region. In addition, all second pixels in the target region share the second cost dataset through the same shared cost storage space, which reduces the memory usage of pixels in the target region when calculating costs. At the same time, it can also reduce the time required for cost aggregation and improve the efficiency of disparity calculation.

[0139] This embodiment discloses an electronic device, the structural schematic diagram of which is shown below. Figure 8 As shown, it includes:

[0140] Image acquisition device 81 and processor 82.

[0141] The image acquisition device 81 is used to obtain a first image and a second image whose relative imaging position relationship is determined.

[0142] The processor 82 is used to determine the target region of the first image, and the region outside the target region is the non-target region; based on each first pixel in the non-target region of the first image and the corresponding pixel in the second image, a first cost calculation is performed to obtain a first cost dataset corresponding to the first pixel, and the first cost calculation includes at least a first cost aggregation process; based on each second pixel in the target region of the first image and the corresponding pixel in the second image, a second cost calculation is performed to obtain a second cost dataset corresponding to the second pixel, and the second cost calculation includes a second cost aggregation process, wherein the number of aggregation directions in the second cost aggregation process is less than the number of aggregation directions in the first cost aggregation process; a disparity map is obtained based on the first cost dataset and the second cost dataset.

[0143] Furthermore, the processor is used for:

[0144] In response to a user-specified operation, determine a first region in a first image, or, based on image data of the first image, identify a first region in the first image where a target imaging subject is located, wherein the target imaging subject is located within the first region; determine a target region in the first image based on the first region, wherein the target region includes at least the target imaging subject within the first region.

[0145] Furthermore, the processor is used for:

[0146] Obtain the first region of the first image;

[0147] Image extraction is performed on the first region to determine the target imaging subject in the first region, and the target imaging subject is determined as the target region of the first image.

[0148] Furthermore, the processor is used for:

[0149] A segmentation model based on a deep neural network mechanism segments the image data of a first image to obtain the first region where the target imaging subject is located in the first image.

[0150] Furthermore,

[0151] Each first pixel corresponds to a first cost dataset;

[0152] All second-pixel points correspond to the same second cost dataset.

[0153] Furthermore, the processor is used for:

[0154] Each first pixel is allocated a separate independent cost storage space to store the first cost dataset corresponding to each first pixel. Each first pixel corresponds to a separate independent cost storage space.

[0155] Furthermore, the processor is used for:

[0156] Allocate shared cost storage space for the second pixel within the target area to store the second cost dataset. All second pixels within the target area share the second cost dataset.

[0157] Furthermore, the processor is used for:

[0158] Obtain the third cost dataset, which includes multiple second cost values, corresponding to each second pixel.

[0159] The second-generation values ​​at corresponding positions in each third cost dataset are shared to obtain shared second-generation values. The shared second-generation values ​​at multiple positions constitute the second cost dataset.

[0160] Furthermore, the processor is used for:

[0161] The second-generation values ​​at each corresponding position are processed by mean, weighted average, or smoothing.

[0162] The electronic device disclosed in this embodiment is implemented based on the parallax determination method disclosed in the above embodiments, and will not be described again here.

[0163] This embodiment discloses an electronic device that acquires a first image and a second image with a determined relative imaging position relationship; determines a target region in the first image, and defines regions outside the target region as non-target regions; performs a first cost calculation on each first pixel in the non-target region of the first image and its corresponding pixel in the second image to obtain a first cost dataset corresponding to the first pixel, wherein the first cost calculation includes at least a first cost aggregation process; performs a second cost calculation on each second pixel in the target region of the first image and its corresponding pixel in the second image to obtain a second cost dataset corresponding to the second pixel, wherein the second cost calculation includes a second cost aggregation process, wherein the number of aggregation directions in the second cost aggregation process is less than the number of aggregation directions in the first cost aggregation process; and obtains a disparity map based on the first cost dataset and the second cost dataset. This scheme divides the images in the image group for which the disparity map needs to be determined into target regions and non-target regions, performs cost aggregation processing on each pixel in the non-target regions, and performs cost aggregation processing on the target regions with fewer aggregation directions than on the non-target regions, thereby reducing the time required for cost aggregation processing and improving the efficiency of disparity calculation.

[0164] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0165] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0166] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0167] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for determining parallax, comprising: Acquire a first image and a second image whose relative imaging positional relationship is determined; The target region of the first image is determined, and the region outside the target region is a non-target region. The target region includes at least the target imaging subject. A first cost calculation is performed on each first pixel in the non-target region of the first image and the corresponding pixel in the second image to obtain a first cost dataset corresponding to the first pixel. The first cost calculation includes at least a first cost aggregation process. A second cost calculation is performed on each second pixel in the target region of the first image and the corresponding pixel in the second image to obtain a second cost dataset corresponding to the second pixel. The second cost calculation includes a second cost aggregation process, wherein the number of aggregation directions in the second cost aggregation process is less than the number of aggregation directions in the first cost aggregation process. A disparity map is obtained based on the first cost dataset and the second cost dataset.

2. The method according to claim 1, wherein, Determining the target region of the first image includes: In response to a user-specified operation, determine a first region in the first image, or, based on the image data of the first image, identify a first region in the first image where the target imaging subject is located, wherein the target imaging subject is located within the first region; The target region in the first image is determined based on the first region, and the target region includes at least the target imaging subject within the first region.

3. The method according to claim 1 or 2, wherein, Determining the target region of the first image includes: Obtain the first region of the first image; Image extraction is performed on the first region to determine the target imaging subject in the first region, and the target imaging subject is determined as the target region of the first image.

4. The method according to claim 2, wherein, The step of identifying the first region where the target imaging subject is located in the first image based on the image data of the first image includes: A segmentation model based on a deep neural network mechanism is used to segment the image data of the first image to obtain the first region where the target imaging subject is located in the first image.

5. The method according to claim 1, wherein, Each first pixel corresponds to a first cost dataset; All second-pixel points correspond to the same second cost dataset.

6. The method according to claim 1, wherein, The first cost calculation includes: Each first pixel is allocated a separate independent cost storage space to store the first cost dataset corresponding to each first pixel. Each first pixel corresponds to a separate independent cost storage space.

7. The method according to claim 6, wherein, The second cost calculation includes: Allocate a shared cost storage space for all second pixels within the target area to store the second cost dataset, and all second pixels within the target area share the second cost dataset.

8. The method according to claim 7, wherein, The shared second cost dataset includes: Obtain the third cost dataset corresponding to each second pixel, wherein the third cost dataset includes multiple second-generation values; The second-generation values ​​at corresponding positions in each third cost dataset are shared to obtain shared second-generation values. The shared second-generation values ​​at multiple positions constitute the second cost dataset.

9. The method according to claim 8, wherein, The process of sharing the second-generation value at each corresponding position includes: The second-generation values ​​at each corresponding position are smoothed.

10. An electronic device, comprising: An image acquisition device for obtaining a first image and a second image whose relative imaging positional relationship is determined; The processor is configured to determine a target region of the first image, wherein the region outside the target region is a non-target region, and the target region includes at least a target imaging subject; and to perform a first cost calculation based on each first pixel in the non-target region of the first image and the corresponding pixel in the second image to obtain a first cost dataset corresponding to the first pixel, wherein the first cost calculation includes at least a first cost aggregation process. A second cost calculation is performed on each second pixel in the target region of the first image and its corresponding pixel in the second image to obtain a second cost dataset corresponding to the second pixel. The second cost calculation includes a second cost aggregation process, wherein the number of aggregation directions in the second cost aggregation process is less than the number of aggregation directions in the first cost aggregation process. A disparity map is obtained based on the first cost dataset and the second cost dataset.

Images