Image feature processing method and device

Abstract

This invention discloses an image feature processing method and apparatus, relating to the field of computer technology. The specific implementation of the method includes: generating a left and right image using a binocular camera; extracting features from the left and / or right image using a feature extraction network to obtain image features; using a preset cost aggregation scaling factor as a step size, traversing from the initial value to the maximum disparity of the image features to obtain multiple disparities to construct a first cost aggregation for aggregation processing, generating a first disparity map; constructing a second cost aggregation based on the image features and the first disparity map for aggregation processing, generating a second disparity map; and using a two-dimensional convolutional neural network to refine the second disparity map to generate target disparity maps of the left and right image features. This implementation is based on a coarse-to-fine two-stage non-full-process cost aggregation method to calculate the disparity maps of the left and right image features, solving the problem of slow speed in existing 3D cost aggregation networks while maintaining sufficient accuracy.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an image feature processing method and apparatus. Background Technology

[0002] Stereo matching, also known as disparity estimation, aims to find a pair of associated matching points in a stereo image and is an important subclass of autonomous driving vision technology. Stereo matching is challenging due to a variety of practical problems, such as large, weakly textured regions (walls, sky, and other backgrounds), occlusion, reflective surfaces, and other factors. The difficulty lies in eliminating ambiguity and blurriness in the matching images.

[0003] The current approach to address these challenges typically employs cost aggregation: GC-Net constructs a cost aggregate and uses a 3D convolutional neural network to aggregate information from it; PSMNet improves upon GC-Net but still uses the aforementioned structure for constructing the cost aggregate. This network framework, which constructs the cost aggregate and uses a 3D convolutional neural network for aggregation, has a profound influence and its advantage is extremely high accuracy. It is still widely used and studied by scholars to this day.

[0004] In developing this invention, the inventors discovered that the prior art has at least the following problems: Because a large cost aggregate is constructed and aggregated using a 3D convolutional neural network, the framework is relatively large (compared to using a 2D convolutional neural network throughout), resulting in a longer time consumption. Furthermore, while the traditional method of constructing the cost aggregate using all values ​​between 0 and the maximum disparity can generate sufficient cost information to achieve high network accuracy, it is also typically a key factor contributing to the slow inference speed of the entire network. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide an image feature processing method and apparatus, which can at least solve the problem that the existing PSMNet framework is too large and uses all values ​​between 0 and the maximum disparity to construct the cost aggregate, which leads to slow network inference speed.

[0006] To achieve the above objectives, according to one aspect of the present invention, an image feature processing method is provided, comprising: generating a left image and a right image using a binocular camera; extracting features from the left image and / or the right image using a feature extraction network to obtain image features; wherein the image features are left image features and / or right image features; traversing from an initial value to the maximum disparity of the image features using a preset cost aggregate scaling factor as a step size to obtain multiple disparities to construct a first cost aggregate for aggregation processing to generate a first disparity map; constructing a second cost aggregate based on the image features and the first disparity map for aggregation processing to generate a second disparity map; and refining the second disparity map using a two-dimensional convolutional neural network to generate a target disparity map of the left image features and the right image features.

[0007] Optionally, when the image feature is a left image feature, the step of constructing a second cost aggregate for aggregation processing based on the image feature and the first disparity map includes: for any point in the left image feature, obtaining the coarse disparity in the first disparity map; wherein the coarse disparity is obtained by coarsely estimating the disparity of the point; obtaining the abscissa and ordinate of the point, calculating the new abscissa after subtracting the value of the coarse disparity from the abscissa, and determining whether there is a point in the right image feature corresponding to the new abscissa and ordinate; if not, performing a zero-filling operation, otherwise, shifting the point along the positive direction of the horizontal axis by the coarse disparity units; wherein the size of the right image feature after shifting remains unchanged; concatenating the shifted right image feature with the first disparity map to obtain a matching tensor, and then subtracting the matching tensor from the left image feature unit by unit in the disparity dimension and taking the absolute value to obtain the second cost aggregate for aggregation processing.

[0008] Optionally, when the image feature is a right image feature, the step of constructing a second cost aggregate for aggregation processing based on the image feature and the first disparity map includes: for any point in the left image feature, obtaining the coarse disparity in the first disparity map; obtaining the x-coordinate and y-coordinate of the point, calculating the new x-coordinate after adding the value of the coarse disparity to the x-coordinate, and determining whether there is a point in the left image feature corresponding to the new x-coordinate and y-coordinate; if not, performing a zero-filling operation, otherwise, shifting the point along the negative x-axis by the coarse disparity units; wherein the size of the shifted left image feature remains unchanged; concatenating the shifted left image feature with the first disparity map to obtain a matching tensor, and then subtracting the matching tensor from the right image feature unit by unit in the disparity dimension and taking the absolute value to obtain the second cost aggregate for aggregation processing.

[0009] Optionally, when the image features include left image features and right image features, the step of constructing a second cost aggregate for aggregation processing based on the image features and the first disparity map includes: simultaneously determining the height and width of any channel in the left / right image features, creating a one-dimensional matrix with values ​​from 1 to the width, repeating the height times in the vertical direction to generate an empty two-dimensional tensor; obtaining the coarse disparity of each point in the first disparity map in any channel, assigning the coarse disparity to the empty two-dimensional tensor one by one according to the point position to obtain the two-dimensional tensor of the left / right image features; padding the leftmost side of the two-dimensional tensor of the right image features with 0; taking the negative value of the two-dimensional tensor of the left image features and performing activation processing to obtain a mapping value, using the mapping value as the horizontal coordinate, and combining it with the vertical coordinate of each point in the left image features to generate mapping coordinates mapped to the two-dimensional tensor of the right image features; constructing a matching tensor based on the mapping coordinates, subtracting the matching tensor from the left image features in the disparity dimension and taking the absolute value to obtain the second cost aggregate for aggregation processing.

[0010] Optionally, the method further includes: generating a disparity interval corresponding to any point based on the coarse disparity, the preset cost aggregate scaling factor, and the preset convolutional network adaptation factor; selecting multiple disparities starting from the leftmost end of the disparity interval with a step size of 1, counting the number of multiple disparities, and obtaining the length of the second cost aggregate in the disparity dimension.

[0011] Optionally, the length of the first cost aggregate in the disparity dimension is the number of the plurality of disparities plus 1.

[0012] Optionally, the method further includes: using a first three-dimensional convolutional neural network to aggregate a first cost aggregate to obtain a first tensor, and then performing regression processing on the first tensor in the disparity dimension; and using a second three-dimensional convolutional neural network to aggregate a second cost aggregate to obtain a second tensor, and then performing regression processing on the second tensor in the disparity dimension.

[0013] Optionally, the method further includes: for any point on any disparity map, obtaining the disparity prediction value and the disparity true value on the any disparity map, and calculating the difference between the two; in response to the absolute value of the difference being less than 1, calculating the smoothing loss of the any point using a preset first function, otherwise calculating the smoothing loss of the any point using a preset second function; accumulating the smoothing losses of all points, taking the ratio of the total smoothing loss to the total number of points on the any disparity map as the loss value of the any disparity map, and combining it with the preset weight values ​​of each convolutional neural network to accumulate the total loss of all disparity maps; adjusting the parameters of each convolutional neural network based on the total loss, and repeating the above iterative process until the total loss after iteration reaches a preset value.

[0014] Optionally, the step of using a two-dimensional convolutional neural network to refine the second disparity map to generate a target disparity map of the left image features and the right image features includes: using a one-time bilinear interpolation method to perform upsampling processing on the first disparity map and the second disparity map twice to obtain two sampling results for concatenation; inputting the concatenation result into a two-dimensional convolutional neural network for processing, and constructing a residual structure to obtain a target disparity map between the left image features and the right image features.

[0015] Optionally, after generating the target disparity map of the left image features and the right image features, the method further includes: performing disparity correction on the left and right images in the binocular camera based on the target disparity map.

[0016] To achieve the above objectives, according to another aspect of the present invention, an image feature processing apparatus is provided, comprising: a feature extraction module, configured to generate a left image and a right image using a binocular camera, and extract features of the left image and / or the right image using a feature extraction network to obtain image features; wherein the image features are left image features and / or right image features; a first disparity map module, configured to traverse from an initial value to the maximum disparity of the image features with a preset cost aggregate scaling factor as the step size, obtain multiple disparities to construct a first cost aggregate for aggregation processing, and generate a first disparity map; a second disparity map module, configured to construct a second cost aggregate based on the left image features, the right image features, and the first disparity map for aggregation processing, and generate a second disparity map; and a target disparity map module, configured to refine the second disparity map using a two-dimensional convolutional neural network to generate a target disparity map of the left image features and the right image features.

[0017] Optionally, when the image feature is a left image feature, the second disparity map module is configured to: for any point in the left image feature, obtain a coarse disparity in the first disparity map; wherein the coarse disparity is obtained by coarsely estimating the disparity of the point; obtain the x-coordinate and y-coordinate of the point, calculate the new x-coordinate after subtracting the value of the coarse disparity from the x-coordinate, and determine whether there is a point in the right image feature corresponding to the new x-coordinate and y-coordinate; if not, perform a zero-filling operation; otherwise, translate the point along the positive x-axis by the coarse disparity units; wherein the size of the translated right image feature remains unchanged; concatenate the translated right image feature with the first disparity map to obtain a matching tensor, and then subtract the matching tensor from the left image feature unit by unit in the disparity dimension and take the absolute value to obtain a second cost aggregate for aggregation processing.

[0018] Optionally, when the image feature is a right image feature, the second disparity map module is configured to: for any point in the left image feature, obtain the coarse disparity in the first disparity map; obtain the x-coordinate and y-coordinate of the point, calculate the new x-coordinate after adding the value of the coarse disparity to the x-coordinate, and determine whether there is a point in the left image feature corresponding to the new x-coordinate and y-coordinate; if not, perform a zero-filling operation; otherwise, shift the point along the negative x-axis by the coarse disparity units; wherein the size of the shifted left image feature remains unchanged; concatenate the shifted left image feature with the first disparity map to obtain a matching tensor, and then subtract the matching tensor from the right image feature unit by unit in the disparity dimension and take the absolute value to obtain a second cost aggregate for aggregation processing.

[0019] Optionally, when the image features include left image features and right image features, the second disparity map module is configured to: simultaneously determine the height and width of any channel in the left / right image features, create a one-dimensional matrix with a value from 1 to the width, repeat the height times in the vertical direction to generate an empty two-dimensional tensor; obtain the coarse disparity of each point in the first disparity map, assign the coarse disparity to the empty two-dimensional tensor according to the point position, and obtain two-dimensional tensors of the left / right image features; and pad the leftmost side of the two-dimensional tensor of the right image features with 0; take the negative value of the two-dimensional tensor of the left image features and perform activation processing to obtain a mapping value, use the mapping value as the horizontal coordinate, combine it with the vertical coordinate of each point in the left image features to generate mapping coordinates mapped to the two-dimensional tensor of the right image features; construct a matching tensor based on the mapping coordinates, subtract the matching tensor from the left image features in the disparity dimension and take the absolute value to obtain a second cost aggregate for aggregation processing.

[0020] Optionally, the second disparity map module is further configured to: generate a disparity interval corresponding to any point based on the coarse disparity, the preset cost aggregate scaling factor, and the preset convolutional network adaptation factor; select multiple disparities starting from the leftmost end of the disparity interval with a step size of 1, count the number of multiple disparities, and obtain the length of the second cost aggregate in the disparity dimension.

[0021] Optionally, the length of the first cost aggregate in the disparity dimension is the number of the plurality of disparities plus 1.

[0022] Optionally, the first disparity map module is configured to: use a first three-dimensional convolutional neural network to aggregate the first cost aggregate to obtain a first tensor, and then perform regression processing on the first tensor in the disparity dimension; and the second disparity map module is configured to: use a second three-dimensional convolutional neural network to aggregate the second cost aggregate to obtain a second tensor, and then perform regression processing on the second tensor in the disparity dimension.

[0023] Optionally, the device further includes an iteration module, configured to: for any point on any disparity map, obtain the predicted disparity value and the true disparity value on the any disparity map, and calculate the difference between the two; in response to the absolute value of the difference being less than 1, calculate the smoothing loss of the any point using a preset first function, otherwise calculate the smoothing loss of the any point using a preset second function; accumulate the smoothing losses of all points, and use the ratio of the total smoothing loss to the total number of points on the any disparity map as the loss value of the any disparity map, and combine it with the preset weight values ​​of each convolutional neural network to accumulate the total loss of all disparity maps; adjust the parameters of each convolutional neural network based on the total loss, and repeat the above iteration process until the total loss after iteration reaches a preset value.

[0024] Optionally, the target disparity map module is used to: perform upsampling processing on the first disparity map and the second disparity map twice using a one-time bilinear interpolation device to obtain two sampling results for concatenation; input the concatenation result into a two-dimensional convolutional neural network for processing, and construct a residual structure to obtain a target disparity map between the left image features and the right image features.

[0025] Optionally, the device further includes a correction module for: performing parallax correction on the left and right images in the binocular camera based on the target parallax map.

[0026] To achieve the above objectives, according to another aspect of the present invention, a cost-aggregation electronic device is provided.

[0027] The electronic device of this invention includes: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement any of the image feature processing methods described above.

[0028] To achieve the above objectives, according to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the image feature processing methods described above.

[0029] According to the solution provided by the present invention, one embodiment of the invention has the following advantages or beneficial effects: the first-stage cost aggregation constructs a smaller first cost aggregate using a portion of the disparity, and then uses a 3D convolutional neural network to aggregate it, ultimately obtaining a coarse disparity map that is more accurate at a larger scale but less accurate at a smaller scale; the second-stage cost aggregation expands the range again by a smaller area based on the coarse disparity map and uses all the disparities within the range to construct a second cost aggregate, and then uses a 3D convolutional neural network to aggregate it, so that the network aggregates the disparity range centered on the coarse disparity again at the smallest scale, outputting a more accurate disparity map; finally, a disparity refinement structure is added to further improve the accuracy of the network.

[0030] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0031] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0032] Figure 1 This is a schematic diagram of the main flow of an image feature processing method according to an embodiment of the present invention;

[0033] Figure 2(a) is a schematic diagram of the first-stage 3D convolutional network process;

[0034] Figure 2(b) is a schematic diagram of the first-stage cost aggregation structure;

[0035] Figure 3 This is a network flowchart for the first and second phases;

[0036] Figure 4 This is a flowchart illustrating an optional image feature processing method according to an embodiment of the present invention;

[0037] Figure 5 This is a flowchart illustrating another optional image feature processing method according to an embodiment of the present invention;

[0038] Figure 6 This is a flowchart illustrating another optional image feature processing method according to an embodiment of the present invention;

[0039] Figure 7 This is a schematic diagram of the main modules of an image feature processing device according to an embodiment of the present invention;

[0040] Figure 8 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0041] Figure 9This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present invention, such as a mobile device or server. Detailed Implementation

[0042] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0043] The terms used in this plan are explained as follows:

[0044] Parallax: The directional difference that occurs when observing the same target from two points at a certain distance. For example, if you extend a finger in front of your eyes, close your right eye and look at it with your left eye; then close your left eye and look at it with your right eye, you will find that the position of the finger relative to a distant object has changed. This is the parallax of viewing the same point from different angles.

[0045] Receptive field: When a receptor is stimulated and excited, nerve impulses (various sensory information) are transmitted to the higher center through the centripetal neurons in the sensory organ. The area of ​​stimulation that a neuron responds to (controls) is called the neuron's receptive field.

[0046] Cost aggregation: Pixel-by-pixel matching is not very accurate. To prevent noise interference, a smoothness constraint is added by penalizing the neighborhood disparity change at each location.

[0047] See Figure 1 The diagram shows the main flowchart of an image feature processing method provided by an embodiment of the present invention, which includes the following steps:

[0048] S101: Generate a left image and a right image using a binocular camera, and extract features from the left image and / or the right image using a feature extraction network to obtain image features; wherein, the image features are left image features and / or right image features;

[0049] S102: Using a preset cost aggregate scaling factor as the step size, traverse from the initial value to the maximum disparity of the image feature to obtain multiple disparities to construct a first cost aggregate for aggregation processing, and generate a first disparity map.

[0050] S103: Based on the image features and the first disparity map, construct a second cost aggregate for aggregation processing to generate a second disparity map;

[0051] S104: Using a two-dimensional convolutional neural network, refine the second disparity map to generate a target disparity map of the left image features and the right image features.

[0052] In the above embodiments, for steps S101 to S102, it is noted that existing methods use 3D convolutional neural networks to aggregate cost aggregates. Therefore, the size of the cost aggregate directly affects its aggregation speed and the inference speed of the entire network. Traditionally, the cost aggregate is constructed from 0 to the maximum disparity (usually 192) in the visual dimension, which is equivalent to constructing the entire search space. Therefore, by reducing the disparity dimension of the cost aggregate, the entire network can maintain a certain level of accuracy while improving speed.

[0053] The network provided in this solution is built on PSMNet3 and uses a segmented, non-full-process cost aggregation method to replace the cost aggregation method in the existing PSMNet, focusing on alleviating the slow speed of existing 3D cost aggregation network frameworks. The overall structure is similar to PSMNet, consisting of four stages: feature extraction, first-stage cost aggregation construction and cost aggregation, second-stage cost aggregation construction and cost aggregation, and disparity refinement. The feature extraction stage is the same as the existing PSMNet, using a feature extraction network to extract high-dimensional features from the left and right images of the stereo camera, obtaining left and right image feature maps.

[0054] The first stage, cost aggregation, aims to construct and aggregate costs, and to perform a coarse estimation of the left and / or right feature maps to obtain the corresponding coarse disparity map (i.e., the first disparity map).

[0055] This stage first occurs in [0, d] max Within the range of ), with c (the cost aggregate scaling factor, which needs to be manually specified based on experimental data statistics, usually greater than 1, such as 4) as the step size, multiple disparities are selected to construct the first cost aggregate, i.e., the sparse cost aggregate; where d max The maximum disparity at the Feature Map level is typically reduced proportionally from the initial maximum disparity as the network downsamples; therefore, the size of the first cost aggregate is approximately one-c times the size of the traditional cost aggregate.

[0056] Furthermore, since the feature extraction network contains two downsampling layers, therefore d max Given one-quarter of the initial maximum disparity, i.e., one-quarter of D, and assuming the constructed first cost aggregate has a length of d1 units in the disparity dimension, then:

[0057] [] indicates rounding down.

[0058] Referring to Figures 2(a) and (b), each convolutional kernel performs a convolution on the output of the previous layer. Assuming the Feature Map size of the first cost aggregate is 32×0.0625D×0.25H×0.25w, convolutional features are extracted sequentially using 3Dconv0 and 3Dconv1, resulting in an output Feature Map size of 32×0.0625D×0.25H×0.25w.

[0059] Next, 3Dstack1 is used to perform dimensionality reduction on the Feature Map output by 3Dconv1. 3Dstack1 includes 3Dconv and 3Dconv(stride=2). Among them, conv(stride=2) performs dimensionality reduction with minimal feature loss and is a commonly used dimensionality reduction operation. Then, 3Dstack2 is used to further reduce the dimensionality of the Feature Map output by 3Dstack1, resulting in a Feature Map of size 64×0.015625D×0.0625H×0.0625W.

[0060] 3Dstack3 takes the Feature Map output from 3Dstack2 as input and performs deconvolution to obtain the deconvolution result, which is used to verify the Feature Map extracted by 3Dstack1. Similarly, 3Dstack4 takes the Feature Map output from 3Dstack3 as input and performs deconvolution to obtain the deconvolution result, which is used to verify the Feature Map extracted by 3Dconv1.

[0061] 3Dconv2 takes the Feature Map output by 3Dstack4 as input for convolution processing to extract features. Then, regression is used to perform linear regression on the features extracted by 3Dconv2, resulting in a linear combination of features of 1×0.25H×0.25W.

[0062] Then, a first 3D convolutional neural network with fewer layers and a simpler structure aggregates the first cost aggregate, ultimately obtaining a first tensor of 0.0625D×0.25H×0.25W. A softmax operation is then performed on this first tensor along the disparity dimension, and regression is performed to obtain a coarse disparity map. Assume the result of the softmax process is denoted by k, and let d... i Indicating the aforementioned "multiple parallaxes," the coarse parallax map obtained in this stage is as follows:

[0063]

[0064] For step S103, the second stage cost aggregate construction and cost aggregation aims to calculate the fine disparity map (i.e., the second disparity map) corresponding to the left / right Feature Map.

[0065] See Figure 3 As shown, the inputs to the second-stage cost aggregation are the left and right Feature Maps and the coarse disparity map obtained in the first stage. A second cost aggregate is constructed using a 3D CNN (Convolutional Neural Network). The size of the first cost aggregate is assumed to be 32×0.0625D×0.25H×0.25w, and the size of the second cost aggregate is also 32×0.0625D×0.25H×0.25w in the experiment. The specific construction process will be detailed later. Figure 4 and Figure 5 The description shown will not be repeated here. Here, cost-volume specifically refers to a left-right disparity search space in the stereo matching problem of computer vision. Weight sharing in neural networks refers to applying information learned from one local region to other parts of the image. Upsampling aims to enlarge the image for display on higher resolution display devices.

[0066] The second cost aggregate is aggregated by a second 3D convolutional neural network to obtain a second tensor of 0.0625D×0.25H×0.25W. This second 3D convolutional neural network is almost identical to the 3D convolutional neural network in PSMNet, the only difference being that the input channels of the first 3D convolutional kernel are 32 in this scheme, and the output feature map size of each layer is different (the disparity dimension of the cost aggregate in this scheme is only one-quarter that of PSMNet). Alternatively, it can be a stack of PSMNet (approximately one-third the size of PSMNet's 3D network). Following the same processing as the first stage, a softmax operation is then performed on the second tensor along the visual dimension, and regression is performed to obtain a refined disparity map.

[0067] Alternatively, this stage can also employ a method that combines non-convolutional neural network algorithms with convolutional neural networks to calculate a coarse disparity map of the left / right feature maps. This approach was also explored. First, a simple 2D convolutional layer was used to reduce the number of channels in the input feature map to 1. Then, the entire cost aggregate was constructed in the same way as the traditional cost aggregate construction method. Since the number of channels is 1, the time consumed in constructing the cost aggregate here is relatively low. Subsequently, two methods were explored to calculate the disparity map: 1. Performing a softmax operation on the disparity dimension and then taking the softargmin; 2. Calculating the disparity map by referring to the method described in the semi-global stereo matching algorithm SGBM.

[0068] It should be noted that although a fine disparity map of the right feature map can be built here, experimental verification shows that the fine disparity map of the left feature map is more accurate than that of the right feature map. Therefore, it is possible to calculate only the fine disparity map of the left feature map.

[0069] For step S104, disparity refinement: A lightweight 2D convolutional neural network is used to further refine the refined disparity map obtained in the second stage to improve the network's accuracy. This structure references the disparity refinement structure of DeepPruner. The refined disparity map obtained by the 3D convolutional neural network is 1×0.25H×0.25w in size, and needs to be upsampled twice to obtain a target disparity map of 1×H×w (the same size as the real disparity map).

[0070] First, a bilinear interpolation upsampling by a factor of two is performed. The output of the first downsampling convolutional layer of the feature extraction network and the result of the final output of the second-stage 3D convolutional neural network, after being upsampled by a factor of two, are concatenated and input into a lightweight 2D convolutional neural network. Finally, a residual structure (plus the disparity map after being upsampled by a factor of one) is constructed to obtain a refined disparity map. Then, bilinear interpolation is performed again to obtain the 1×H×w target disparity map. Unlike DeepPruner, this scheme does not aggregate information through a simple 2D convolutional layer after the first bilinear interpolation. The lightweight 2D convolutional network structure is shown in Table 1.

[0071] Table 1

[0072] name Layer settings Output Feature Map Size enter 45×0.50H×0.50W 2Dconv0 3×3×3,45 45×0.50H×0.50W 2Dconvl 3×3×3,45 45×0.50H×0.50W 2Dconv2 3×3×3,45 45×0.50H×0.50W 2Dconv3 3×3×3,45,dila=2 45×0.50H×0.50W 2Dconv4 3×3×3,45,dila=4 45×0.50H×0.50W 2Dconv5 3×3×3,45 45×0.50H×0.50W 2Dconv6 3×3×3,1 1×0.50H×0.50W

[0073] Stereo vision systems in binocular cameras have wide applications in robot navigation, intelligent monitoring, and autonomous driving. Therefore, accurate and rapid binocular stereo matching is crucial for the real-time deployment of stereo vision systems on mobile devices, such as eliminating ambiguity and blurriness in matched images. Thus, after obtaining the target disparity map between the features of the left and right images from the binocular camera, this map can be used to perform disparity correction on the left and right images, or to determine whether the binocular camera is qualified.

[0074] The method provided in the above embodiments is based on the aggregation of two non-full-process cost aggregates from coarse to fine, and calculates the disparity map of left and right image features. This solves the problem of slow speed of existing 3D cost aggregation networks, while maintaining sufficient accuracy.

[0075] See Figure 4The diagram illustrates an optional image feature processing method according to an embodiment of the present invention, comprising the following steps:

[0076] S401: For any point in the features of the left image, obtain the coarse disparity in the first disparity map; wherein, the coarse disparity is obtained by performing a coarse disparity estimate on the point;

[0077] S402: Obtain the x-coordinate and y-coordinate of any point, calculate the new x-coordinate after subtracting the value of the coarse parallax from the x-coordinate, and determine whether there is a point in the right image features that corresponds to the new x-coordinate and y-coordinate.

[0078] S403: If it does not exist, perform a zero-filling operation; otherwise, translate the point along the positive horizontal axis by the coarse parallax units; wherein the right image feature size remains unchanged after translation.

[0079] S404: Concatenate the translated right image features with the first disparity map to obtain a matching tensor. Then, subtract the matching tensor from the left image features unit by unit along the disparity dimension and take the absolute value to obtain a second cost aggregate for aggregation processing.

[0080] In the above embodiments, for steps S401 to S404, the specific process of constructing the second cost aggregate is described here, which only requires generating a coarse disparity map and a fine disparity map corresponding to the left FeatureMap.

[0081] The coarse parallax that each pixel on the left Feature Map can obtain may be different. Suppose a point p(x,y) on the left Feature Map has a different coarse parallax value in the coarse parallax map. If the coarse disparity on the second cost aggregate is d, then the disparity interval of point p in the second cost aggregate is (d-(c+σ),d+(c+σ)); where c is the scaling factor of the aforementioned cost aggregate, and σ is a natural number representing the adaptation factor of the convolutional network, which also needs to be manually defined. To ensure that the size of the Feature Map in the disparity dimension in the subsequent 3D convolutional neural network is not less than the size of the convolutional kernel, and to avoid network errors or deviations from expectations, σ should be set as small as possible, such as 2.

[0082] Using a step size of 1, multiple disparities are progressively selected starting from the leftmost end of the disparity interval (d-(c+σ), d+(c+σ)) to determine the length of the second cost aggregate. Assuming the second cost aggregate has a total length of d² units in the disparity dimension, then d² = 2 × (c+σ). Therefore, the first segment of the cost aggregation network aggregates information at a larger scale, while the second segment's disparity interval covers both step sizes of the first stage, aggregating information at a smaller scale with a step size of 1.

[0083] Suppose that point p(x,y) needs to construct a second cost aggregate with a coarse parallax d. Then, the point (xd,y) on the right Feature Map needs to be shifted d units to the right. Unlike traditional cost aggregate construction methods, at the same time, each point on the left Feature Map may need to be constructed with a different coarse parallax, which can cause problems in cost aggregate construction. For example, some points on the left Feature Map may have x-coordinates smaller than their coarse parallax d, so no corresponding point can be found in the right Feature Map for shifting. In this case, 0 needs to be filled in, while some points do not need to be filled in. Points exceeding the maximum value are discarded to maintain the size of the shifted right Feature Map.

[0084] The translated right Feature Map is concatenated with the coarse disparity map to obtain the matching tensor. Then, the matching tensor is subtracted from the left Feature Map unit by unit along the disparity dimension and the absolute value is taken to obtain the second cost aggregate.

[0085] The above describes the method for calculating the second cost aggregate based on the coarse disparity map of the left Feature Map. In practice, it can also be calculated based on the coarse disparity map of the right Feature Map. The operation steps are roughly the same as those described above, with only a few differences, such as: shifting the point (x+d,y) on the left Feature Map to the left by d units, filling empty positions with 0; concatenating the shifted left Feature Map with the coarse disparity map; subtracting it from the right Feature Map and taking the absolute value, finally obtaining the fine disparity map corresponding to the right Feature Map, which will not be elaborated further here.

[0086] The method provided in the above embodiments expands the coarse disparity map obtained in the first stage to a certain range and performs matching cost aggregation again to obtain a more refined disparity map, and the resulting cost aggregate is smaller than that of the prior art.

[0087] See Figure 5 The diagram illustrates another optional image feature processing method according to an embodiment of the present invention, including the following steps:

[0088] S501: Simultaneously determine the height and width of any channel in the left / right image features, create a one-dimensional matrix with a value from 1 to the width, repeat the height times in the vertical direction to generate an empty two-dimensional tensor;

[0089] S502: Obtain the coarse disparity of each point in the first disparity map in any channel, assign the coarse disparity to the empty two-dimensional tensor one by one according to the point position to obtain the two-dimensional tensor of the left / right image features; and pad the leftmost of the two-dimensional tensor of the right image features with 0.

[0090] S503: Take the negative value of the two-dimensional tensor of the left image feature and perform activation processing to obtain the mapping value. Use the mapping value as the horizontal coordinate and combine it with the vertical coordinate of each point on the left image feature to generate the mapping coordinates mapped to the two-dimensional tensor of the right image feature.

[0091] S504: Construct a matching tensor based on the mapped coordinates, subtract the left image features from the matching tensor in the disparity dimension and take the absolute value to obtain a second cost aggregate for aggregation processing.

[0092] In the above embodiments, for steps S501 to S504, the specific process of constructing the second cost aggregate is described here, but it is necessary to generate a coarse disparity map and a fine disparity map corresponding to the left FeatureMap, and a coarse disparity map corresponding to the right FeatureMap respectively.

[0093] according to Figure 4 The description states that while traversal can effectively solve the aforementioned problem of no corresponding points, multiple traversals are slow and time-consuming in Python, which is undesirable. Therefore, a clever method is proposed to construct the second cost aggregate, specifically:

[0094] Assuming each channel in the Feature Map has a size of h*w, cost aggregates are constructed simultaneously on each Feature Map channel. First, a one-dimensional matrix with values ​​from 1 to w is created and repeated h times in the vertical direction to obtain a two-dimensional tensor t of size h*w, which is initially empty. Based on the coarse parallax of each point on the left Feature Map, a two-dimensional tensor m of size h*w is constructed; based on the coarse parallax of each point on the right Feature Map, another two-dimensional tensor of size h*w is constructed, and zeros are padded at the leftmost end to form a two-dimensional tensor r of size h*(w+1).

[0095] Calculate the difference between the two-dimensional vectors tm. After performing a ReLU operation on the difference, the values ​​of each point in the left image features are: the coordinates of the coarse disparity corresponding to each point in the final constructed matching tensor under the current disparity dimension and current channel, along the horizontal axis of the two-dimensional tensor r. Assume that at a certain point p in the final constructed matching tensor under the current disparity dimension and current channel... * (x * ,y * After performing a ReLU operation on the result of tm, (x * ,y* If the value of ) is v, then point p * (x * ,y * The value of ) is equal to the mapped coordinates (v, y) on r. * This can be viewed as a mapping relationship between m and r. By subtracting the matching tensor constructed based on the mapping coordinates from the left Feature Map unit by unit along the disparity dimension and taking the absolute value, the final constructed second cost aggregate can be obtained.

[0096] It should be noted that the matching cost aggregation in the second stage is equivalent to expanding the coarse disparity map obtained in the first stage to a certain range and re-aggregating the matching costs. This means that the final disparity is considered to have high reliability within a certain range centered on the coarse disparity. Therefore, the coarse disparity calculated in the first stage should have high accuracy at larger scales, but does not need to have high accuracy at smaller scales.

[0097] The method provided in the above embodiments uses matrix assignment to replace part of the traversal, and the traversal part only needs to traverse the disparity interval to complete the construction of the second cost aggregate.

[0098] See Figure 6 The diagram illustrates another optional image feature processing method according to an embodiment of the present invention, including the following steps:

[0099] S601: For any point on any disparity map, obtain the disparity prediction value and the disparity true value on the disparity map, and calculate the difference between the two.

[0100] S602: In response to the absolute value of the difference being less than 1, the smoothing loss of any point is calculated using a preset first function; otherwise, the smoothing loss of any point is calculated using a preset second function.

[0101] S603: Accumulate the smoothing loss of all points, and use the ratio of the total smoothing loss to the total number of points on any disparity map as the loss value of any disparity map. Combine this with the preset weight value of each convolutional neural network to accumulate the total loss of all disparity maps.

[0102] S604: Adjust the parameters of each convolutional neural network based on the total loss, and repeat the above iterative process until the total loss after iteration reaches the preset value.

[0103] In the above implementation, for steps S601 to S603, the loss function is applied in the training phase of the 3D convolutional neural network. There are a total of 4 outputs in the training phase of the network, but only one is used for inference. A loss function is constructed for each output, and three of the loss functions are consistent with the existing PSMNet.

[0104] Assuming the output losses at different levels in a 3D convolutional neural network are denoted by O1, O2, and O3 respectively, unlike the existing PSMNet, this network outputs a coarse disparity map and constructs a loss function based on it, denoted by O0. All four loss functions are represented using L1 notation, then:

[0105]

[0106]

[0107] Where N represents the total number of points on the disparity map, d i This represents the true disparity value of point i. This represents the disparity prediction value for point i.

[0108] Assuming the final loss is represented by L, then: L=μ0O0+μ1O1+μ2O2+μ3O3; where μ0, μ1, μ2, and μ3 represent weight parameters. μ1, μ2, and μ3 are consistent with the settings of the existing PSMNet, which are 0.5, 0.7, and 1 respectively (adjustable in practice). μ0 is preferably set to 1.

[0109] For step S604, the parameters of the 3D convolutional neural network and the 2D convolutional neural network can be adjusted according to the loss L to repeat the above process and continuously test until the total loss reaches the preset value.

[0110] The method provided in the above embodiments uses the SmoothLoss function as the optimization objective to make the gradient of backpropagation change more smoothly with the error.

[0111] This invention provides a two-segment 3D cost aggregation network, from coarse to fine, called a segmented non-full-process cost aggregation network. Each segment of cost aggregation uses only a portion of the disparity from 0 to the maximum to construct the cost aggregate. Specifically:

[0112] 1. In the first stage, starting from 0 and ending at the maximum disparity, the disparity is taken once every interval c to construct the first cost aggregate. A 3D cost aggregation network with fewer layers is used to aggregate and optimize it to obtain a coarse disparity map. The computational time complexity is about 1 / c of ​​the original method.

[0113] 2. The second stage is based on the coarse disparity map obtained in the first stage. Assuming that a point p in the left feature map has a disparity of d on the coarse disparity map, the disparity range is (d-(c+σ),d+(c+σ)). Then, a fine disparity map is calculated through a 3D convolutional neural network. The computation time complexity is about 2 / 48*(c+σ) of the original cost aggregation method.

[0114] 3. Both the first-stage matching cost aggregation network and the second-stage matching cost aggregation network contain two downsampling operations and have a minimum convolutional kernel size of 3. Therefore, in a sense, the second-stage Feature Map has a global receptive field in the disparity dimension, while the receptive field of PSMNet in the disparity dimension is smaller than the maximum disparity.

[0115] See Figure 7 The diagram shows a schematic representation of the main modules of an image feature processing device 700 provided in an embodiment of the present invention, including:

[0116] The feature extraction module 701 is used to generate a left image and a right image using a stereo camera, and to extract features of the left image and / or the right image using a feature extraction network to obtain image features; wherein, the image features are left image features and / or right image features;

[0117] The first disparity map module 702 is used to traverse from the initial value to the maximum disparity of the image features with a preset cost aggregate scaling factor as the step size, obtain multiple disparities to construct a first cost aggregate for aggregation processing, and generate a first disparity map; wherein, the image features are left image features and / or right image features.

[0118] The second disparity map module 703 is used to construct a second cost aggregate based on the left image features, the right image features and the first disparity map to perform aggregation processing and generate a second disparity map;

[0119] The target disparity map module 704 is used to refine the second disparity map using a two-dimensional convolutional neural network to generate a target disparity map of the left image features and the right image features.

[0120] In the apparatus of this invention, when the image feature is a left image feature, the second disparity map module 703 is used to: for any point in the left image feature, obtain the coarse disparity in the first disparity map; obtain the abscissa and ordinate of the point, calculate the new abscissa after subtracting the coarse disparity from the abscissa, and determine whether there is a point in the right image feature corresponding to the new abscissa and ordinate; if not, perform a zero-filling operation; otherwise, translate the point along the positive direction of the horizontal axis by the coarse disparity units; wherein the size of the translated right image feature remains unchanged; concatenate the translated right image feature with the first disparity map to obtain a matching tensor, and then subtract the matching tensor from the left image feature unit by unit in the disparity dimension and take the absolute value to obtain a second cost aggregate for aggregation processing.

[0121] In the apparatus of this invention, when the image feature is a right image feature, the second disparity map module 703 is used to: for any point in the left image feature, obtain the coarse disparity in the first disparity map; obtain the abscissa and ordinate of the point, calculate the new abscissa after adding the value of the coarse disparity to the abscissa, and determine whether there is a point in the left image feature corresponding to the new abscissa and ordinate; if not, perform a zero-filling operation; otherwise, shift the point along the negative direction of the horizontal axis by the coarse disparity units; wherein the size of the shifted left image feature remains unchanged; concatenate the shifted left image feature with the first disparity map to obtain a matching tensor, and then subtract the matching tensor from the right image feature unit by unit in the disparity dimension and take the absolute value to obtain a second cost aggregate for aggregation processing.

[0122] In the apparatus of this invention, when the image features include left image features and right image features, the second disparity map module 703 is used to: simultaneously determine the height and width of any channel in the left / right image features, create a one-dimensional matrix with a value from 1 to the width, repeat the height times in the vertical direction to generate an empty two-dimensional tensor; obtain the coarse disparity of each point in the first disparity map, assign the coarse disparity to the empty two-dimensional tensor one by one according to the point position to obtain the two-dimensional tensor of the left / right image features; and pad the leftmost side of the two-dimensional tensor of the right image features with 0; take the negative value of the two-dimensional tensor of the left image features and perform activation processing to obtain a mapping value, use the mapping value as the horizontal coordinate, combine it with the vertical coordinate of each point in the left image features to generate the mapping coordinates mapped to the two-dimensional tensor of the right image features; construct a matching tensor based on the mapping coordinates, subtract the matching tensor from the left image features in the disparity dimension and take the absolute value to obtain a second cost aggregate for aggregation processing.

[0123] In the device of the present invention, the second disparity map module 703 is further configured to: generate a disparity interval corresponding to any point based on the coarse disparity, the preset cost aggregate scaling factor, and the preset convolutional network adaptation factor; select multiple disparities starting from the leftmost end of the disparity interval with a step size of 1, count the number of multiple disparities, and obtain the length of the second cost aggregate in the disparity dimension.

[0124] In the apparatus of the present invention, the length of the first cost aggregate in the disparity dimension is the number of the plurality of disparities plus 1.

[0125] In the apparatus of the present invention, the first disparity map module 702 is used to: perform aggregation processing on the first cost aggregate using a first three-dimensional convolutional neural network to obtain a first tensor, and then perform regression processing on the first tensor in the disparity dimension; and the second disparity map module 703 is used to: perform aggregation processing on the second cost aggregate using a second three-dimensional convolutional neural network to obtain a second tensor, and then perform regression processing on the second tensor in the disparity dimension.

[0126] The apparatus of this invention further includes an iteration module, configured to: for any point on any disparity map, obtain the predicted disparity value and the true disparity value on the disparity map, and calculate the difference between the two; in response to the absolute value of the difference being less than 1, calculate the smoothing loss of the point using a preset first function, otherwise calculate the smoothing loss of the point using a preset second function; accumulate the smoothing losses of all points, and use the ratio of the total smoothing loss to the total number of points on the disparity map as the loss value of the disparity map, and combine it with the preset weight values ​​of each convolutional neural network to accumulate the total loss of all disparity maps; adjust the parameters of each convolutional neural network based on the total loss, and repeat the above iteration process until the total loss after iteration reaches a preset value.

[0127] In the apparatus of this invention, the target disparity map module 703 is used to: perform upsampling processing on the first disparity map and the second disparity map twice using a one-time bilinear interpolation device to obtain two sampling results for cascading; input the cascading result into a two-dimensional convolutional neural network for processing, and construct a residual structure to obtain the target disparity map between the left image features and the right image features.

[0128] The apparatus of the present invention further includes a correction module, used to: perform disparity correction on the left and right images in the binocular camera based on the target disparity map.

[0129] Furthermore, the specific implementation details of the device described in the embodiments of the present invention have been described in detail in the above-described method, so the details will not be repeated here.

[0130] Figure 8 An exemplary system architecture 800 to which embodiments of the present invention can be applied is shown, including terminal devices 801, 802, 803, network 804, and server 805 (only an example).

[0131] Terminal devices 801, 802, and 803 can be various electronic devices with displays and web browsing capabilities, and can be equipped with various communication client applications. Users can use terminal devices 801, 802, and 803 to interact with server 805 via network 804 to receive or send messages, etc.

[0132] Network 804 is a medium used to provide a communication link between terminal devices 801, 802, 803 and server 805. Network 804 can include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0133] Server 805 can be a server that provides various services for performing graph feature extraction, coarse disparity map, fine disparity map, and target disparity map generation operations.

[0134] It should be noted that the method provided in the embodiments of the present invention is generally executed by server 805, and correspondingly, the device is generally located in server 805.

[0135] It should be understood that Figure 8 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0136] The following is for reference. Figure 9 It shows a schematic diagram of the structure of a computer system 900 suitable for implementing a terminal device of the present invention. Figure 9 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0137] like Figure 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 902 or programs loaded from storage section 908 into random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the system 900. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0138] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 910 as needed so that computer programs read from it can be installed into storage section 908 as needed.

[0139] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit (CPU) 901, it performs the functions defined above in the system of this invention.

[0140] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0142] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor may be described as including a feature extraction module, a first disparity map module, a second disparity map module, and a target disparity map module. The names of these modules do not necessarily limit the module itself; for example, the target disparity map module may also be described as a "refined disparity map module."

[0143] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include:

[0144] A left and right image is generated using a binocular camera, and features of the left and / or right image are extracted using a feature extraction network to obtain image features; wherein, the image features are left image features and / or right image features;

[0145] Using a preset cost aggregate scaling factor as the step size, the image features are iterated from the initial value to the maximum disparity to obtain multiple disparities to construct a first cost aggregate for aggregation processing, thereby generating a first disparity map.

[0146] Based on the image features and the first disparity map, a second cost aggregate is constructed for aggregation processing to generate a second disparity map.

[0147] A two-dimensional convolutional neural network is used to refine the second disparity map to generate a target disparity map of the left image features and the right image features.

[0148] According to the technical solution of the present invention, the first stage of cost aggregation constructs a smaller first cost aggregate using a portion of the disparity, and then uses a 3D convolutional neural network to aggregate it, ultimately obtaining a coarse disparity map that is more accurate at a larger scale but less accurate at a smaller scale; the second stage of cost aggregation expands the range again by a smaller range based on the coarse disparity map and uses all the disparities within the range to construct a second cost aggregate, and then uses a 3D convolutional neural network to aggregate it, so that the network aggregates the disparity range centered on the coarse disparity again at the smallest scale, outputting a more accurate disparity map; finally, a disparity refinement structure is added to further improve the accuracy of the network.

[0149] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An image feature processing method, characterized in that, include: A left and right image is generated using a binocular camera, and features of the left and / or right image are extracted using a feature extraction network to obtain image features; wherein, the image features are left image features and / or right image features; Using a preset cost aggregate scaling factor as the step size, the image features are iterated from the initial value to the maximum disparity to obtain multiple disparities to construct a first cost aggregate for aggregation processing, thereby generating a first disparity map. Based on the image features and the first disparity map, a second cost aggregate is constructed for aggregation processing to generate a second disparity map. The second disparity map is refined using a two-dimensional convolutional neural network to generate a target disparity map of the left image features and the right image features. This process includes: performing upsampling on the first disparity map and the second disparity map twice using a one-time bilinear interpolation method to obtain two sampling results for concatenation; inputting the concatenation result into a two-dimensional convolutional neural network for processing and constructing a residual structure to obtain a target disparity map between the left image features and the right image features.

2. The method according to claim 1, characterized in that, When the image feature is a left image feature, the step of constructing a second cost aggregate based on the image feature and the first disparity map for aggregation processing includes: For any point in the left image features, obtain the coarse disparity in the first disparity map; wherein, the coarse disparity is obtained by performing a coarse disparity estimate on the point; Obtain the x-coordinate and y-coordinate of any point, calculate the new x-coordinate after subtracting the value of the coarse parallax from the x-coordinate, and determine whether there is a point in the right image features that corresponds to the new x-coordinate and y-coordinate. If it does not exist, fill it with 0; otherwise, translate the point along the positive horizontal axis by the coarse parallax units; wherein the right image feature size remains unchanged after translation. The translated right image features are concatenated with the first disparity map to obtain a matching tensor. Then, the matching tensor is subtracted from the left image features unit by unit along the disparity dimension and the absolute value is taken to obtain a second cost aggregate for aggregation processing.

3. The method according to claim 1, characterized in that, When the image feature is a right image feature, the step of constructing a second cost aggregate based on the image feature and the first disparity map for aggregation processing includes: For any point in the left image features, obtain the coarse disparity in the first disparity map; Obtain the x-coordinate and y-coordinate of any point, calculate the new x-coordinate by adding the value of the coarse parallax to the x-coordinate, and determine whether there is a point in the left image features that corresponds to the new x-coordinate and y-coordinate. If it does not exist, fill it with 0; otherwise, shift the point along the negative horizontal axis by the coarse parallax units; wherein the left image feature size remains unchanged after the shift. The translated left image features are concatenated with the first disparity map to obtain a matching tensor. Then, the matching tensor is subtracted from the right image features unit by unit along the disparity dimension and the absolute value is taken to obtain a second cost aggregate for aggregation processing.

4. The method according to claim 1, characterized in that, When the image features include left image features and right image features, the step of constructing a second cost aggregate based on the image features and the first disparity map for aggregation processing includes: Simultaneously determine the height and width of any channel in the left / right image features, create a one-dimensional matrix with values ​​from 1 to the width, and repeat the height times in the vertical direction to generate an empty two-dimensional tensor; Obtain the coarse disparity of each point in any channel in the first disparity map, and assign the coarse disparity to the empty two-dimensional tensor one by one according to the point position to obtain the two-dimensional tensor of the left / right image features; and pad the leftmost of the two-dimensional tensor of the right image features with 0; The two-dimensional tensor of the left image feature is negativeized and activated to obtain a mapping value. The mapping value is used as the horizontal coordinate, and combined with the vertical coordinate of each point on the left image feature, a mapping coordinate is generated that is mapped onto the two-dimensional tensor of the right image feature. A matching tensor is constructed based on the mapped coordinates. The matching tensor is subtracted from the left image feature in the disparity dimension and the absolute value is taken to obtain a second cost aggregate for aggregation processing.

5. The method according to any one of claims 2-4, characterized in that, The method further includes: Based on the coarse disparity, the preset cost aggregate scaling factor, and the preset convolutional network adaptation factor, a disparity interval corresponding to any point is generated. With a step size of 1, select multiple disparities starting from the leftmost end of the disparity interval, count the number of multiple disparities, and obtain the length of the second cost aggregate in the disparity dimension.

6. The method according to claim 5, characterized in that, The length of the first cost aggregate in the disparity dimension is the number of the plurality of disparities plus 1.

7. The method according to claim 6, characterized in that, The method further includes: The first cost aggregate is aggregated using a first 3D convolutional neural network to obtain a first tensor, and then the first tensor is regressed along the disparity dimension; and The second cost aggregate is aggregated using a second three-dimensional convolutional neural network to obtain a second tensor, which is then regressed along the disparity dimension.

8. The method according to claim 7, characterized in that, The method further includes: For any point on any disparity map, obtain the predicted disparity value and the true disparity value on the disparity map, and calculate the difference between the two. If the absolute value of the difference is less than 1, the smoothing loss at any point is calculated using a preset first function; otherwise, the smoothing loss at any point is calculated using a preset second function. The smoothing loss of all points is accumulated, and the ratio of the total smoothing loss to the total number of points on any disparity map is used as the loss value of any disparity map. This is combined with the preset weight value of each convolutional neural network to accumulate and obtain the total loss of all disparity maps. The parameters of each convolutional neural network are adjusted based on the total loss, and the above iterative process is repeated until the total loss after iteration reaches the preset value.

9. The method according to claim 1, characterized in that, After generating the target disparity maps of the left image features and the right image features, the method further includes: Based on the target disparity map, disparity correction is performed on the left and right images in the binocular camera.

10. An image feature processing apparatus, characterized in that, include: The feature extraction module is used to generate a left image and a right image using a stereo camera, and to extract features from the left image and / or the right image using a feature extraction network to obtain image features; wherein, the image features are left image features and / or right image features; The first disparity map module is used to iterate from the initial value to the maximum disparity of the image feature with a preset cost aggregate scaling factor as the step size, obtain multiple disparities to construct a first cost aggregate for aggregation processing, and generate a first disparity map. The second disparity map module is used to construct a second cost aggregate based on the left image features, the right image features and the first disparity map to perform aggregation processing and generate a second disparity map; The target disparity map module is used to refine the second disparity map using a two-dimensional convolutional neural network to generate a target disparity map of the left image features and the right image features. This includes: performing two upsampling operations on the first disparity map and the second disparity map using a one-time bilinear interpolation method to obtain two sampling results for concatenation; inputting the concatenation result into a two-dimensional convolutional neural network for processing and constructing a residual structure to obtain a target disparity map between the left image features and the right image features.

11. An electronic device, characterized in that, include: one or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-9.

12. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-9.

Images