Image processing method, chip circuit, device, electronic device and storage medium

Abstract

Provided are an image processing method, a chip circuit, a device, an electronic device, and a storage medium. The image processing method comprises: obtaining boundary features of an image to be processed; based on the obtained boundary features of the image to be processed, performing triangulation processing with boundary restrictions on the image to be processed to obtain a triangulation result that fits the boundary; and performing densification calculation processing related to disparity or depth on the image to be processed according to the triangulation result to obtain a densified disparity map or depth map corresponding to the image to be processed.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, a chip circuit, a device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Stereo vision is an important topic in the field of computer vision. Its purpose is to reconstruct the three-dimensional geometric information of the scene so that people can obtain the depth information in the scene. It has important applications in the fields of three-dimensional environment perception and modeling, robot navigation, object tracking and detection, and image segmentation. Stereo matching is one of the key technologies of stereo vision. It performs pixel registration on two or more images, establishes the correspondence between features, and matches the image points of the same spatial physical point in different images to obtain the corresponding depth information.

[0003] In the application field of stereo vision, obtaining dense disparity maps or depth maps is very necessary to accurately obtain the depth information of the scene. However, in the traditional stereo matching process, due to occlusion or weak texture, repeated texture and other reasons, image matching often produces matching errors, resulting in the traditional stereo matching algorithm often calculating sparse or semi-dense disparity maps.

[0004] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0005] According to one aspect of the present disclosure, an image processing method is provided. The image processing method includes: obtaining boundary features of an image to be processed; based on the obtained boundary features of the image to be processed, performing triangulation processing with boundary restrictions on the image to be processed to obtain a triangulation result that fits the boundary; and according to the triangulation result, performing densification calculation processing related to disparity or depth on the image to be processed to obtain a densified disparity map or depth map corresponding to the image to be processed.

[0006] According to another aspect of the present disclosure, a chip circuit is provided, comprising: a circuit unit configured to execute the steps of the above-mentioned image processing method.

[0007] According to another aspect of the present disclosure, an image processing device is provided, comprising the chip circuit as described above.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising one or more processors and a memory. The memory stores a program comprising instructions, and when the instructions are executed by the one or more processors, the electronic device executes the method as described above.

[0009] According to another aspect of the present disclosure, one or more non-transitory computer-readable media are provided on which a program including instructions is stored. When the instructions are executed by one or more processors of an electronic device, the electronic device executes the method as described above.

[0010] Further features and advantages of the present disclosure will become apparent from the exemplary embodiments described below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0012] Figure 1 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure;

[0013] Figure 2 is a structural block diagram showing a first convolutional neural network for extracting image comprehensive features of an image to be processed according to an exemplary embodiment of the present disclosure;

[0014] Figure 3 is a structural block diagram showing a second convolutional neural network for extracting boundary features of a first image comprehensive feature and an image to be processed itself according to an exemplary embodiment of the present disclosure;

[0015] Figure 4 is a structural block diagram showing a convolutional neural network including two extraction subnetworks for acquiring boundary features of an image to be processed according to an exemplary embodiment of the present disclosure;

[0016] Figure 5A and Figure 5B is a schematic diagram showing an image to be processed according to an exemplary embodiment of the present disclosure;

[0017] Figure 5C and Figure 5D They are shown based on Figure 4 The convolutional neural network shown is Figure 5A and Figure 5B The schematic diagram of the boundary features obtained by extracting the boundary features of the image to be processed is shown;

[0018] Figure 6 is a block diagram illustrating the structure of an exemplary computing device that can be used in the exemplary embodiments. DETAILED DESCRIPTION

[0019] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0020] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0021] In the application field of stereo vision, obtaining dense depth maps or disparity maps is very necessary to accurately obtain the depth information of the scene. However, in the traditional stereo matching process, due to occlusion or weak texture, repeated texture and other reasons, image matching often produces matching errors, resulting in the traditional stereo matching algorithm often calculating sparse or semi-dense disparity maps, thus affecting the accuracy of the determined scene depth information.

[0022] It will be clearly seen in the detailed description below that the image processing method according to the embodiment of the present disclosure can utilize the boundary features of the image to be processed to perform triangulation processing on the image to be processed with boundary restrictions to obtain a triangulation result that fits the boundary, and based on the triangulation result that fits the boundary, perform densification calculation processing related to disparity or depth on the image to be processed to obtain a densified disparity map or depth map corresponding to the image to be processed. Thus, a relatively accurate dense disparity map or depth map can be obtained in a relatively simple manner, which is conducive to improving the accuracy of the scene depth information determined subsequently, and will not burden the computing power of the computing device, which is conducive to achieving a balance between image processing efficiency and processing accuracy.

[0023] In order to better understand the embodiments of the present disclosure, Figure 1 The image processing method of the exemplary embodiment of the present disclosure is described in detail. Figure 1As shown, the image processing method may, for example, include: obtaining boundary features of an image to be processed (step S101); based on the obtained boundary features of the image to be processed, performing triangulation processing on the image to be processed with boundary restrictions to obtain a triangulation result that fits the boundary (step S102); and based on the triangulation result, performing densification calculation processing related to disparity or depth on the image to be processed to obtain a densified disparity map or depth map corresponding to the image to be processed (step S103).

[0024] According to some embodiments, the image to be processed may be any image in an image pair for a target scene acquired by a multi-camera. According to other embodiments, the image to be processed may be a disparity image or a depth image obtained by performing disparity or depth calculation processing on an image pair for a target scene acquired by a multi-camera, for example, a disparity image or a depth image obtained based on a pair of feature points in the image pair (rather than the image data of the entire image or a relatively large image area). For example, in the related art, when performing disparity or depth calculation on a face image, the disparity or depth of sparse facial feature points on the face (for example, feature points at positions such as eyebrows, eyes, nose, mouth, and facial contours) may be obtained to obtain a corresponding sparse or less dense disparity image or depth image, which may be used as the image to be processed in the present disclosure. It can be understood that although the present disclosure uses the example that the image to be processed can be a disparity image or a depth image obtained based on a feature point pair in an image pair, the image to be processed can also be a disparity image or a depth image obtained by using other disparity or depth calculation methods.

[0025] In other words, the image processing method described in the present disclosure can not only be used to perform parallax or depth densification processing on the collected original images of the target scene (such as images that have not been processed by parallax or depth calculation), but can also be used to optimize and complete the acquired disparity images or depth images (such as sparse or not dense enough disparity images or depth images), and the application scenarios are more extensive.

[0026] It should be noted that, in the present disclosure, the multi-eye camera refers to a camera that is configured with two, three or even more cameras and is capable of capturing static or dynamic images. It can cover scenes of different perspectives or ranges through the configured multiple cameras to enhance the camera's ability to detect objects in the scene. Taking a binocular camera configured with two cameras (e.g., a left camera and a right camera) as an example, for any scene, the binocular camera can obtain two images of the scene of the same size but with different corresponding shooting perspectives (e.g., a left-eye image and a right-eye image) through the configured two cameras. The two images can be used to determine the displacement (e.g., horizontal displacement) of the object in the scene between the corresponding pixel points in the two images, i.e., parallax, so as to determine depth information such as the distance of the object.

[0027] According to some embodiments, for an acquired image to be processed, obtaining boundary features of the image to be processed may include: extracting image comprehensive features of the image to be processed to obtain first image comprehensive features of the image to be processed; and extracting boundary features (a specific feature of the image) based on the extracted first image comprehensive features and the image to be processed itself to obtain boundary features of the image to be processed.

[0028] Therefore, when extracting boundary features, not only the data from the high level (such as the extracted comprehensive image features of the image to be processed) but also the data from the bottom level (such as the image to be processed itself) are considered, so that the global information can be integrated to make the acquired boundary features more accurate.

[0029] It can be understood that in the present disclosure, in addition to extracting boundary features based on the extracted first image comprehensive features and the image to be processed itself to obtain the boundary features of the image to be processed, it is also possible to extract boundary features based on one of the extracted first image comprehensive features and the image to be processed itself to obtain the boundary features of the image to be processed.

[0030] According to some embodiments, the image comprehensive features (e.g., the first image comprehensive features) obtained by extracting the image comprehensive features of the image to be processed may include various information that can reflect the basic features of the image to be processed, for example, including but not limited to the basic structural features, semantic features, boundary features, texture features, color features, or object shape features of the image to be processed, etc.

[0031] According to some embodiments, the basic structural features may refer to features used to reflect various fine structures of an image. The semantic features may refer to features that can distinguish different objects or objects of different categories in an image. The boundary features may refer to features that can reflect boundary information of an object or region in an image. The texture features, color features, and object shape features may refer to features that can be used to reflect the texture, color, and shape of an object contained in an image, respectively.

[0032] According to some embodiments, the extracting of image comprehensive features of the image to be processed to obtain the first image comprehensive feature of the image to be processed includes: extracting image comprehensive features of different scales of the image to be processed to obtain multiple image comprehensive features of different sizes for the image to be processed; and fusing the multiple image comprehensive features of different sizes to obtain the first image comprehensive feature.

[0033] Therefore, since the final required first image comprehensive feature can be obtained by fusing image comprehensive features of different scales (i.e., fusing global or different levels of information), the accuracy of the obtained first image comprehensive feature can be improved so that it can better reflect the various features of the image to be processed.

[0034] In the present disclosure, the extracted image features (including comprehensive image features, a specific feature of an image such as a boundary feature, etc.) can be represented in the form of a corresponding image or picture. The size of each image or picture (including an image to be processed, a disparity map, a depth map, etc.) may refer to the size of a single channel of each image, which may be represented by the height and width of the image, for example, may be represented as H×W, where H represents the height of the image, W represents the width of the image, and the units of both may be pixels. Of course, this is just an example, and the size of the image may also be represented by one or more parameters that can reflect the number of pixels, amount of data, storage capacity, or clarity of the image. In addition, it can be understood that the image mentioned in the present disclosure may be a grayscale image or a color image. For a grayscale image, the number of channels is 1, and for a color image, since it may have three color channels of R, G, and B, the number of channels may be 3, that is, the actual size of the color image may be represented as H×W×3. In addition, in the present disclosure, the size of the image to be processed (i.e., the size of the original image that has not been processed by upsampling, etc.) can be determined based on parameters such as the size and number of pixels of the sensor of the multi-eye camera used to capture the image.

[0035] According to some embodiments, the size of each of the multiple image comprehensive features with different sizes obtained by extracting image comprehensive features of different scales from the image to be processed may not be greater than (for example, may be smaller than) the size of the image to be processed. For example, taking the size of the image to be processed as H×W and extracting three different sizes of image comprehensive features as an example, the sizes of the three different sizes of image comprehensive features extracted may be respectively

[0036] According to some embodiments, before fusing the plurality of image comprehensive features with different sizes, the method further includes: performing upsampling and / or dilated convolution processing on the image comprehensive features of non-maximum size among the plurality of image comprehensive features with different sizes, so that the processed size thereof is consistent with the maximum size. For example, if three image comprehensive features of different sizes are still extracted and the sizes of the extracted three image comprehensive features of different sizes are respectively

[0037] According to some embodiments, the operation of extracting image comprehensive features of different scales from the image to be processed can be implemented based on multiple convolution layers in a CNN (Convolutional Neural Network). Extracting image features (such as the first image comprehensive features) based on a convolutional neural network can obtain a larger perception domain, thereby making the extracted image features more accurate and more efficient.

[0038] Correspondingly, the operation of fusing the plurality of image comprehensive features of different sizes, and the operation of upsampling and / or dilated convolution processing the image comprehensive features of non-maximum size among the plurality of image comprehensive features of different sizes, etc., can also be implemented by the fusion processing layer (e.g., add layer), upsampling layer (e.g., upsample layer) and / or dilated convolution processing layer in the corresponding convolutional neural network, respectively. It is understandable that the number of convolutional layers and / or upsampling layers in the convolutional neural network can be set according to the actual processing accuracy requirements and / or speed requirements.

[0039] For example, the following will be combined with Figure 2 Table 1 and Table 1 schematically illustrate the structure of a first convolutional neural network 200 for extracting comprehensive image features of an image to be processed.

[0040] Table 1: Description of a possible network structure of the first convolutional neural network 200

[0043] like Figure 2As shown in Table 1, the first convolutional neural network 200 may include 7 convolutional layers layer1 to layer7, 2 upsample layers and an add layer. The input image, i.e., the image to be processed (which may be represented as input img, and although the input image is schematically illustrated by taking a color image as an example, it can be understood that the input image may also be a single-channel grayscale image) may obtain an image comprehensive feature of a first size (e.g., 1 / 4 size) after being processed by layer1 to layer3, and the image comprehensive feature of the first size may obtain an image comprehensive feature of a second size (e.g., 1 / 8 size) after being further processed by layer4 and layer5, and the image comprehensive feature of the second size may obtain an image comprehensive feature of a third size (e.g., 1 / 16 size) after being further processed by layer6 and layer7. In addition, the image comprehensive features of the second size (e.g., 1 / 8 size) output by layer5 and the image comprehensive features of the third size (e.g., 1 / 16 size) output by layer7 can be fused (e.g., superimposed) with the image comprehensive features of the first size (e.g., 1 / 4 size) output by layer3 through the add layer after corresponding upsampling processing (e.g., upsampling processing to 1 / 4 size) to obtain the desired first image comprehensive features (which can be expressed as imgFeat).

[0044] It is understandable that although Figure 2 In the example shown in Table 1, the first image comprehensive features extracted to obtain a 1 / 4 size are used as an example for explanation. According to the actual processing accuracy requirements and / or speed requirements, the first image comprehensive features extracted can also be other sizes, such as full size, 1 / 2 size, 1 / 8 size, 1 / 16 size, etc. Correspondingly, the number of convolutional layers of the first convolutional neural network can also be adjusted accordingly.

[0045] In addition, it can be seen from Table 1 that each convolution layer can be a module (Convbn) composed of conv2d (two-dimensional convolution), batch normalization (batch normalization) and relu (linear rectifier function) activation functions, and the features output by each convolution layer, such as the extracted image comprehensive features of multiple sizes, can have the same or different number of channels. For example, the image comprehensive features of the first size output by layer3 and the image comprehensive features of the second size output by layer5 can have different numbers of channels (for example, 32 and 64 respectively), the image comprehensive features of the second size output by layer5 and the image comprehensive features of the third size output by layer7 can have the same number of channels (for example, both can be 64, etc.), and so on. In addition, it can be known from the relevant technology that the number of channels of the features output by each convolution layer is related to the number of convolution kernels of the corresponding convolution layer, and the convolution kernels of each convolution layer can be 1×1, 3×3 convolution kernels, etc.

[0046] In addition, as mentioned above, Figure 2 The structure of the first convolutional neural network 200 for extracting the comprehensive features of the first image given in Table 1 is only an example of network design. In fact, the number of layers of the first convolutional neural network 200, the number of features of each layer, and the composition of the convolutional modules of each convolutional layer, etc. can all be changed. In addition, in network design, the method of redundancy first and then simplification can also be followed, that is, the network can be designed to be relatively large and have better abstraction capabilities, and then it can be streamlined according to the trained network, and unimportant network layer branches can be removed to obtain the required first convolutional neural network.

[0047] According to some embodiments, after extracting the first image comprehensive feature, boundary feature extraction is performed based on the extracted first image comprehensive feature and the image to be processed itself to obtain the boundary feature of the image to be processed, which may include: extracting boundary features of different scales for the first image comprehensive feature to obtain a plurality of boundary features of different sizes for the first image comprehensive feature; splicing a portion of the plurality of boundary features of different sizes with a second image comprehensive feature obtained by performing image comprehensive feature extraction on the image to be processed itself to obtain an image mixed feature; and splicing another portion of the plurality of boundary features of different sizes with a boundary feature obtained by performing boundary feature extraction on the image mixed feature to obtain the boundary feature of the image to be processed.

[0048] When extracting boundary features, not only data of different scales from the high level (such as the extracted comprehensive image features of the image to be processed) but also data from the bottom level (such as the image to be processed itself) are considered, thereby achieving the effect of fusing global information to make the acquired boundary features more accurate.

[0049] According to some embodiments, the size of each of the plurality of boundary features of the first image comprehensive feature with different sizes may not be greater than the size of the first image comprehensive feature. For example, taking the first image comprehensive feature with a size of 1 / 4 and two boundary features of different sizes can be extracted for the first image comprehensive feature, the extracted boundary features of the two different sizes may have a size of 1 / 4 and a size of 1 / 8, respectively. It can be understood that the description of the number of size types of the extracted boundary features and the size of each boundary feature here is only an example, and according to the requirements of actual processing accuracy and / or speed, it can also be set to other numbers and other sizes (for example, a combination of the two sizes of 1 / 8 and 1 / 16, a combination of the three sizes of 1 / 4, 1 / 8, and 1 / 16, etc.). In addition, it can also be understood that each of the plurality of boundary features with different sizes extracted may have one or more boundary features of a specific size, that is, the plurality of boundary features with different sizes extracted may also include two or more boundary features of the same size, and there is no limitation on this.

[0050] According to some embodiments, the size of the second image comprehensive feature and the size of the boundary feature obtained by performing boundary feature extraction on the image mixed feature may be no larger than the size of the first image comprehensive feature. By way of example, still taking the example that the first image comprehensive feature has a size of 1 / 4, the extracted second image comprehensive feature and the boundary feature obtained by performing boundary feature extraction on the image mixed feature may both be corresponding sizes of 1 / 4. It is to be understood that the description of the sizes of the corresponding features extracted here is only an example, and according to the actual processing accuracy requirements and / or speed requirements, it can also be set to other sizes (for example, the second image comprehensive feature may be 1 / 4 in size, the boundary feature obtained by performing boundary feature extraction on the image mixed feature may be 1 / 8 in size, or both may be 1 / 8 in size, etc.).

[0051] In addition, it can be understood that before splicing a part of the multiple boundary features with different sizes with the second image comprehensive feature obtained by performing image comprehensive feature extraction on the image to be processed itself, and / or before splicing another part of the multiple boundary features with different sizes with the boundary feature obtained by performing boundary feature extraction on the image mixed feature, the corresponding features with smaller sizes can also be upsampled and / or dilated convolution processed so that their processed size is consistent with the size of other features to be spliced.

[0052] For example, taking the case where a part of the multiple boundary features with different sizes includes boundary features of 1 / 4 size and 1 / 8 size and the size of the second image comprehensive feature is 1 / 4 size, before splicing the part of the multiple boundary features with different sizes with the second image comprehensive feature, the 1 / 8 size boundary feature may be upsampled to 1 / 4 size and then spliced ​​to obtain a 1 / 4 size image mixed feature.

[0053] In addition, it is worth noting that one of the "part" of the multiple boundary features with different sizes and the "other part" of the multiple boundary features with different sizes may only include boundary features of one size, and the sizes of the boundary features included in the two may overlap with each other, as long as the combination of the two can have boundary features of multiple different sizes.

[0054] According to some embodiments, the operations of extracting boundary features of different scales from the first image comprehensive features, extracting image comprehensive features from the image to be processed itself to obtain the second image comprehensive features, and extracting boundary features from the mixed image features can be implemented based on multiple convolutional layers in a convolutional neural network. Extracting image features based on a convolutional neural network can obtain a larger perception domain, thereby making the extracted image features more accurate and more efficient.

[0055] Correspondingly, the splicing processing operations, upsampling processing and / or dilated convolution processing operations mentioned above can also be implemented by the splicing processing layer (e.g., concat layer, which can be abbreviated as cat layer), upsampling layer (e.g., upsample layer) and / or dilated convolution processing layer in the corresponding convolutional neural network, respectively.

[0056] For example, the following will be combined with Figure 3 Table 2 schematically illustrates the structure of a second convolutional neural network 300 for extracting boundary features based on the comprehensive features of the first image and the image to be processed itself.

[0057] Table 2: Description of a possible network structure of the second convolutional neural network 300

[0059] like Figure 3 As can be seen from Table 2, the second convolutional neural network 300 may include 11 convolutional layers layer1 to layer11, 3 upsample layers and 3 concat layers, and may use the output imgFeat of the first convolutional neural network 200 and the image to be processed (which may be represented as input img) as input data. Specifically, after the input img is processed by layer4 and layer5, the corresponding second image comprehensive features (which may have, for example, a 1 / 4 size) may be obtained, and after the imgFeat is processed by layer1, the boundary features of the first size (for example, a 1 / 4 size) may be obtained, and after the boundary features of the first size are processed by layer2, layer3, upsample1 and cat1, the boundary features of the second size may be obtained. The boundary features of the second size can also obtain two other boundary features of the first size (for example, a 1 / 4 size) after being processed by layer8, or upsample3 and layer10, respectively. Here, the boundary features of the first size (e.g., 1 / 4 size) output by layer1, the boundary features of the second size (e.g., 1 / 8 size) output by layer6, and the boundary features of the first size (e.g., 1 / 4 size) output by layer8 and layer10 can constitute the aforementioned plurality of boundary features of different sizes. In addition, the boundary features of the second size (e.g., 1 / 8 size) output by layer6 can be spliced ​​with the boundary features of the first size (e.g., 1 / 4 size) output by layer1 and the second image comprehensive features of the first size (e.g., 1 / 4 size) output by layer5 at the cat2 layer after upsampling (e.g., upsampling to 1 / 4 size) to obtain a feature mixed with basic image features, wherein the boundary features of the second size output by layer6 and the boundary features of the first size output by layer1 can constitute a part of the aforementioned plurality of boundary features of different sizes. After being processed by layer7 and layer9, the mixed feature (which may have a size of 1 / 4) can be spliced ​​with the boundary features of the first size (for example, the size of 1 / 4) output by layer8 and layer10 (wherein the boundary features of the first size output by layer8 and layer10 may constitute another part of the multiple boundary features of different sizes mentioned above) at the cat3 layer, and the desired boundary feature (which may be represented as output edge) can be obtained after being processed by layer11.

[0060] In addition, it can be seen from Table 2 that each convolution layer can be a module (Convbn) composed of conv2d, batch normalization and relu activation functions, and the features output by each convolution layer, such as the extracted multiple boundary features of different sizes, can have the same or different number of channels. For example, the boundary features output by layer1 and the boundary features output by layer6 can have different numbers of channels (for example, 32 and 96 respectively), the boundary features output by layer8 and the boundary features output by layer10 can have the same number of channels (for example, both can be 1, etc.), and so on. In addition, it can be known from the relevant technology that the number of channels of the features output by each convolution layer is related to the number of convolution kernels of the corresponding convolution layer, and the convolution kernels of each convolution layer can be 1×1, 3×3 convolution kernels, etc.

[0061] Understandably, Figure 3 The structure of the second convolutional neural network 300 given in Table 2 is only an example of network design. In fact, the number of layers of the second convolutional neural network 300, the number of features of each layer, and the composition of the convolutional modules of each convolutional layer, etc. can all be changed. In network design, the method of redundancy first and then simplification can also be followed, that is, the network can be designed to be relatively large and have better abstraction capabilities, and then it can be simplified according to the trained network, and unimportant network layer branches can be removed to obtain the desired second convolutional neural network.

[0062] In addition, in the present disclosure, the first convolutional neural network 200 and the second convolutional neural network 300 can be constituted together as corresponding extraction subnetworks. Figure 4 A convolutional neural network 400 for obtaining boundary features of an image to be processed is shown. For example, FIG. 5A to FIG. 5D As shown, based on Figure 4 The convolutional neural network 400 shown in FIG. Figure 5A and Figure 5B When the image to be processed shown in FIG. 1 (although a grayscale image is used as an example for schematic illustration, as mentioned above, the image to be processed may be a color image) is processed, the following can be obtained: Figure 5C and Figure 5D The clearer boundary features are shown. In addition, the experiment shows that Figure 4 The parameters of the convolutional neural network 400 shown occupy less resources, which can be much lower than the size of CNN-based edge detection networks in related technologies, such as HED (holistically-nested edge detection) networks, RCF (Richer Convolutional Features) networks, etc., and the extraction efficiency and extraction effect are also higher.

[0063] It is understood that although the present disclosure is based on Figure 4 The process of obtaining the boundary features of the image to be processed described in step S101 is explained by taking the convolutional neural network shown in the figure as an example to extract boundary features. In the present disclosure, other methods can also be used to obtain the boundary features of the image to be processed. For example, the boundary features of the image to be processed can be obtained by extracting boundary features using a CNN-based edge detection network in the related art.

[0064] After the boundary features of the image to be processed are acquired in step S101, the triangulation processing operation with boundary restrictions (i.e., boundary constraints) described in step S102 can be performed. According to some embodiments, based on the acquired boundary features of the image to be processed, the image to be processed is subjected to triangulation processing with boundary restrictions to obtain a triangulation result that fits the boundary, which may include: determining a set of support points in the image to be processed that fits the boundary based on the acquired boundary features of the image to be processed; and based on the determined set of support points, triangulating the image to be processed to obtain a plurality of triangular facets that fit the boundary, wherein the vertices of each triangular facet in the plurality of triangular facets are support points in the set of support points.

[0065] In other words, according to the acquired boundary features of the image to be processed, a triangulation processing operation limited by the boundary (i.e., bounded by the boundary) can be performed on the image to be processed, so that a triangulation result that is more closely aligned with the boundary can be obtained compared to the ordinary triangulation processing operation in the related art. The triangulation result that is closely aligned with the boundary can make the parallax or depth in each triangular facet obtained smoother and more continuous, which is more conducive to the subsequent parallax or depth densification processing and improves the accuracy of the final parallax map or depth map.

[0066] According to some embodiments, determining a set of support points in the image to be processed that are aligned with the boundary based on the acquired boundary features of the image to be processed may include: selecting pixel points from the image to be processed as support points to obtain the set of support points by sampling at an interval set distance, wherein selecting pixel points from the image to be processed as support points includes: selecting support points from the boundary features of the image to be processed (for example, from the boundary represented by the boundary features); or, performing feature point detection on the image to be processed, and using the detected feature points as support points to obtain the set of support points, wherein performing feature point detection on the image to be processed includes: performing feature point detection on the boundary features of the image to be processed.

[0067] Therefore, by considering selecting the pixel points in the extracted boundary features as support points, a set of support points that better fits the boundary can be obtained, thereby making the triangulation processing result more accurate and more conducive to subsequent parallax or depth densification processing.

[0068] According to some embodiments, when determining the support points by adopting the interval setting distance sampling method, the appropriate distance value can be selected according to the size of the image to be processed and the actual required processing speed and / or accuracy. For example, along the specified direction of the image to be processed, such as the horizontal, vertical or diagonal direction, etc., the corresponding pixel points can be selected as support points every 1 to 3 pixels or 3 to 5 pixels, etc. It can be understood that the number of support points can determine the number of triangular facets obtained by triangulation, and the number of triangular facets can determine the complexity of the algorithm required for parallax or depth densification processing, the processing speed and processing accuracy, etc., and can therefore be reasonably set according to actual needs.

[0069] According to some embodiments, when determining the support points by performing feature point detection on the image to be processed, various feature point detection algorithms in related technologies may be used to complete the process. Typical feature point detection algorithms may include, for example, SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), FAST (Features From Accelerated Segment Test), BRIEF (Binary Robust Independent Elementary Features), ORB (Oriented FAST and Rotated BRIEF), Sobel filtering, etc. Feature point detection algorithms may also include algorithms in current technologies that use deep learning to obtain feature points and their feature descriptions, or various other algorithms in future technologies for detecting feature points. In order not to obscure the subject matter claimed for protection in the present disclosure, the details of these algorithms are not described in detail herein.

[0070] It is understandable that the aforementioned method of determining the set of support points in the image to be processed is more suitable for the scene where the image to be processed is a non-parallax image or a non-depth image (for example, the image to be processed is the original image for the target scene). When the image to be processed is a parallax image or a depth image, since the pixels contained therein already have corresponding parallax values ​​or depth values, the pixels that are closer to the boundary among the pixels with corresponding parallax values ​​or depth values ​​can be directly used as support points based on the consideration of boundary features to obtain the support point set.

[0071] According to some embodiments, based on the triangulation result, the image to be processed is subjected to densification calculation processing related to disparity or depth to obtain a densified disparity map or depth map corresponding to the image to be processed, including: for each of the multiple triangular facets, obtaining the disparity value or depth value of the vertices of the triangular facet; and based on the disparity value or depth value of the vertices of the triangular facet, determining the disparity value or depth value of at least one other point in the triangular facet, so as to obtain a densified disparity map or depth map corresponding to the image to be processed.

[0072] According to some embodiments, when the image to be processed is a non-disparity image or a non-depth image (for example, when the image to be processed is an original image for a target scene), the disparity value or depth value of the vertex of the triangular patch can be calculated based on the feature descriptor matching method. According to some embodiments, calculating the disparity value or depth value of the vertex of the triangular patch based on the feature descriptor matching method may include: determining the feature descriptor of the vertex of the triangular patch and the feature descriptor of the candidate matching point in the target image of the image to be processed (that is, the image with another perspective corresponding to the image to be processed); matching the feature descriptor of the vertex with the feature descriptor of the candidate matching point in the target image to determine the feature descriptor in the target image that can successfully match the feature descriptor of the vertex; and determining the disparity value or depth value of the vertex based on the displacement difference (for example, horizontal displacement difference) between the position of the candidate matching point (the candidate matching point is the actual matching point of the vertex) corresponding to the feature descriptor in the target image that can successfully match the feature descriptor of the vertex and the position of the vertex in the image to be processed. It can be understood that the disparity value of the vertex can be directly calculated based on the displacement difference between the vertex and the matching point, and the depth value of the vertex can be calculated based on the calculated disparity value according to the relationship between the depth value and the disparity value (the relationship is related to the parameters of the multi-view camera used for image acquisition, such as focal length, etc.).

[0073] In the present disclosure, various algorithms for generating feature descriptors in the related art can be used to determine the feature descriptors of vertices and / or candidate matching points, including but not limited to SIFT algorithm, SURF algorithm, etc. In addition, various methods for matching feature descriptors in the related art can be used to determine whether the feature descriptors in the corresponding image pair can be matched successfully, including but not limited to threshold-based methods (i.e., if the distance between feature descriptors is less than a certain threshold, they can be considered to match each other), nearest neighbor-based methods (i.e., if feature descriptor b is the nearest neighbor of feature descriptor a, and the distance between the two is less than a certain threshold, they can be considered to match each other), etc. In addition, in order to improve the accuracy of matching, various other criteria or algorithms in the related art can also be used in combination to filter out feature descriptors or matching points that are incorrectly matched, for example, filtering can be performed by left-right-check (left-right consistency check), filtering can be performed by comparing the cost of the best matching point with the second-best matching point, etc. In order not to obscure the subject matter claimed for protection in the present disclosure, the details of these algorithms are not described in detail here.

[0074] It is understandable that in the present disclosure, when the image to be processed is a disparity image or a depth image, since some of the pixels contained therein already have corresponding disparity values ​​or depth values, and when selecting support points, the pixels with corresponding disparity values ​​or depth values ​​that are closer to the boundary can be directly used as support points based on the boundary features. Therefore, the disparity values ​​or depth values ​​of these support points can be directly obtained from the corresponding disparity image or depth image without the need for additional calculation. In addition, although the present disclosure uses feature descriptor matching to illustrate the calculation process of the disparity, the present disclosure is not limited to using other methods in the related art to calculate the disparity of vertices.

[0075] According to some embodiments, determining the disparity value or depth value of at least one other point in the triangular patch based on the disparity values ​​or depth values ​​of the vertices of the triangular patch may include: calculating parameter values ​​of the plane equation for the triangular patch based on the disparity values ​​or depth values ​​of the vertices of the triangular patch and the positions of the vertices of the triangular patch in the image to be processed to determine the plane equation for the triangular patch; and determining the disparity value or depth value of at least one other point in the triangular patch based on the determined plane equation for the triangular patch and the position of the at least one other point in the triangular patch in the image to be processed.

[0076] In other words, the disparity or depth of each non-vertex in the triangle can be calculated based on the smoothness of the disparity or depth in the triangle. For example, taking the calculation of disparity as an example, if the disparity in the same triangle is considered to be smooth, the disparity of the points in the triangle will conform to the following plane equation: d=au+bv+c; wherein the parameters a, b, c are the parameters that control the plane equation, u, v are the positions of the pixels, and d is the disparity of the pixels. Since a triangle can have three support points (vertices), and the u, v, and d of these three support points are all known, the specific values ​​of the parameters a, b, and c can be calculated by solving a set of linear equations based on the known u, v, and d of the three support points, thereby determining the plane equation for the triangle, and then determining the disparity of other points in the triangle based on the positions of other points in the triangle to achieve densification of the disparity.

[0077] According to some other embodiments, determining the disparity value or depth value of at least one other point in the triangular facet based on the disparity values ​​or depth values ​​of the vertices of the triangular facet may include: calculating the disparity value or depth value of at least one other point in the triangular facet using maximum a posteriori probability estimation based on the disparity values ​​or depth values ​​of the vertices of the triangular facet.

[0078] According to some embodiments, taking the calculation of disparity as an example, using maximum a posteriori probability estimation to calculate the disparity value of at least one other point in the triangular patch may include: determining the disparity prior of the image to be processed; determining the image likelihood of the target image based on the image to be processed and its disparity according to the disparity values ​​of the vertices of the triangular patch in the image to be processed; determining the product of the disparity prior of the image to be processed and the image likelihood of the target image; and determining the optimal solution of the product to calculate the disparity value of at least one other point in the triangular patch.

[0079] For example, taking the calculation of disparity as an example, the disparity value of at least one other point in the triangle patch may be calculated based on the following maximum a posteriori probability estimation calculation formula:

[0080] It is understandable that, although the present disclosure uses the above two triangular interpolation methods to illustrate how to determine the disparity value or depth value of at least one other point in a triangular facet according to the disparity value or depth value of the vertex of the triangular facet, the present disclosure is not limited to using other triangular interpolation methods in the related art to calculate the disparity value or depth value of at least one other point in the triangular facet. For example, the median or average value of the disparity value or depth value of the three vertex pixels of the triangular facet can also be used as the disparity value or depth value of other points in the triangular facet, and so on.

[0081] In addition, in the present disclosure, the triangulation process can be a Delaunay triangulation process. Compared with other triangulation methods, the Delaunay triangulation method has many advantages, including but not limited to: (1) Closest: A triangle is formed by the three nearest points, and the line segments (sides of the triangle) do not intersect; (2) Uniqueness: No matter where the construction starts in the region, a consistent result will be obtained in the end; (3) Optimality: If the diagonals of the convex quadrilateral formed by any two adjacent triangles can be interchanged, then the smallest angle of the six internal angles of the two triangles will not become larger; (4) Most regular: If the smallest angle of each triangle in the triangulation network is arranged in ascending order, the value obtained by the arrangement of the Delaunay triangulation network is the largest; (5) Regionality: When adding, deleting, or moving a vertex, it will only affect the adjacent triangles, etc.

[0082] Furthermore, it is to be understood that, in addition to the Delaunay triangulation method, the present disclosure is not limited to using other triangulation methods in the related art to perform boundary-restricted triangulation processing on the image to be processed.

[0083] The exemplary method according to the present disclosure has been described above in conjunction with the accompanying drawings. Below, exemplary embodiments of exemplary devices, electronic devices, etc. of the present disclosure will be further described in conjunction with the accompanying drawings.

[0084] According to another aspect of the present disclosure, a chip circuit is provided, comprising: a circuit unit configured to execute the steps of the method described in the present disclosure.

[0085] According to another aspect of the present disclosure, an image processing device is provided, comprising: the chip circuit described in the present disclosure. In addition, the image processing device may also include a camera (eg, a multi-view camera) configured to acquire images.

[0086] According to another aspect of the present disclosure, an electronic device is provided, including: a processor; and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the electronic device executes the method described in the present disclosure.

[0087] According to another aspect of the present disclosure, a computer-readable storage medium storing a program is provided. The program includes instructions, which, when executed by a processor of an electronic device, cause the electronic device to execute the method described in the present disclosure.

[0088] As used herein, the term "circuit" may refer to a portion of or include an application specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and / or memory (shared, dedicated, or group) that executes one or more software or firmware programs, a combinational logic circuit that provides the functionality, and / or other suitable hardware components. In some embodiments, the circuit or the functionality associated with the circuit may be implemented by one or more software or firmware modules. In some embodiments, the circuit may include logic that is at least partially operable in hardware. The embodiments described herein may be implemented as a system using any appropriately configured hardware and / or software.

[0089] Reference Figure 6 , a computing device 600 will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The computing device 600 can be any machine configured to perform processing and / or calculations, and can be, but is not limited to, a workstation, a server, a desktop computer, a laptop computer, a tablet computer, a personal digital assistant, a smart phone, a vehicle-mounted computer, or any combination thereof. The above-mentioned image processing apparatus or electronic device can be implemented in whole or in part by the computing device 600 or a similar device or system.

[0090] The computing device 600 may include elements connected to or in communication with the bus 602 (possibly via one or more interfaces). For example, the computing device 600 may include a bus 602, one or more processors 604, one or more input devices 606, and one or more output devices 608. The one or more processors 604 may be any type of processor and may include, but are not limited to, one or more general-purpose processors and / or one or more special-purpose processors (e.g., special processing chips). The input device 606 may be any type of device capable of inputting information to the computing device 600 and may include, but are not limited to, a mouse, a keyboard, a touch screen, a microphone, and / or a remote controller. The output device 608 may be any type of device capable of presenting information and may include, but are not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The computing device 600 may also include or be connected to a storage device 610, which may be any storage device that is non-temporary and can enable data storage, and may include but is not limited to a disk drive, an optical storage device, a solid-state memory, a floppy disk, a flexible disk, a hard disk, a magnetic tape or any other magnetic medium, an optical disk or any other optical medium, a ROM (read-only memory), a RAM (random access memory), a cache memory and / or any other memory chip or cartridge, and / or any other medium from which a computer can read data, instructions and / or code. The storage device 610 may be detachable from the interface. The storage device 610 may have data / programs (including instructions) / code for implementing the above methods and steps. The computing device 600 may also include a communication device 612. The communication device 612 may be any type of device or system that enables communication with an external device and / or with a network, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication device and / or a chipset, such as Bluetooth. TM devices, 1302.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0091] The computing device 600 may also include a working memory 614, which may be any type of working memory that may store programs (including instructions) and / or data useful for the operation of the processor 604, and may include, but is not limited to, random access memory and / or read-only memory devices.

[0092] Software elements (programs) may be located in the working memory 614, including but not limited to an operating system 616, one or more application programs 618, drivers, and / or other data and code. Instructions for performing the above methods and steps may be included in one or more application programs 618. The executable code or source code of the instructions of the software elements (programs) may be stored in a non-transitory computer-readable storage medium (such as the storage device 610 described above) and may be stored in the working memory 614 (possibly compiled and / or installed) when executed. The executable code or source code of the instructions of the software elements (programs) may also be downloaded from a remote location.

[0093] It should also be understood that various modifications may be made according to specific requirements. For example, custom hardware may also be used, and / or specific elements may be implemented in hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. For example, some or all of the disclosed methods and apparatus may be implemented by programming hardware (e.g., programmable logic circuits including field programmable gate arrays (FPGAs) and / or programmable logic arrays (PLAs)) in assembly language or hardware programming languages ​​(such as VERILOG, VHDL, C++) using logic and algorithms according to the present disclosure.

[0094] It should also be understood that the foregoing method can be implemented through a server-client model. For example, the client can receive data input by the user and send the data to the server. The client can also receive data input by the user, perform a portion of the processing in the foregoing method, and send the processed data to the server. The server can receive data from the client, and execute the foregoing method or another portion of the foregoing method, and return the execution result to the client. The client can receive the execution result of the method from the server, and present it to the user, for example, through an output device.

[0095] It should also be understood that the components of computing device 600 can be distributed across a network. For example, some processing can be performed using one processor while other processing can be performed by another processor remote from the one processor. Other components of computing system 600 can also be similarly distributed. In this way, computing device 600 can be interpreted as a distributed computing system that performs processing at multiple locations.

[0096] Some exemplary aspects of the disclosure are described below.

[0097] Aspect 1. An image processing method, comprising:

[0098] Obtaining boundary features of the image to be processed;

[0099] Based on the acquired boundary features of the image to be processed, performing triangulation processing with boundary restrictions on the image to be processed to obtain a triangulation result that fits the boundary; and

[0100] According to the triangulation result, a densification calculation process related to disparity or depth is performed on the image to be processed to obtain a densified disparity map or depth map corresponding to the image to be processed.

[0101] Aspect 2. A method as described in Aspect 1, wherein the image to be processed is any image in a pair of images of a target scene captured by a multi-camera.

[0102] Aspect 3. A method as described in Aspect 1, wherein the image to be processed is a disparity image or a depth image obtained by performing disparity or depth calculation processing on an image pair of a target scene acquired by a multi-camera.

[0103] Aspect 4. The method according to aspect 3, wherein the disparity image or the depth image is a disparity image or a depth image obtained based on a pair of feature points in the pair of images.

[0104] Aspect 5. The method according to any one of aspects 1 to 4, wherein the step of obtaining the boundary features of the image to be processed comprises:

[0105] Extracting comprehensive image features from the image to be processed to obtain a first comprehensive image feature of the image to be processed; and

[0106] Boundary features are extracted based on the extracted first image comprehensive features and the image to be processed itself to obtain boundary features of the image to be processed.

[0107] Aspect 6. The method according to aspect 5, wherein the step of extracting the image comprehensive feature of the image to be processed to obtain the first image comprehensive feature of the image to be processed comprises:

[0108] Extracting comprehensive image features of different scales from the image to be processed to obtain a plurality of comprehensive image features of different sizes for the image to be processed; and

[0109] The plurality of image comprehensive features with different sizes are fused to obtain the first image comprehensive feature.

[0110] Aspect 7. The method according to aspect 6, wherein before fusing the plurality of image comprehensive features of different sizes, the method further comprises:

[0111] Upsampling processing and / or dilation convolution processing are performed on the image comprehensive features of non-maximum size among the multiple image comprehensive features of different sizes, so that the processed size thereof is consistent with the maximum size.

[0112] Aspect 8. The method as described in Aspect 6, wherein the size of each of the plurality of image comprehensive features having different sizes is not greater than the size of the image to be processed.

[0113] Aspect 9. A method as described in Aspect 6, wherein the operation of extracting comprehensive image features of different scales from the image to be processed is implemented based on multiple convolutional layers in a convolutional neural network.

[0114] Aspect 10. The method according to aspect 5, wherein the step of extracting boundary features based on the extracted first image comprehensive features and the image to be processed itself to obtain the boundary features of the image to be processed comprises:

[0115] Extracting boundary features of different scales from the first image comprehensive features to obtain a plurality of boundary features of different sizes from the first image comprehensive features;

[0116] splicing a part of the plurality of boundary features with different sizes with a second image comprehensive feature obtained by extracting image comprehensive features from the image to be processed itself, so as to obtain an image mixed feature; and

[0117] Another part of the plurality of boundary features with different sizes is spliced ​​with the boundary features obtained by performing boundary feature extraction on the image mixed features to obtain boundary features of the image to be processed.

[0118] Aspect 11. The method according to Aspect 10, wherein the size of each boundary feature of the plurality of boundary features having different sizes of the first image comprehensive feature is not greater than the size of the first image comprehensive feature.

[0119] Aspect 12. A method as described in Aspect 10, wherein the size of the second image comprehensive feature and the size of the boundary feature obtained by performing boundary feature extraction on the image mixed feature are not greater than the size of the first image comprehensive feature.

[0120] Aspect 13. A method as described in Aspect 10, wherein the operation of extracting boundary features of different scales on the first image comprehensive features, the operation of extracting image comprehensive features on the image to be processed itself to obtain second image comprehensive features, and the operation of extracting boundary features on the mixed image features are implemented based on multiple convolutional layers in a convolutional neural network.

[0121] Aspect 14. The method according to any one of aspects 1 to 4, wherein, based on the acquired boundary features of the image to be processed, performing triangulation processing with boundary restrictions on the image to be processed to obtain a triangulation result that fits the boundary, comprises:

[0122] Determining a set of support points in the image to be processed that fits the boundary based on the acquired boundary features of the image to be processed; and

[0123] According to the determined support point set, the image to be processed is triangulated to obtain a plurality of triangular facets that fit the boundary, wherein the vertex of each triangular facet in the plurality of triangular facets is a support point in the support point set.

[0124] Aspect 15. The method according to aspect 14, wherein determining a set of support points in the image to be processed that fits the boundary based on the acquired boundary features of the image to be processed comprises:

[0125] Adopting an interval setting distance sampling method, selecting pixel points from the image to be processed as support points to obtain the support point set, wherein selecting pixel points from the image to be processed as support points includes: selecting support points from the boundary features of the image to be processed; or,

[0126] Feature point detection is performed on the image to be processed, and the detected feature points are used as support points to obtain the support point set, wherein the feature point detection on the image to be processed includes: feature point detection on the boundary features of the image to be processed.

[0127] Aspect 16. The method according to aspect 14, wherein, according to the triangulation result, performing a densification calculation process related to disparity or depth on the image to be processed to obtain a densified disparity map or depth map corresponding to the image to be processed, comprising:

[0128] For each triangular facet, obtaining a disparity value or a depth value of a vertex of the triangular facet; and

[0129] According to the disparity values ​​or depth values ​​of the vertices of the triangular facet, the disparity value or depth value of at least one other point in the triangular facet is determined to obtain a densified disparity map or depth map corresponding to the image to be processed.

[0130] Aspect 17. The method according to aspect 16, wherein, for the image to be processed which is a non-disparity image or a non-depth image, the disparity value or the depth value of the vertices of the triangular facet is calculated by:

[0131] Determine feature descriptors of the vertices of the triangular face and feature descriptors of candidate matching points in a target image of the image to be processed;

[0132] Matching the feature descriptor of the vertex with the feature descriptor of the candidate matching point in the target image to determine a feature descriptor in the target image that can successfully match the feature descriptor of the vertex; and

[0133] The disparity value or depth value of the vertex is determined based on the displacement difference between the position of the candidate matching point in the target image corresponding to the feature descriptor in the target image that can successfully match the feature descriptor of the vertex and the position of the vertex in the image to be processed.

[0134] Aspect 18. The method according to aspect 16, wherein determining the disparity value or depth value of at least one other point in the triangular facet according to the disparity value or depth value of the vertex of the triangular facet comprises:

[0135] Calculating parameter values ​​of a plane equation for the triangular patch according to the disparity values ​​or depth values ​​of the vertices of the triangular patch and the positions of the vertices of the triangular patch in the image to be processed to determine the plane equation for the triangular patch; and

[0136] The disparity value or depth value of at least one other point in the triangular patch is determined according to the determined plane equation for the triangular patch and the position of the at least one other point in the triangular patch in the image to be processed.

[0137] Aspect 19. The method according to aspect 16, wherein determining the disparity value or depth value of at least one other point in the triangular facet according to the disparity value or depth value of the vertex of the triangular facet comprises:

[0138] According to the disparity values ​​or depth values ​​of the vertices of the triangular patch, the disparity value or depth value of at least one other point in the triangular patch is calculated by using maximum a posteriori probability estimation.

[0139] Aspect 20. The method according to any one of Aspects 1 to 4, wherein the triangulation process is a Delaunay triangulation process.

[0140] Aspect 21. A chip circuit comprising:

[0141] A circuit unit configured to perform the steps of the method according to any one of aspects 1-20.

[0142] Aspect 22. An image processing device, comprising the chip circuit as described in Aspect 21.

[0143] Aspect 23. An electronic device comprising:

[0144] Processor; and

[0145] A memory storing a program, the program comprising instructions which, when executed by the processor, cause the electronic device to perform a method according to any one of aspects 1-20.

[0146] Aspect 24. A computer-readable storage medium storing a program, the program comprising instructions, which, when executed by a processor of an electronic device, cause the electronic device to perform a method according to any one of aspects 1-20.

[0147] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. An image processing method, comprising: Acquire boundary features of the image to be processed, wherein the boundary features of the image to be processed include features that can reflect boundary information of an object in the image to be processed, wherein acquiring the boundary features of the image to be processed includes: Using a first convolutional neural network, extracting image comprehensive features of the image to be processed to obtain a first image comprehensive feature of the image to be processed, including: extracting image comprehensive features of different scales of the image to be processed to obtain a plurality of image comprehensive features of different sizes of the image to be processed; and fusing the plurality of image comprehensive features of different sizes to obtain the first image comprehensive feature; and Using a second convolutional neural network, extracting boundary features based on the extracted first image comprehensive features and the image to be processed itself to obtain boundary features of the image to be processed, including: extracting boundary features of different scales on the first image comprehensive features to obtain a plurality of boundary features of different sizes of the first image comprehensive features; splicing a portion of the plurality of boundary features of different sizes with a second image comprehensive feature obtained by performing image comprehensive feature extraction on the image to be processed itself to obtain an image mixed feature; and splicing another portion of the plurality of boundary features of different sizes with a boundary feature obtained by performing boundary feature extraction on the image mixed feature to obtain the boundary features of the image to be processed; Based on the acquired boundary features of the image to be processed, performing triangulation processing with boundary restrictions on the image to be processed to obtain a triangulation result that fits the boundary; and According to the triangulation result, a densification calculation process related to disparity or depth is performed on the image to be processed to obtain a densified disparity map or depth map corresponding to the image to be processed.

2. The method of claim 1, wherein: The image to be processed is any image in an image pair for a target scene collected by a multi-camera.

3. The method of claim 1, wherein: The image to be processed is a disparity image or a depth image obtained by performing disparity or depth calculation processing on an image pair for a target scene collected by a multi-view camera.

4. The method of claim 3, wherein: The disparity image or the depth image is a disparity image or a depth image obtained based on a pair of feature points in the pair of images.

5. The method of claim 1, wherein: Before fusing the plurality of image comprehensive features of different sizes, the method further includes: Upsampling processing and / or dilation convolution processing are performed on the image comprehensive features of non-maximum size among the multiple image comprehensive features of different sizes, so that the processed size thereof is consistent with the maximum size.

6. The method of claim 1, wherein: The size of each of the multiple image comprehensive features with different sizes is not greater than the size of the image to be processed.

7. The method of claim 1, wherein: The operation of extracting comprehensive image features of different scales from the image to be processed is achieved based on multiple convolutional layers in a convolutional neural network.

8. The method of claim 1, wherein: The size of each of the plurality of boundary features with different sizes of the first image comprehensive feature is not greater than the size of the first image comprehensive feature.

9. The method of claim 1, wherein: The size of the second image comprehensive feature and the size of the boundary feature obtained by performing boundary feature extraction on the image mixed feature are not greater than the size of the first image comprehensive feature.

10. The method of claim 1, wherein: The operations of extracting boundary features of different scales on the first image comprehensive features, extracting image comprehensive features on the image to be processed itself to obtain second image comprehensive features, and extracting boundary features on the image mixed features are implemented based on multiple convolutional layers in a convolutional neural network.

11. The method according to any one of claims 1 to 4, wherein: Based on the acquired boundary features of the image to be processed, performing triangulation processing with boundary restrictions on the image to be processed to obtain a triangulation result that fits the boundary, including: Determining a set of support points in the image to be processed that fits the boundary based on the acquired boundary features of the image to be processed; and According to the determined support point set, the image to be processed is triangulated to obtain a plurality of triangular facets that fit the boundary, wherein the vertex of each triangular facet in the plurality of triangular facets is a support point in the support point set.

12. The method of claim 11, wherein: Determining a set of support points in the image to be processed that fit the boundary based on the acquired boundary features of the image to be processed includes: Adopting an interval setting distance sampling method, selecting pixel points from the image to be processed as support points to obtain the support point set, wherein selecting pixel points from the image to be processed as support points includes: selecting support points from the boundary features of the image to be processed; or, Feature point detection is performed on the image to be processed, and the detected feature points are used as support points to obtain the support point set, wherein the feature point detection on the image to be processed includes: feature point detection on the boundary features of the image to be processed.

13. The method of claim 11, wherein: According to the triangulation result, performing a densification calculation process related to disparity or depth on the image to be processed to obtain a densified disparity map or depth map corresponding to the image to be processed, including: For each triangular facet, obtaining a disparity value or a depth value of a vertex of the triangular facet; and According to the disparity values ​​or depth values ​​of the vertices of the triangular facet, the disparity value or depth value of at least one other point in the triangular facet is determined to obtain a densified disparity map or depth map corresponding to the image to be processed.

14. The method of claim 13, wherein: For the image to be processed that is a non-disparity image or a non-depth image, the disparity value or depth value of the vertices of the triangular facet is calculated in the following manner: Determine feature descriptors of the vertices of the triangular face and feature descriptors of candidate matching points in the target image of the image to be processed; Matching the feature descriptor of the vertex with the feature descriptor of the candidate matching point in the target image to determine a feature descriptor in the target image that can successfully match the feature descriptor of the vertex; as well as The disparity value or depth value of the vertex is determined based on the displacement difference between the position of the candidate matching point in the target image corresponding to the feature descriptor in the target image that can successfully match the feature descriptor of the vertex and the position of the vertex in the image to be processed.

15. The method of claim 13, wherein: The step of determining the disparity value or depth value of at least one other point in the triangular facet according to the disparity value or depth value of the vertex of the triangular facet comprises: According to the disparity values ​​or depth values ​​of the vertices of the triangular patch and the positions of the vertices of the triangular patch in the image to be processed, the parameter values ​​of the plane equation for the triangular patch are calculated to determine the plane equation for the triangular patch; and according to the determined plane equation for the triangular patch and the position of at least one other point in the triangular patch in the image to be processed, the disparity value or depth value of the at least one other point is determined.

16. The method of claim 13, wherein: The step of determining the disparity value or depth value of at least one other point in the triangular facet according to the disparity value or depth value of the vertex of the triangular facet comprises: According to the disparity values ​​or depth values ​​of the vertices of the triangular patch, the disparity value or depth value of at least one other point in the triangular patch is calculated by using maximum a posteriori probability estimation.

17. The method according to any one of claims 1 to 4, wherein: The triangulation process is a Delaunay triangulation process.

18. A chip circuit, comprising: A circuit unit configured to perform the steps of the method according to any one of claims 1-17.

19. An image processing device, comprising the chip circuit according to claim 18.

20. An electronic device, comprising: processor; as well as A memory storing a program, the program comprising instructions, which when executed by the processor cause the electronic device to perform the method according to any one of claims 1-17.

21. A computer-readable storage medium storing a program, the program comprising instructions, which, when executed by a processor of an electronic device, cause the electronic device to perform the method according to any one of claims 1 to 17.

22. A computer program product comprising program code portions for executing the method according to any one of claims 1 to 17 when the computer program product is executed on one or more computing devices.

Images