A multi-view image depth prediction method and system based on probability volume analysis

Abstract

The present invention discloses a multi-view image depth prediction method and system based on probability volume analysis, which are designed to adopt a multi-scale cascade structure model; uniformly sample the input depth range; then implement an adaptive depth optimization strategy, calculate the adaptive depth optimization interval through single-peak curve fitting, and sample the depth hypothesis; construct feature bodies of each source perspective, aggregate the feature bodies into cost bodies; form the cost bodies into probability bodies, and store the confidence corresponding to each hypothesized depth; in view of the fuzzy single-peak problem caused in actual scenes, split the probability body into a main probability body and a secondary probability body as input, and calculate two groups of depth hypotheses respectively; then obtain new main probability bodies and secondary probability bodies respectively, and merge them to obtain the probability body of the current scale; finally obtain a depth map; iterate at the second scale and the third scale; perform sampling of depth hypotheses and formation of probability bodies at the fourth scale, and finally obtain a depth map with the same resolution as the input image.

Description

Background Art

[0002] Existing learning-based multi-view stereo models mainly focus on predicting depth maps through a cascade structure to achieve more robust reconstruction. However, they often emphasize improving the quality of stereo matching while ignoring the importance of depth assumptions.

[0003] In general, most current methods focus on improving the quality of depth map prediction by optimizing feature extraction networks, introducing viewpoint attention, etc. to obtain more reliable matching costs, or constraining the model by improving loss functions, etc. Few research methods focus on probability bodies, which are a key link in hypothesis-verification. They store the probability of each depth hypothesis and are usually used to directly calculate depth maps or calculate training losses.

[0004] Based on this, a multi-view image depth prediction method based on probability volume analysis is studied and developed. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a multi-view image depth prediction method and system based on probability volume analysis in view of the deficiencies in the above-mentioned prior art, so as to solve the technical problem of improving the prediction quality of multi-view depth maps based on deep learning.

[0006] The present invention adopts the following technical solutions: A multi-view image depth prediction method based on probability volume analysis comprises the following steps: S1. Design a multi-scale cascade structure model; S2. In the initial stage, the input depth range is uniformly sampled; then, based on the probability volume generated at the previous scale, an adaptive depth optimization strategy is implemented, and the adaptive depth optimization interval is calculated by unimodal curve fitting to sample the depth hypothesis; S3, construct feature bodies of each source view, and then aggregate the feature bodies into cost bodies; the cost bodies are formed into probability bodies, storing the confidence corresponding to each hypothesized depth; S4. To solve the fuzzy single peak problem caused in the actual scene, the probability body is split into a main probability body and a secondary probability body; the main probability body and the secondary probability body are used as inputs of step S2 to calculate two sets of depth hypotheses respectively; then step S3 is executed to obtain new main probability bodies and secondary probability bodies respectively, and the probability bodies of the current scale are merged by considering the probability distribution; finally, a depth map is obtained; S5, iterating step S4 at the second scale and the third scale; executing steps S2 and S3 at the fourth scale, and finally obtaining a depth map with the same resolution as the input image.

[0007] Preferably, the input of the multi-scale cascade structure model is a multi-view RGB image of the same scene or object, an internal and external parameter matrix, and a corresponding depth range; a multi-scale feature map corresponding to each input image is extracted using a feature pyramid network with shared weights.

[0008] Preferably, step S2 specifically includes: S201, fitting curves at the first scale and the third scale to estimate the depth prediction intervals at the second scale and the fourth scale; fitting the curve at the second scale as the depth prediction interval at the third scale; S202, first fitting the first scale and the third scale fitting curves, and then adding a first-order term at the exponential position as the final expression; S203, by probability As the probability threshold, we get the refined depth prediction interval and .

[0009] Preferably, the first scale and the third scale fitting curves are:

[0010] The fitted curve in the second scale is:

[0011] in, , represents the assumed depth, Represents the predicted depth obtained by the Soft Argmin function, p for The corresponding probability value, a and b are the parameters to be fitted.

[0012] Preferably, the parameters a and b to be fitted are:

[0013]

[0014] in, is the number of depth hypotheses, Assuming the depth With predicted depth The absolute value of the difference, for The corresponding probability value.

[0015] Preferably, the feature maps of each view, the internal and external parameter matrices and the depth hypothesis are used as input, and the feature bodies of each source view are constructed through differentiable homography transformation and bilinear interpolation, and then the feature bodies are aggregated into cost bodies; the cost bodies are regularized by 3DUNet and activated using functions to form probability bodies, which store the confidence corresponding to each hypothesized depth.

[0016] Preferably, in step S4, the probability body is a three-dimensional tensor; At the lth scale, the probability body Split into primary probability body and secondary probability body; In the According to the main probability body and the secondary probability body of the lth scale, two sets of deep hypotheses are obtained, and a shared weight network is used to calculate the The main probability body and sub-probability body of the scale; find the depth value index corresponding to the maximum probability; Then, in addition to the depth index left and right adjacent depth, and the probability values ​​of other depths are set to 0, and the main probability volume corresponding to the main peak is obtained. .

[0017] Preferably, if the sum of the secondary peak probability values ​​is less than a threshold α, the mask tensor M is used to mark the pixel positions where the secondary peaks exist.

[0018] Preferably, the corresponding depth optimization interval is calculated based on the main probability body and the secondary probability body and their depth assumptions and , Will comply with And split mask The probability of the pixel location and the depth hypothesis are replaced.

[0019] In a second aspect, an embodiment of the present invention provides a multi-view image depth prediction system based on probability volume analysis, comprising: Design module, the design adopts a multi-scale cascade structure model; The calculation module, in the initial stage, uniformly samples the input depth range; then, based on the probability volume generated at the previous scale, an adaptive depth optimization strategy is implemented, and the adaptive depth optimization interval is calculated through unimodal curve fitting to sample the depth hypothesis; Aggregation module, constructs feature volumes of each source view, and then aggregates the feature volumes into cost volumes; the cost volumes are transformed into probability volumes, storing the confidence corresponding to each hypothesized depth; The splitting module splits the probability body into a main probability body and a secondary probability body to solve the fuzzy single peak problem caused in actual scenarios. The main probability body and the secondary probability body are used as inputs of the calculation module to calculate two sets of depth hypotheses respectively. The aggregation module is then executed to obtain new main probability bodies and secondary probability bodies respectively, and the probability body of the current scale is obtained by merging them considering the probability distribution. Finally, the depth map is obtained. The output module iterates the splitting module at the second scale and the third scale; the calculation module and the aggregation module are executed at the fourth scale, and finally a depth map with the same resolution as the input image is obtained.

[0020] In a third aspect, a computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the multi-view image depth prediction method based on probability volume analysis when executing the computer program.

[0021] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, including a computer program, which, when executed by a processor, implements the steps of the multi-view image depth prediction method based on probability volume analysis.

[0022] In a fifth aspect, a chip comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the multi-view image depth prediction method based on probability volume analysis when executing the computer program.

[0023] In a sixth aspect, an embodiment of the present invention provides an electronic device, including a computer program, which, when executed by the electronic device, implements the steps of the multi-view image depth prediction method based on probability volume analysis.

[0024] Compared with the prior art, the present invention has at least the following beneficial effects: A multi-view image depth prediction method based on probabilistic volume analysis is proposed. An adaptive depth hypothesis method based on unimodal fitting and a probabilistic volume split-merge module are designed to improve the validity and fault tolerance of the depth hypothesis. This method is implemented using a cascade structure, which improves the quality of depth map prediction while maintaining computational efficiency.

[0025] Furthermore, the model uses multi-view RGB images of the same scene or object as the main input, and uses the internal and external parameter matrices and the corresponding depth range as auxiliary information to predict the depth map corresponding to the RGB image. The feature pyramid network is used to extract multi-scale feature maps, so that the depth map can be optimized from coarse to fine in the subsequent process. Dense depth assumptions are used at low resolution to obtain a coarse initial depth map, and sparse depth assumptions are used at high resolution to gradually optimize the initial depth map. This strategy not only conforms to the laws of multi-view geometry but also reduces the network's memory requirements.

[0026] Furthermore, the essence of MVS is a one-dimensional search for corresponding points on the corresponding polar lines of the reference perspective pixel points on the source perspective. For ordered depth hypotheses from small to large, the search gradually approaches the corresponding points and then gradually moves away. This is manifested in the probability body as an obvious unimodal distribution of probability values ​​along the depth dimension, and the larger the probability value, the closer the corresponding hypothetical depth is to the true depth. By analyzing the probability values ​​along the depth dimension, the scope of the depth search can be narrowed. Step S2 fits the "depth-probability value" into corresponding unimodal curves at different scales. Through adaptive calculation of the probability threshold, the depth prediction interval can be more reasonably optimized to improve the effectiveness of the depth hypothesis.

[0027] Furthermore, step S3 provides other necessary processes for depth map prediction, namely, constructing the cost body and regularization processing. Through differentiable homography transformation and bilinear interpolation, as well as aggregation operations, the matching information of multi-view features at different depths is uniformly stored in the cost body. At this time, the feature matching information is multi-dimensional, which improves the robustness of the network model. Since 3DUet has good performance in filtering noise and is memory-friendly, the 3DUet network is then regularized to filter noise while converting the multi-dimensional matching information into a one-dimensional probability value to obtain the confidence of each hypothesized depth.

[0028] Furthermore, step S4 designs a probability body splitting-merging module, which adopts the strategy of splitting the probability body to construct the primary and secondary depth optimization intervals respectively, aiming to deal with the fuzzy single peak phenomenon that may appear in angular edges, weak textures and high-reflection areas, thereby improving the fault tolerance of depth prediction.

[0029] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0030] In summary, from the perspective of probability volume analysis, the present invention proposes an adaptive depth hypothesis method based on unimodal fitting and a probability volume split-merge method to improve the effectiveness and fault tolerance of depth hypothesis in the multi-view depth prediction process based on deep learning, thereby improving the final depth map prediction quality. At the same time, the model is implemented in a cascade structure, maintaining good computational efficiency and memory friendliness.

[0031] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of the present invention; Figure 2 A schematic diagram of a computer device provided by an embodiment of the present invention; Figure 3 A block diagram of a chip provided according to an embodiment of the present invention; Figure 4 The point cloud reconstruction result map on the DTU dataset; Figure 5 Point cloud reconstruction results for the Tanks&Temples dataset. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0035] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0036] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0037] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0038] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0039] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0040] The present invention provides a multi-view image depth prediction method based on probability volume analysis. From the perspective of probability volume analysis, a learning-based multi-view image depth prediction model, called ProbMVSNet, is adopted. Unimodal functions suitable for different scales are designed, and based on depth hypothesis-probability volume parameter fitting, the depth optimization interval is adaptively calculated according to the probability threshold. In addition, considering that matching noise may lead to blurred unimodals, a probability volume split-merge module is introduced to construct the primary and secondary depth optimization intervals, thereby improving the fault tolerance of the model.

[0041] The present invention provides a multi-view image depth prediction method based on probability volume analysis, comprising the following steps: S1. The model adopts a multi-scale cascade structure design, and the input is a multi-view RGB image of the same scene or object, an internal and external parameter matrix, and the corresponding depth range. First, a feature pyramid network with shared weights is used to extract the multi-scale feature map corresponding to each input image. S2, sampling depth assumption; In the initial stage, the input depth range is uniformly sampled. Then, based on the probability volume generated at the previous scale, an adaptive depth optimization strategy is implemented, and the adaptive depth optimization interval is calculated by unimodal curve fitting to sample the depth hypothesis.

[0042] S201. Since the probabilities of the Gaussian curve and the Laplace curve are too concentrated in the center, the expressions are optimized to a certain extent. Finally, according to formula (1), the curves are fitted at the first and third scales to estimate the depth prediction intervals of the second and fourth scales. According to formula (2), the curve is fitted at the second scale as the depth prediction interval of the third scale.

[0043] (1) (2) in, , represents the assumed depth, Represents the predicted depth obtained by the Soft Argmin function, p for The corresponding probability value, a and b are the parameters to be fitted.

[0044] S202. In order to simplify the calculation, equation (1) is first fitted as follows: (3) Then add the first-order term at the exponential position as the final expression; compared with the least squares method in matrix form, the general form of least squares method is simpler to calculate.

[0045] Therefore, according to this method, the curve is fitted; first, take the logarithm of both sides of equation (3) to obtain: (4) The objective function of the least squares method is constructed based on the above formula: (5) in, , represents the i-th hypothesis depth, for The corresponding probability value.

[0046] Let its partial derivative with respect to parameter a be 0, then the value of parameter a is: (6) Similarly, for formula (2), we get: (7) S203, at this time, the probability As the probability threshold, the refined depth prediction interval is obtained.

[0047] For formula (1), let ,have to: (8) Find the root formula, and we get (9) Among them, negative roots have been discarded.

[0048] This results in the next scale depth refinement interval ; Similarly, the depth refinement interval obtained by formula (2) is .

[0049] S3, taking the feature map of each view, the internal and external parameter matrix and the depth hypothesis as input, constructs the feature volume of each source view through differentiable homography transformation and bilinear interpolation, and then aggregates these feature volumes into a cost volume; the cost volume is regularized by 3DUNet and activated by the softmax function to form a probability volume, which stores the confidence corresponding to each hypothesized depth; S4. Design a probability volume splitting module to solve the fuzzy single peak problem caused by noise and other reasons in actual scenes; split the probability volume into a main probability volume and a secondary probability volume. Take the main probability volume and the secondary probability volume as the input of step S2, and calculate two sets of depth hypotheses respectively; execute step S3 to obtain new main probability volume and secondary probability volume respectively, and merge them by considering the probability distribution to obtain the probability volume of the current scale. Finally, calculate the depth map by regression or classification method; The probability volume is a three-dimensional tensor represented as ; At the lth scale, the probability body is split into the main probability body and the secondary probability body; In the We obtain two sets of deep hypotheses based on the main probability body and the secondary probability body of the lth scale, and calculate the probability of the first scale through a network with shared weights. The main probability body and sub-probability body of the scale; find the depth value index corresponding to the maximum probability; Then, in addition to the depth index left and right adjacent depth, and the probability values ​​of other depths are set to 0, thus obtaining the main probability volume corresponding to the main peak . For the secondary peak, the process is the opposite; In addition, if the sum of the secondary peak probability values ​​is less than the threshold α, it is considered that there is no secondary peak. For the subsequent probability volume merging, the mask tensor M is used to mark the pixel positions where the secondary peak exists. It should be noted that since the sum of the probabilities is not 1 at this time, the predicted depth is calculated by the following formula: (10) The main probability volume and the secondary probability volume calculated above are used as inputs of step S2 to calculate two sets of depth hypotheses respectively; step S3 is executed to obtain new main probability volumes and secondary probability volumes respectively, and they are merged by considering the probability distribution to obtain the probability volume of the current scale.

[0050] Specifically, considering that the sharper the probability distribution of the probability volume along the depth dimension, the more accurate the depth prediction is. The corresponding depth optimization interval is calculated based on the main probability volume and the secondary probability volume and their depth assumptions. And split mask The probability of the pixel location and the depth hypothesis are replaced.

[0051] S5, iterating step S4 at the second and third scales; executing steps S2 and S3 at the fourth scale, and finally obtaining a depth map with the same resolution as the input image.

[0052] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0053] In another embodiment of the present invention, a multi-view image depth prediction system based on probability volume analysis is provided. The system can be used to implement the above-mentioned multi-view image depth prediction method based on probability volume analysis. Specifically, the multi-view image depth prediction system based on probability volume analysis includes a design module, a calculation module, an aggregation module, a splitting module and an output module.

[0054] Among them, the design module adopts a multi-scale cascade structure model; The calculation module, in the initial stage, uniformly samples the input depth range; then, based on the probability volume generated at the previous scale, an adaptive depth optimization strategy is implemented, and the adaptive depth optimization interval is calculated through unimodal curve fitting to sample the depth hypothesis; Aggregation module, constructs feature volumes of each source view, and then aggregates the feature volumes into cost volumes; the cost volumes are transformed into probability volumes, storing the confidence corresponding to each hypothesized depth; The splitting module splits the probability body into a main probability body and a secondary probability body to solve the fuzzy single peak problem caused in actual scenarios. The main probability body and the secondary probability body are used as inputs of the calculation module to calculate two sets of depth hypotheses respectively. The aggregation module is then executed to obtain new main probability bodies and secondary probability bodies respectively, and the probability body of the current scale is obtained by merging them considering the probability distribution. Finally, the depth map is obtained. The output module iterates the splitting module at the second scale and the third scale; the calculation module and the aggregation module are executed at the fourth scale, and finally a depth map with the same resolution as the input image is obtained.

[0055] In another embodiment of the present invention, a terminal device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the multi-view image depth prediction method based on probability volume analysis, including: The design adopts a multi-scale cascade structure model; in the initial stage, the input depth range is uniformly sampled; then based on the probability body generated at the previous scale, an adaptive depth optimization strategy is implemented, and the adaptive depth optimization interval is calculated through unimodal curve fitting to sample the depth hypothesis; construct the feature body of each source perspective, and then aggregate the feature body into a cost body; the cost body is formed into a probability body, and the confidence corresponding to each hypothesized depth is stored; in order to solve the fuzzy unimodal problem caused in the actual scene, the probability body is split into a main probability body and a secondary probability body; the main probability body and the secondary probability body are used as inputs to calculate two sets of depth hypotheses respectively; then new main probability bodies and secondary probability bodies are obtained respectively, and the probability body of the current scale is obtained by merging them considering the probability distribution; finally, a depth map is obtained; iterate at the second and third scales; at the fourth scale, sampling of depth hypotheses and formation of probability bodies are performed, and finally a depth map with the same resolution as the input image is obtained.

[0056] See also Figure 2 , the terminal device is a computer device, and the computer device 60 of this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the computer program 63 is executed by the processor 61, the multi-view image depth prediction method based on probability volume analysis in the embodiment is implemented. To avoid repetition, it is not described one by one here. Alternatively, when the computer program 63 is executed by the processor 61, the functions of each model / unit in the multi-view image depth prediction system based on probability volume analysis in the embodiment are implemented. To avoid repetition, it is not described one by one here.

[0057] The computer device 60 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will appreciate that Figure 2 This is only an example of the computer device 60 and does not constitute a limitation of the computer device 60. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0058] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0059] The memory 62 may be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 may also be an external storage device of the computer device 60, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer device 60.

[0060] Furthermore, the memory 62 may include both an internal storage unit of the computer device 60 and an external storage device. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 may also be used to temporarily store data that has been output or is to be output.

[0061] See also Figure 3 , the terminal device is a chip, and the chip 600 of this embodiment includes a processor 622, which may be one or more, and a memory 632 for storing a computer program executable by the processor 622. The computer program stored in the memory 632 may include one or more modules, each corresponding to a set of instructions. In addition, the processor 622 may be configured to execute the computer program to perform the above-mentioned multi-view image depth prediction method based on probability volume analysis.

[0062] In addition, the chip 600 may further include a power supply component 626 and a communication component 650, wherein the power supply component 626 may be configured to perform power management of the chip 600, and the communication component 650 may be configured to implement communication, for example, wired or wireless communication, of the chip 600. In addition, the chip 600 may further include an input / output interface 658. The chip 600 may operate based on an operating system stored in the memory 632.

[0063] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs. It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory (Non-VolatileMemory), such as at least one disk memory.

[0064] One or more instructions stored in a computer-readable storage medium may be loaded and executed by a processor to implement corresponding steps of the multi-view image depth prediction method based on probability volume analysis in the above embodiment; one or more instructions in the computer-readable storage medium are loaded and executed by the processor as follows: The design adopts a multi-scale cascade structure model; in the initial stage, the input depth range is uniformly sampled; then based on the probability body generated at the previous scale, an adaptive depth optimization strategy is implemented, and the adaptive depth optimization interval is calculated through unimodal curve fitting to sample the depth hypothesis; construct the feature body of each source perspective, and then aggregate the feature body into a cost body; the cost body is formed into a probability body, and the confidence corresponding to each hypothesized depth is stored; in order to solve the fuzzy unimodal problem caused in the actual scene, the probability body is split into a main probability body and a secondary probability body; the main probability body and the secondary probability body are used as inputs to calculate two sets of depth hypotheses respectively; then new main probability bodies and secondary probability bodies are obtained respectively, and the probability body of the current scale is obtained by merging them considering the probability distribution; finally, a depth map is obtained; iterate at the second and third scales; at the fourth scale, sampling of depth hypotheses and formation of probability bodies are performed, and finally a depth map with the same resolution as the input image is obtained.

[0065] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0066] The present invention performs model training and testing on the public datasets DTU dataset and Tanks&Temples dataset. After predicting the depth map, it is fused into a 3D point cloud model through post-processing. Figure 4 and Figure 5 The point cloud reconstruction results on the DTU dataset and Tanks&Temples dataset are shown respectively.

[0067] In summary, the present invention provides a multi-view image depth prediction method and system based on probability body analysis. By designing a unimodal function suitable for different scales and fitting the depth hypothesis-probability body parameters, the depth optimization interval is adaptively calculated to improve the prediction accuracy of the depth map. Considering the fuzzy unimodal problem caused by matching noise, a probability body split-merge module is introduced to construct the primary and secondary depth optimization intervals to improve the fault tolerance of the model. The above method is implemented on the basic cascade structure, which significantly improves the performance of depth map prediction.

[0068] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0069] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0070] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0071] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0072] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0073] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0074] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0075] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0076] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0077] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0078] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A multi-view image depth prediction method based on probability volume analysis, characterized in that: The following steps are involved: S1. Design a multi-scale cascade structure model; S2, in the initial stage, the input depth range is uniformly sampled; Then, based on the probability body generated at the previous scale, an adaptive depth optimization strategy is implemented, and the adaptive depth optimization interval is calculated through single-peak curve fitting to sample the depth hypothesis; S3, construct feature bodies of each source view, and then aggregate the feature bodies into cost bodies; the cost bodies are formed into probability bodies, storing the confidence corresponding to each hypothesized depth; S4. To solve the fuzzy single peak problem caused in the actual scene, the probability body is split into a main probability body and a secondary probability body; the main probability body and the secondary probability body are used as inputs of step S2 to calculate two sets of depth hypotheses respectively; then step S3 is executed to obtain new main probability bodies and secondary probability bodies respectively, and the probability bodies of the current scale are merged by considering the probability distribution; finally, a depth map is obtained; S5, iterating step S4 at the second scale and the third scale; Steps S2 and S3 are performed at the fourth scale to finally obtain a depth map with the same resolution as the input image.

2. The multi-view image depth prediction method based on probability volume analysis according to claim 1, characterized in that: The input of the multi-scale cascade structure model is multi-view RGB images of the same scene or object, internal and external parameter matrices, and the corresponding depth range; a feature pyramid network with shared weights is used to extract the multi-scale feature map corresponding to each input image.

3. The multi-view image depth prediction method based on probability volume analysis according to claim 1, characterized in that: Step S2 is specifically as follows: S201, fitting curves at the first scale and the third scale to estimate the depth prediction intervals at the second scale and the fourth scale; fitting the curve at the second scale as the depth prediction interval at the third scale; S202, first fitting the first scale and the third scale fitting curves, and then adding a first-order term at the exponential position as the final expression; S203, by probability As the probability threshold, we get the refined depth prediction interval and .

4. The multi-view image depth prediction method based on probability volume analysis according to claim 3, characterized in that: The first scale and three scale fitting curves are: The fitted curve in the second scale is: in, , represents the assumed depth, Represents the predicted depth obtained by the Soft Argmin function, p for The corresponding probability value, a and b are the parameters to be fitted.

5. The multi-view image depth prediction method based on probability volume analysis according to claim 4, characterized in that: The parameters a and b to be fitted are: in, is the number of depth hypotheses, Assuming the depth With predicted depth The absolute value of the difference, for The corresponding probability value.

6. The multi-view image depth prediction method based on probability volume analysis according to claim 1, characterized in that: Taking the feature maps of each view, the internal and external parameter matrices and the depth hypothesis as input, the feature volume of each source view is constructed through differentiable homography transformation and bilinear interpolation, and then the feature volume is aggregated into a cost volume; the cost volume is regularized by 3DUNet and activated using a function to form a probability volume, which stores the confidence corresponding to each hypothesized depth.

7. The multi-view image depth prediction method based on probability volume analysis according to claim 1, characterized in that: In step S4, the probability body is a three-dimensional tensor; At the lth scale, the probability body Split into primary probability body and secondary probability body; In the According to the main probability body and the secondary probability body of the lth scale, two sets of deep hypotheses are obtained, and a shared weight network is used to calculate the The main probability body and sub-probability body of the scale; find the depth value index corresponding to the maximum probability; Then, in addition to the depth index left and right adjacent depth, and the probability values ​​of other depths are set to 0, and the main probability volume corresponding to the main peak is obtained. .

8. The multi-view image depth prediction method based on probability volume analysis according to claim 7, characterized in that: If the sum of the secondary peak probability values ​​is less than the threshold α, the mask tensor M is used to mark the pixel position where the secondary peak exists.

9. The multi-view image depth prediction method based on probability volume analysis according to claim 8, characterized in that: Calculate the corresponding depth optimization interval based on the main probability body and the secondary probability body and their depth assumptions and , Will comply with And split mask The probability of the pixel location and the depth hypothesis are replaced.

10. A multi-view image depth prediction system based on probabilistic volume analysis, characterized in that: include: Design module, the design adopts a multi-scale cascade structure model; The computation module,in the initial stage, uniformly samples the input depth range; Then, based on the probability body generated at the previous scale, an adaptive depth optimization strategy is implemented, and the adaptive depth optimization interval is calculated through single-peak curve fitting to sample the depth hypothesis; Aggregation module, constructs feature volumes of each source view, and then aggregates the feature volumes into cost volumes; the cost volumes are transformed into probability volumes, storing the confidence corresponding to each hypothesized depth; The splitting module splits the probability body into a main probability body and a secondary probability body to solve the fuzzy single peak problem caused in actual scenarios. The main probability body and the secondary probability body are used as inputs of the calculation module to calculate two sets of depth hypotheses respectively. The aggregation module is then executed to obtain new main probability bodies and secondary probability bodies respectively, and the probability body of the current scale is obtained by merging them considering the probability distribution. Finally, the depth map is obtained. Output module, iterative splitting module at the second and third scales; The calculation module and aggregation module are executed at the fourth scale, and finally a depth map with the same resolution as the input image is obtained.

Images