Binocular parallax image restoration method, readable storage medium and equipment
By training a disparity map inpainting model and using multimodal feature fusion and adversarial training with holed disparity maps and original image information, the problem of poor repair of holed regions in traditional disparity map inpainting is solved, and the accuracy of disparity map inpainting and the effect of 3D reconstruction are improved.
Patent Information
- Application Number
- CN202511402788.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-28
- Publication Date
- 2026-01-09
AI Technical Summary
Traditional stereo matching algorithms produce serious mismatches in real-world measurement scenarios, resulting in gaps in the disparity map. Existing disparity map repair methods use simple image filling, leading to poor repair results.
A pre-trained disparity map repair model is used to repair the hole region by utilizing information from the hole disparity map and the original camera image. Adversarial training is performed through a combination of generator, feature extraction module, multimodal feature fusion and discriminator to complete the repair of the hole region.
It improves the accuracy and effectiveness of disparity map inpainting, solves the problem of poor inpainting caused by simple image filling, and achieves higher quality 3D reconstruction.
Smart Images
Figure CN121304497A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a binocular parallax map repair method, a readable storage medium, and a device, belonging to the field of image processing technology. Background Technology
[0002] The process of acquiring disparity in binocular stereo vision mainly relies on the stereo matching algorithm of binocular vision. However, in actual measurement scenarios, traditional stereo matching algorithms often produce serious mismatches due to factors such as occlusion between objects in the scene, reflections on object surfaces, and a large number of weak texture areas on object surfaces. This results in disparity maps with many hole areas. Traditional disparity map repair methods mainly rely on multi-layer mean filtering and univariate quadratic interpolation to perform simple image filling, which greatly reduces the accuracy of the obtained disparity map and leads to poor final 3D reconstruction results. Summary of the Invention
[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a binocular disparity map repair method, readable storage medium and device, which can simultaneously utilize information from the hole disparity map and the original camera image to repair holes, improve the repair progress, and solve the problem that the current disparity map repair method uses a simple image filling method to "hard repair" the disparity map, resulting in poor repair effect.
[0004] To solve the above-mentioned technical problems, the present invention is implemented using the following technical solution: This invention provides a method for repairing binocular disparity maps, comprising: Obtain the hole disparity map A of the target scene, the mask image A corresponding to the hole region in the hole disparity map A, and the original image A of any one of the binocular images of the target scene; Input the hole disparity map A, the mask image A, and the original image A into the pre-trained disparity map insulation model to obtain the complete disparity map of the target scene; The training method for the disparity map restoration model includes: Obtain disparity maps B for multiple random scenes, mask images B corresponding to the multiple disparity maps B, and the original image B of any one of the binocular images corresponding to the multiple disparity maps B; The mask image B covers the corresponding disparity map B, forming a holed disparity map B; A disparity map inpainting model is trained using a holed disparity map B and the original image B, enabling the disparity map inpainting model to completely repair the holed regions in the holed disparity map B.
[0005] Furthermore, the disparity map repair model includes: The generator includes an encoder, a feature extraction module, and a decoder connected in sequence; wherein the encoder is used to perform feature encoding on the input hole disparity map, mask image, and original image; The feature extraction module is used to extract a first initial feature from the holed disparity map and a second initial feature from the original image; The multimodal feature fusion module is used to fuse the first initial feature and the second initial feature to obtain the enhanced feature; A decoder is used to generate a hole-repairing disparity map based on the enhanced features; The discriminator is used to determine whether the hole repair parallax map has been completely repaired.
[0006] Furthermore, the step of training a disparity map inpainting model using a holed disparity map B and the original image B, enabling the disparity map inpainting model to completely repair the holed regions in the holed disparity map B, includes: The first initial feature and the second initial feature are obtained from the hole disparity map B and the original image B, respectively; The first and second initial features are fused to obtain the enhanced features. Based on the enhanced features, a corresponding hole repair disparity map is generated; Adversarial training is conducted based on the hole repair disparity map until the hole regions in the newly generated hole repair disparity map are completely repaired.
[0007] Furthermore, the fusion process of the first initial feature and the second initial feature to obtain the enhanced feature includes: Obtain the attention weights corresponding to the first initial feature and the second initial feature respectively; The attention weights are calculated using the following formula: ; ; This represents the attention weight of the first initial feature. Indicates the first initial feature; This represents the attention weights of the second initial feature. Indicates the second initial feature; This indicates max pooling. Indicates average pooling. This represents a multilayer perceptron. express sigmoid Activation function; The first initial feature and the second initial feature are concatenated based on the obtained attention weights to obtain the first fused feature; The splicing process is represented by the following formula: ; Indicates the first fusion feature; This indicates element-wise multiplication. Indicates feature concatenation operation; The first and second initial features are concatenated to obtain the concatenated features. The splicing process is represented by the following formula: ; Indicates splicing characteristics; Based on the obtained splicing features, spatial attention weights are calculated; Spatial attention weights are calculated using the following formula: ; Indicates spatial attention weights; express The largest feature set, express The average feature set; The spliced features are spliced based on the obtained spatial attention weights to obtain the second fused feature; The splicing process is represented by the following formula: ; Indicates the second fusion feature; The enhanced features are calculated using the following formula: ; This indicates enhanced features.
[0008] Furthermore, the adversarial training based on the hole repair disparity map, until the hole regions in the newly generated hole repair disparity map are completely repaired, includes: Based on the newly generated hole repair disparity map, determine the difference between the newly generated hole repair disparity map and disparity map B; Adversarial training is performed based on the difference between the newly generated hole repair disparity map and disparity map B until the disparity map repair model can repair the hole region.
[0009] Furthermore, determining the difference between the newly generated hole repair disparity map and disparity map B based on the newly generated hole repair disparity map includes: The first training loss is established to represent the difference in generated content between the newly generated hole-repaired disparity map and disparity map B: ; Indicates the first training loss; Indicates the determination of the loss against the opponent. ; This indicates a loss in generated content. ; This represents the newly generated hole repair parallax map; Represents a disparity map; This represents the weighting of the generated content loss. This indicates the output of the discriminator. Indicates the generator output; This represents the L1 norm.
[0010] Furthermore, determining the difference between the newly generated hole repair disparity map and the existing disparity map based on the newly generated hole repair disparity map includes: A second training loss is established to represent the discriminative difference between the newly generated hole-repaired disparity map and the disparity map: ; Indicates the second training loss; This represents the discrimination loss on the disparity map; ; This indicates the discrimination loss of the disparity map for cavity repair; ; This represents the newly generated hole repair parallax map; Represents a disparity map; This indicates the output of the discriminator. This indicates the generator output.
[0011] Furthermore, the adversarial training based on the difference between the newly generated hole-repairing disparity map and the disparity map, until the disparity map inpainting model can repair the hole region, includes: Establish the adversarial training objective function, expressed as: ; This means minimizing the discriminant difference between the newly generated hole repair disparity map and the original disparity map; This means maximizing the difference in generated content between the newly generated hole repair disparity map and the existing disparity map; This represents the objective function with the discriminator and generator as variables; The discriminator represents the disparity map restoration model; Represents the generator of the disparity map restoration model; Indicates the discriminant loss for the disparity map: ; This indicates the output of the discriminator. Represents a disparity map; This indicates the discriminant loss for the disparity map of cavity repair: ; This represents the generator output. This represents the newly generated hole repair parallax map; Indicates the disparity map dataset Take a disparity map from the dataset, calculate the discrimination loss of the disparity map, and take the average. This represents the newly generated hole repair parallax map. obey The probability distribution is used to calculate the discrimination loss of the disparity map for cavity repair and take the average. = .
[0012] In another aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the binocular disparity map repair method as described above.
[0013] The present invention also provides a computer device, comprising: Memory, used to store instructions; A processor is configured to execute the instructions, causing the device to perform the binocular disparity map repair method as described above.
[0014] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: 1. This invention employs a pre-trained disparity map repair model. It repairs the hole disparity map based on the mask image corresponding to the hole region in the hole disparity map and the original image of any one of the binocular images corresponding to the hole disparity map. This effectively utilizes the information in the hole disparity map and the original camera image, improving the repair progress and solving the problem that current disparity map repair methods use simple image filling to "hard repair" the disparity map, resulting in poor repair effects.
[0015] 2. In the process of training the disparity map repair model, this invention conducts adversarial training based on the difference between the newly generated hole repair disparity map and the disparity map, thereby improving the training effect. Attached Figure Description
[0016] Figure 1 This is a flowchart of a binocular parallax map repair method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the disparity map repair model provided in an embodiment of the present invention; Figure 3 This is a flowchart of the disparity map repair model training process provided in this embodiment of the invention; Figure 4 This is a schematic diagram of the fusion processing procedure provided in an embodiment of the present invention. Detailed Implementation
[0017] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention. Example
[0018] like Figure 1 As shown, a binocular disparity map repair method includes: Obtain the hole disparity map A of the target scene, the mask image A corresponding to the hole region in the hole disparity map A, and the original image A of any one of the binocular images of the target scene; Input the hole disparity map A, the mask image A and the original image A into the pre-trained disparity map repair model to repair the hole regions in the hole disparity map A and obtain the complete disparity map of the target scene. like Figure 2 As shown, the disparity map repair model includes: The generator includes an encoder, a feature extraction module, and a decoder connected in sequence; wherein the encoder includes three convolutional layers connected in sequence for feature encoding of the input hole disparity map, mask image, and original image; The feature extraction module includes four stacked and interconnected AOT (Aggregated Contextual Transformation) modules; used to extract the first initial feature from the holed disparity map and the second initial feature from the original image; The multimodal feature fusion module is used to fuse the first initial feature and the second initial feature to obtain the enhanced feature; The decoder consists of three convolutional layers connected in sequence. Each layer of the decoder embeds a multimodal feature fusion module to generate a hole-repairing disparity map based on the enhanced features. The discriminator is used to determine whether the hole repair parallax map has been completely repaired.
[0019] Training methods for disparity map restoration models include: Obtain disparity maps B for multiple random scenes, mask images B corresponding to the multiple disparity maps B, and the original image B of any one of the binocular images corresponding to the multiple disparity maps B; It should be noted that in this example, the original images used are all left-eye images from the binocular images; The mask image B covers the corresponding disparity map B, forming a holed disparity map B; like Figure 3 As shown, a disparity map inpainting model is trained using a holed disparity map B and the original image B, enabling the model to completely repair the holed regions in the holed disparity map B. Specifically: like Figure 4 As shown, the first initial feature and the second initial feature are obtained from the hole disparity map B and the original image B, respectively; The first and second initial features are fused together, and the enhanced features are calculated using the following formula: Obtain the attention weights corresponding to the first initial feature and the second initial feature respectively; The attention weights are calculated using the following formula: ; ; This represents the attention weight of the first initial feature. Indicates the first initial feature; This represents the attention weights of the second initial feature. Indicates the second initial feature; This indicates max pooling. Indicates average pooling. This represents a multilayer perceptron (MLP). express sigmoid Activation function; The first initial feature and the second initial feature are concatenated based on the obtained attention weights to obtain the first fused feature; The splicing process is represented by the following formula: ; Indicates the first fusion feature; This represents element-wise multiplication, as shown in the diagram. ; This represents the feature concatenation operation, as shown in the figure. ; The first and second initial features are concatenated to obtain the concatenated features. The splicing process is represented by the following formula: ; Indicates splicing characteristics; Based on the obtained splicing features, spatial attention weights are calculated; Spatial attention weights are calculated using the following formula: ; Indicates spatial attention weights; express The largest feature set, express The average feature set; The spliced features are spliced based on the obtained spatial attention weights to obtain the second fused feature; The splicing process is represented by the following formula: ; Indicates the second fusion feature; The enhanced features are calculated using the following formula: ; Indicates enhanced features; Based on the enhanced features, a corresponding hole repair disparity map is generated.
[0020] Adversarial training is performed based on the hole repair disparity map until the hole regions in the newly generated hole repair disparity map are completely repaired, including: Based on the newly generated hole repair disparity map, determine the differences between the newly generated hole repair disparity map and disparity map B, specifically: The first training loss is established to represent the difference in generated content between the newly generated hole-repaired disparity map and disparity map B: ; Indicates the first training loss; Indicates the determination of the loss against the opponent. ; This indicates a loss in generated content. ; This represents the newly generated hole repair parallax map; Represents a disparity map; This represents the weighting of the generated content loss. This represents the discriminator of the disparity map restoration model. This indicates the output of the discriminator; This represents the generator of the disparity map repair model. Indicates the generator output; express L 1-norm; It should be noted that the newly generated hole repair parallax map is... The evaluation is performed by a discriminator with fixed parameters, and the discriminator outputs... Then calculate the countermeasure loss. ; Among them, if the generator generates a hole repair disparity map If the quality is poor, such as discontinuity in the parallax of the cavity repair area, the discriminator outputs... Approaching 0, combating losses It will increase significantly; The generator received Then, calculate the loss of the generator's own generated content. .
[0021] A second training loss is established to represent the discriminative difference between the newly generated hole-repaired disparity map and the disparity map: ; Indicates the second training loss; This represents the discrimination loss on the disparity map; ; This indicates the discrimination loss of the disparity map for cavity repair; ; Adversarial training is performed based on the difference between the newly generated hole-repairing disparity map and disparity map B until the disparity map insulation model can repair the hole region. Specifically: Establish an adversarial training objective function so that the adversarial training results satisfy the objective function; The adversarial training objective function is expressed as: ; This means minimizing the discriminant difference between the newly generated hole repair disparity map and the original disparity map; This means maximizing the difference in generated content between the newly generated hole repair disparity map and the existing disparity map; This represents the objective function with the discriminator and generator as variables; Indicates the disparity map dataset Take a disparity map from the dataset, calculate the discrimination loss of the disparity map, and take the average. This represents the newly generated hole repair parallax map. obey The probability distribution is used to calculate the discrimination loss of the disparity map for cavity repair and take the average. = .
[0022] Example 2 A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following methods: Obtain the hole disparity map A of the target scene, the mask image A corresponding to the hole region in the hole disparity map A, and the original image A of any one of the binocular images of the target scene; Input the hole disparity map A, the mask image A, and the original image A into the pre-trained disparity map insulation model to obtain the complete disparity map of the target scene; The training methods for the disparity map restoration model include: Obtain disparity maps B for multiple random scenes, mask images B corresponding to the multiple disparity maps B, and the original image B of any one of the binocular images corresponding to the multiple disparity maps B; The mask image B covers the corresponding disparity map B, forming a holed disparity map B; A disparity map inpainting model is trained using a holed disparity map B and the original image B, enabling the disparity map inpainting model to completely repair the holed regions in the holed disparity map B.
[0023] Example 3 A computer device, comprising: Memory, used to store instructions; A processor is configured to execute the instructions, causing the device to perform the following method: Obtain the hole disparity map A of the target scene, the mask image A corresponding to the hole region in the hole disparity map A, and the original image A of any one of the binocular images of the target scene; Input the hole disparity map A, the mask image A, and the original image A into the pre-trained disparity map insulation model to obtain the complete disparity map of the target scene; The training methods for the disparity map restoration model include: Obtain disparity maps B for multiple random scenes, mask images B corresponding to the multiple disparity maps B, and the original image B of any one of the binocular images corresponding to the multiple disparity maps B; The mask image B covers the corresponding disparity map B, forming a holed disparity map B; A disparity map inpainting model is trained using a holed disparity map B and the original image B, enabling the disparity map inpainting model to completely repair the holed regions in the holed disparity map B.
[0024] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage CD-ROMs, optical storage, etc.) containing computer-usable program code.
[0025] This application is described with reference to flowchart illustrations of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each step in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the process. Figure 1 One or more processes or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0026] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 The function specified in one or more processes.
[0027] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 Steps of a specified function in one or more processes.
[0028] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.
Claims
1. A method for repairing binocular disparity maps, characterized in that, include: Obtain the hole disparity map A of the target scene, the mask image A corresponding to the hole region in the hole disparity map A, and the original image A of any one of the binocular images of the target scene; Input the hole disparity map A, the mask image A, and the original image A into the pre-trained disparity map insulation model to obtain the complete disparity map of the target scene; The training method for the disparity map restoration model includes: Obtain disparity maps B for multiple random scenes, mask images B corresponding to the multiple disparity maps B, and the original image B of any one of the binocular images corresponding to the multiple disparity maps B; The mask image B covers the corresponding disparity map B, forming a holed disparity map B; A disparity map inpainting model is trained using a holed disparity map B and the original image B, enabling the disparity map inpainting model to completely repair the holed regions in the holed disparity map B.
2. The binocular disparity map repair method according to claim 1, characterized in that, The disparity map restoration model includes: The generator includes an encoder, a feature extraction module, and a decoder connected in sequence; wherein the encoder is used to perform feature encoding on the input hole disparity map, mask image, and original image; The feature extraction module is used to extract a first initial feature from the holed disparity map and a second initial feature from the original image; The multimodal feature fusion module is used to fuse the first initial feature and the second initial feature to obtain the enhanced feature; A decoder is used to generate a hole-repairing disparity map based on the enhanced features; The discriminator is used to determine whether the hole repair parallax map has been completely repaired.
3. The binocular disparity map repair method according to claim 1, characterized in that, The disparity map inpainting model is trained using a holed disparity map B and the original image B, enabling the model to completely repair the holed regions in the holed disparity map B, including: The first initial feature and the second initial feature are obtained from the hole disparity map B and the original image B, respectively; The first and second initial features are fused to obtain the enhanced features. Based on the enhanced features, a corresponding hole repair disparity map is generated; Adversarial training is conducted based on the hole repair disparity map until the hole regions in the newly generated hole repair disparity map are completely repaired.
4. The binocular disparity map repair method according to claim 3, characterized in that, The process of fusing the first initial feature and the second initial feature to obtain the enhanced feature includes: Obtain the attention weights corresponding to the first initial feature and the second initial feature respectively; The attention weights are calculated using the following formula: ； ； This represents the attention weight of the first initial feature. Indicates the first initial feature; This represents the attention weights of the second initial feature. Indicates the second initial feature; This indicates max pooling. Indicates average pooling. This represents a multilayer perceptron. express sigmoid Activation function; The first initial feature and the second initial feature are concatenated based on the obtained attention weights to obtain the first fused feature; The splicing process is represented by the following formula: ； Indicates the first fusion feature; This indicates element-wise multiplication. Indicates feature concatenation operation; The first and second initial features are concatenated to obtain the concatenated features. The splicing process is represented by the following formula: ； Indicates splicing characteristics; Based on the obtained splicing features, spatial attention weights are calculated; Spatial attention weights are calculated using the following formula: ； Indicates spatial attention weights; express The largest feature set, express The average feature set; The spliced features are spliced based on the obtained spatial attention weights to obtain the second fused feature; The splicing process is represented by the following formula: ； Indicates the second fusion feature; The enhanced features are calculated using the following formula: ； This indicates enhanced features.
5. The binocular disparity map repair method according to claim 3, characterized in that, The adversarial training based on the hole repair disparity map, until the hole regions in the newly generated hole repair disparity map are completely repaired, includes: Based on the newly generated hole repair disparity map, determine the difference between the newly generated hole repair disparity map and disparity map B; Adversarial training is performed based on the difference between the newly generated hole repair disparity map and disparity map B until the disparity map repair model can repair the hole region.
6. The binocular disparity map repair method according to claim 5, characterized in that, The step of determining the difference between the newly generated hole repair disparity map and disparity map B based on the newly generated hole repair disparity map includes: The first training loss is established to represent the difference in generated content between the newly generated hole-repaired disparity map and disparity map B: ； Indicates the first training loss; Indicates the determination of the loss against the opponent. ; This indicates a loss in generated content. ; This represents the newly generated hole repair parallax map; Represents a disparity map; This represents the weighting of the generated content loss. This indicates the output of the discriminator. Indicates the generator output; This represents the L1 norm.
7. The binocular disparity map repair method according to claim 5, characterized in that, The step of determining the difference between the newly generated hole repair disparity map and the existing disparity map based on the newly generated hole repair disparity map includes: A second training loss is established to represent the discriminative difference between the newly generated hole-repaired disparity map and the disparity map: ； Indicates the second training loss; This represents the discrimination loss on the disparity map; ； This indicates the discrimination loss of the disparity map for cavity repair; ； This represents the newly generated hole repair parallax map; Represents a disparity map; This indicates the output of the discriminator. This indicates the generator output.
8. The binocular disparity map repair method according to claim 5, characterized in that, The adversarial training based on the difference between the newly generated hole repair disparity map and the disparity map continues until the disparity map repair model can repair the hole region, including: Establish the adversarial training objective function, expressed as: ； This means minimizing the discriminant difference between the newly generated hole repair disparity map and the original disparity map; This means maximizing the difference in generated content between the newly generated hole repair disparity map and the existing disparity map; This represents the objective function with the discriminator and generator as variables; The discriminator represents the disparity map restoration model; Represents the generator of the disparity map restoration model; Indicates the discriminant loss for the disparity map: ； This indicates the output of the discriminator. Represents a disparity map; This indicates the discriminant loss for the disparity map of cavity repair: ； This represents the generator output. This represents the newly generated hole repair parallax map; Indicates the disparity map dataset Take a disparity map from the dataset, calculate the discrimination loss of the disparity map, and take the average. This represents the newly generated hole repair parallax map. obey The probability distribution is used to calculate the discrimination loss of the disparity map for cavity repair and take the average. = .
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the binocular disparity map repair method as described in any one of claims 1 to 8.
10. A computer device, characterized in that, include: Memory, used to store instructions; A processor for executing the instructions, causing the device to perform the binocular disparity map repair method as described in any one of claims 1 to 8.