A parallax prediction method, apparatus, device and storage medium

Abstract

The application discloses a disparity prediction method and device, equipment and a storage medium, wherein the method comprises: obtaining two groups of RGB images and infrared spectrum images by shooting through a binocular camera composed of two groups of RGB cameras / infrared cameras, and performing feature fusion on the two groups of images respectively to obtain corresponding fusion feature images and corresponding feature map slice sequences; predicting the similarity of slices at the same position in the first feature map slice sequence and the second feature map slice sequence to obtain a similarity sequence, and generating a similarity filtering map according to the similarity sequence; constructing a to-be-filtered 4D cost volume according to the first fusion feature map and the second fusion feature map; filtering the to-be-filtered 4D cost volume according to the similarity filtering map to obtain a filtered 4D cost volume; and generating a disparity regression prediction map according to the filtered 4D cost volume. The application can improve the stability of disparity calculation in a binocular stereo system, reduce the artificial cost in the process, and improve the efficiency.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more particularly to a parallax prediction method, apparatus, device, and storage medium. Background Technology

[0002] Parallax, in a binocular stereo vision system, refers to the difference in position between the projected points of the same point in a scene on the image planes of the two cameras when the same scene is captured by two cameras from different angles. These cameras are typically horizontally positioned to simulate human eyes. For any given point in the scene, the greater the parallax between its projected points in the left and right camera images, the closer that point is to the camera. A disparity map is an important output of binocular stereo vision. It is an image with the size of either of the two camera output images as a reference, and each pixel's value is the disparity value. Specifically, a disparity map is a two-dimensional image where the position of each pixel corresponds to a pixel in the original image, and the value of that pixel represents the disparity value of that point in the left and right camera images. The importance of the disparity map lies in its inclusion of distance information about the distance of various points in a scene or object from the camera. By overlaying multiple disparity maps onto the entire surface of the scanned object, a complete stereo model of the scanned object can be reconstructed, enabling precise stereoscopic measurements of the object's size and volume, or other industrial quality inspection operations.

[0003] Therefore, efficiently reconstructing local stereo modeling of actual industrial scenes / objects is the foundation and key to rapidly reconstructing the overall model and using the overall model for further operations. However, traditional computer vision-based binocular stereo modeling often uses image feature-based left and right view disparity calculation methods. Its corresponding pixel matching often relies on manually designed image features and neighborhood statistics methods, which are difficult to apply to various situations such as complex lighting environments, reflective surfaces, dim lighting, and surfaces without texture features. The stability is low, and the manual cost is high and the efficiency is low, which greatly restricts the application and development of binocular stereo reconstruction technology. Summary of the Invention

[0004] This invention provides a disparity prediction method, apparatus, device, and storage medium, aiming to improve the stability of disparity calculation in binocular stereo systems, reduce labor costs in the process, and improve efficiency.

[0005] In a first aspect, embodiments of the present invention provide a disparity prediction method, the method comprising:

[0006] A first image group and a second image group obtained by capturing the target object with a preset binocular camera are acquired, and feature fusion is performed on the first image group and the second image group respectively to obtain a first fused feature image and a second fused feature image, as well as a first feature image slice sequence corresponding to the first fused feature image and a second feature image slice sequence corresponding to the second fused feature image. Each group of cameras in the binocular camera includes an RGB camera and an infrared camera. Both the first image group and the second image group include RGB images and infrared spectral images. The slices at the same position in the first feature image slice sequence and the second feature image slice sequence are of equal size.

[0007] The similarity of slices at the same position in the first feature map slice sequence and the second feature map slice sequence is predicted to obtain a similarity sequence, and a similarity filter map is generated based on the similarity sequence.

[0008] Construct the 4D cost body to be filtered based on the first and second fused feature maps;

[0009] The 4D cost volume to be filtered is filtered according to the similarity filtering graph to obtain the filtered 4D cost volume.

[0010] A disparity regression prediction map is generated based on the filtered 4D cost volume. The value of each pixel in the disparity regression prediction map is the predicted disparity value corresponding to the corresponding pixel in the reference image. The reference image is any image in the first image group and the second image group.

[0011] In a second aspect, embodiments of the present invention provide a disparity prediction device, the disparity prediction device comprising:

[0012] The feature fusion unit is used to acquire a first image group and a second image group obtained by a preset binocular camera capturing a target object, and to perform feature fusion on the first image group and the second image group respectively to obtain a first fused feature image and a second fused feature image, as well as a first feature image slice sequence corresponding to the first fused feature image and a second feature image slice sequence corresponding to the second fused feature image. Each group of cameras in the binocular camera includes an RGB camera and an infrared camera. The first image group and the second image group both include RGB images and infrared spectral images. The slices at the same position in the first feature image slice sequence and the second feature image slice sequence are of equal size.

[0013] The similarity prediction unit is used to predict the similarity between slices at the same position in the first feature map slice sequence and the second feature map slice sequence, obtain a similarity sequence, and generate a similarity filter map based on the similarity sequence.

[0014] The cost body construction unit is used to construct the 4D cost body to be filtered based on the first fused feature map and the second fused feature map.

[0015] The filtering unit is used to filter the 4D cost volume to be filtered according to the similarity filtering map to obtain the filtered 4D cost volume.

[0016] The prediction unit is used to generate a disparity regression prediction map based on the filtered 4D cost volume. The value of each pixel in the disparity regression prediction map is the predicted disparity value corresponding to the corresponding pixel in the reference image. The reference image is any one of the first image group and the second image group.

[0017] Thirdly, embodiments of the present invention also provide a disparity prediction device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of any disparity prediction method provided in the embodiments of the present invention.

[0018] Fourthly, embodiments of the present invention also provide a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to cause the electronic device to perform the steps of any of the disparity prediction methods provided in the embodiments of the present invention.

[0019] The beneficial effects of this invention are as follows:

[0020] This invention utilizes a binocular camera system consisting of two sets of RGB cameras and two sets of infrared cameras to capture two sets of RGB images and infrared spectral images. Features from these two sets of images are then fused to obtain corresponding fused feature images and a sequence of feature map slices. By fusing the features of both RGB visible light and infrared spectra, a more accurate, reliable, and stable image output is achieved compared to a single RGB image or infrared spectral image, thus improving the accuracy and reliability of subsequent similarity matching and disparity prediction. Next, this invention predicts the similarity of slices at the same position in the first and second feature map slice sequences to obtain a similarity sequence. A similarity filter map is then generated based on this similarity sequence for filtering the 4D cost volume constructed from the first and second fused feature maps. This suppresses spatial redundancy information in the 4D cost volume and enhances stereo matching-related information. Furthermore, this invention generates an accurate and reliable disparity regression prediction map based on the filtered 4D cost volume. Since this disparity regression prediction map is based on the fused RGB and infrared spectral features and the prediction process does not rely on manual design, this invention improves the stability of disparity calculation in binocular stereo systems, reduces manual costs, and increases efficiency. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the left and right views and parallax diagram taken by a binocular stereo vision system;

[0023] Figure 2 This is a schematic diagram illustrating the principle of parallax and depth calculation for the left and right cameras in a binocular stereo vision system.

[0024] Figure 3 This is a schematic diagram illustrating the principle of 3D object reconstruction.

[0025] Figure 4 This is a schematic diagram of a 3D scan and local spatial modeling of the inner wall of a wine vat;

[0026] Figure 5 This is a flowchart illustrating one embodiment of the disparity prediction method provided in this invention.

[0027] Figure 6 This is a rendering of the active infrared spectral stereo scanning effect in an embodiment of the present invention;

[0028] Figure 7 This is a flowchart illustrating one embodiment of the disparity prediction method provided in this invention.

[0029] Figure 8 This is a schematic diagram of the disparity prediction device provided in the embodiments of the present invention;

[0030] Figure 9 This is a schematic diagram of the disparity prediction device provided in an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Furthermore, in the description of the embodiments of the present invention, the terms "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance. Thus, features defined with "first" or "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present invention, "multiple" means two or more, unless otherwise explicitly specified.

[0032] This invention provides a parallax prediction method, apparatus, device, and storage medium.

[0033] Specifically, this embodiment will be described from the perspective of a parallax prediction device, which can be integrated into a parallax prediction equipment. The parallax prediction equipment can be an image sensor chip, an image sensor, a camera, or an electronic device such as a mobile phone or a computer. That is, the parallax prediction method of this embodiment can be executed by the parallax prediction equipment.

[0034] The following detailed description is provided in conjunction with the accompanying drawings. In this embodiment, the execution subject is a disparity prediction device. It should be noted that the order of description in the following embodiments is not intended to limit the preferred order of the embodiments. Although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be performed in a different order than that shown in the accompanying drawings.

[0035] Parallax, in a binocular stereo vision system, refers to the difference in position between the projected points of the same point in a scene on the image planes of the two cameras when the same scene is captured by two cameras from different angles. These two cameras are typically horizontally positioned to simulate human eyes. For a given point in the scene, a larger parallax between its projected points in the left and right camera images indicates that the point is closer to the camera; conversely, a smaller parallax indicates that the point is farther away. A parallax map is an important output of binocular stereo vision. It is an image with the same size as either of the two camera outputs, where each pixel represents the parallax value. Figure 1 As shown. Specifically, a disparity map is a two-dimensional image in which the position of each pixel corresponds to a pixel in the original image, and the value of that pixel represents the disparity value of that point in the left and right camera images.

[0036] The importance of a disparity map lies in its inclusion of distance information about the distance between various points in a scene or object and the camera, such as... Figure 2 As shown. (Refer to...) Figure 2 Two cameras O with positional offset on the same horizontal plane l and O r They photograph the same object, and the pixel coordinates on the image are different. For the same feature point P, at camera O... l and O r The image points are p and p', respectively. The two upward arrows represent the camera's principal axes of image capture (parallel). The two horizontal lines containing the image points represent the imaging plane (equal in length along the x-direction). The two cameras O... l and O rThe line connecting the points is the baseline. Taking the left boundary of the photograph as the zero-starting edge in the x-direction, the x-coordinates of points p and p' in the corresponding phase plane are x and x', respectively. l and x r By using parallax and the principle of similar triangles, the depth information of a spatial point P from the camera can be calculated. Let camera O be... l O r The distance between the two points is B, the distance between the imaging points p and p' is b = p' - p, the depth is z, the focal length is f, and the parallax is d = x. l -x r Then we have:

[0037] b / B=(zf) / z

[0038] (Bd) / B=(zf) / z

[0039] Therefore, we can conclude that:

[0040] z = fB / d

[0041] Therefore, for a registered stereo camera, the depth of a spatial location point from the camera baseline can be calculated based on the position difference (parallax) between two corresponding pixels at the same spatial location point in the left and right views. Thus, the problem of determining the depth of a spatial location point is transformed into the problem of finding corresponding pixels in two image planes and calculating their parallax.

[0042] Understandably, after determining the depth of an object's various spatial locations from the camera baseline, a 3D reconstruction of the object can be performed. A disparity map typically only covers a portion of the surface of a spatial object, such as... Figure 3 As shown, by using multiple disparity maps covering the entire surface of the scanned object, a complete 3D model of the scanned object can be reconstructed, enabling precise 3D measurements of the object's dimensions and volume, or other industrial quality inspection operations. Specifically, after obtaining disparity maps from multiple angles corresponding to the object / scene, these maps can be converted into a series of point clouds in 3D space. The point cloud sets are then stitched together, triangulated, and missing faces are filled in. Finally, meshing is performed to form a continuous mesh surface representing the spatial shape of the object / scene, i.e., 3D reconstruction of the scene / object. In industrial 3D quality inspection, acquiring a single disparity map is the foundation for complete 3D reconstruction, which in turn is the basis for precise 3D measurements of the workpiece's dimensions and volume after 3D scanning.

[0043] Therefore, it can be seen that efficiently reconstructing local 3D models of actual industrial scenes / objects is the foundation and key to quickly reconstructing the overall model and using the overall model for further operations.

[0044] For example, in the industrial sector, the measurement and calibration of the volume and capacity of industrial containers is fundamental to their subsequent application, maintenance, and monitoring, and directly reflects the digital application of industrial metrological assets. For instance, storage tanks, reaction tanks, and mixing tanks in the petroleum and chemical industries, as well as containers and vehicle-mounted transport tanks used in the industrial transportation sector, require precise volume measurement and calibration before being put into use to estimate their available storage capacity. Furthermore, for example… Figure 4 This document demonstrates an example of 3D scanning and local spatial modeling of the inner wall of a wine vat. Large and medium-sized industrial containers used in industries such as liquor / beer and brewing, especially 1-2 ton earthenware jars, sauce vats, and wine barrels used for long-term fermentation and storage of base liquor / original brews, are also required to undergo precise volume measurement for business management purposes such as accurate inventory, credit collateral, entrusted storage, and real-time monitoring, in order to meet the corresponding capacity estimation and inventory requirements.

[0045] However, current traditional methods for measuring the volume of large and medium-sized industrial containers still largely rely on methods such as rough estimation of the tank's internal shape using geometry and water density measurement. The rough estimation method typically has a large error range for large and medium-sized tanks and is not suitable for tanks with irregular internal shapes. The water density measurement method usually requires adding a fixed amount of water to the tank and measuring the liquid level or the mass of injected water to determine the tank volume, but this method is inconvenient to operate, has high manual operation costs, and cannot meet the rapid measurement needs of massive numbers of tanks in real-world applications (such as wine cellars, which typically contain tens to hundreds of thousands of vats). Currently, accurate tank volume measurement is increasingly being accomplished using methods such as laser scanning. Laser scanning uses a handheld portable scanner to scan the inner wall of the tank, and then forms a local grid through relative registration. Multiple local grids are then stitched together into an overall model by grid matching. However, the portable laser scanning method involves a series of steps, such as attaching / removing reflective markers to the inner wall, pre-scanning the calibration points, manual fine scanning, and confirmation of scanning results during operation. The method has high accuracy but significantly low efficiency, making it difficult to apply to efficient and low-cost measurement occasions involving a large number of industrial tanks, such as liquor storage.

[0046] In the example above, in order to quickly reconstruct the overall volume model of the inner wall of the container / tank, it is necessary to first efficiently reconstruct the local three-dimensional model of the actual industrial scene / object, such as the inner wall of the industrial container / tank.

[0047] Currently, most existing technologies employ computer vision-based binocular stereo modeling to calculate the disparity of points in the left and right views of a real industrial scene / object, and then calculate depth based on the disparity to complete local stereo modeling. However, traditional computer vision-based binocular stereo modeling often uses image feature-based methods to calculate the disparity between the left and right views. Its matching of corresponding points often relies on manually designed image features and neighborhood statistics methods, which are difficult to apply to various situations such as complex lighting environments, reflective surfaces, dim lighting, and surfaces without texture features. The stability is low, and the manual cost is high and the efficiency is low, which greatly restricts the application and development of binocular stereo reconstruction technology.

[0048] Definitions:

[0049] Industrial quality inspection refers to the use of industrial cameras, lenses, light sources, and other related vision equipment. It utilizes various sensing technologies such as visible light, infrared light, electromagnetic waves, and lasers to automatically collect the size, texture, and other attributes of industrial components. Machine vision software and artificial intelligence algorithms are then used to inspect the surface, dimensions, and other industrial quality aspects. Based on the inspection results, statistics, alarms, and feedback are output to guide subsequent automated processing. Common industrial quality inspection applications include workpiece dimensional measurement and surface defect detection.

[0050] 3D scanning is a high-tech process integrating optics, mechanics, electronics, and computer technology. It is primarily used to scan the spatial shape, structure, and color of objects to obtain a 3D model of the object's surface. Common technologies include structured light, phase measurement, laser scanning, and binocular vision. In the field of industrial quality inspection, 3D scanning is widely used for workpiece dimension measurement, volume measurement, and assembly inspection due to its advantages of high precision, non-contact measurement, and rapid data acquisition.

[0051] Stereo matching is a technique that extracts depth information of an object / scene by comparing images of the same object / scene taken from different angles by two or more cameras (usually two horizontally placed cameras to simulate human eyes). Common inputs for stereo matching are images of the object / scene from both left and right perspectives, and the output is a disparity map or depth map corresponding to the object / scene. Stereo matching is fundamental to reconstructing 3D spatial models of scenes or objects.

[0052] To address the aforementioned problems, this invention discloses a disparity prediction method. Please refer to [reference needed]. Figure 5 The specific process of this disparity prediction method can be summarized in steps S10 to S50, wherein:

[0053] Step S10: Obtain a first image group and a second image group obtained by capturing the target object with a preset binocular camera, and perform feature fusion on the first image group and the second image group respectively to obtain a first fused feature image and a second fused feature image, as well as a first feature image slice sequence corresponding to the first fused feature image and a second feature image slice sequence corresponding to the second fused feature image.

[0054] Each camera group in the binocular camera system includes an RGB camera and an infrared (IR) camera. The first image group and the second image group both include an RGB image of the target object captured by the RGB camera and an infrared spectral image of the target object captured by the infrared camera. The slices (patch) at the same position in the first feature map slice sequence and the second feature map slice sequence are of equal size.

[0055] In this embodiment, the target object is the object for which disparity calculations are to be performed at various points in space, which may be the aforementioned industrial container or other objects. The first feature map slice sequence corresponding to the first fused feature map refers to the feature map slice sequence composed of all slices of the first fused feature map, and the second feature map slice sequence corresponding to the second fused feature map refers to the feature map slice sequence composed of all slices of the second fused feature map.

[0056] It should be noted that RGB cameras capture red, green, and blue light in a scene using photosensitive elements (such as CCD or CMOS sensors), and then reconstruct images by processing the different intensities of these colors. In contrast, infrared cameras form images by detecting infrared radiation emitted or reflected by objects. All objects with temperatures above absolute zero emit infrared radiation. Infrared camera imaging is unaffected by adverse factors in RGB imaging, such as complex lighting environments, reflections, dim lighting, and surfaces without texture features. Therefore, this embodiment of the invention captures two sets of RGB / infrared spectral images using a binocular camera and fuses the RGB visible light and infrared spectral features. Compared to a single RGB or infrared spectral image, this achieves more accurate, reliable, and stable image output, thereby improving the accuracy and reliability of subsequent similarity matching and disparity prediction.

[0057] Specifically, for the fusion of RGB images and infrared spectral images in the first image group, and the fusion of RGB images and infrared spectral images in the second image group, this embodiment of the invention can employ deep learning methods such as multi-input networks, attention mechanisms, or feature fusion networks. A first fused feature image and a second fused feature image are obtained through feature extraction. Simultaneously, this embodiment of the invention also slices and aggregates the first and second fused feature images to obtain a first feature map slice sequence and a second feature map slice sequence. Therefore, this embodiment of the invention provides two output forms for the fusion of RGB images and infrared spectral images: a complete feature image (i.e., the first and second fused feature images) and a sliced ​​feature map sequence (i.e., the first feature map slice sequence and the second feature map slice sequence), which are used for subsequent 4D cost volume construction and multi-geometric scale similarity matrix calculation, respectively.

[0058] It should be noted that the feature map slices in the first feature map slice sequence do not overlap with each other, and the feature map slices in the second feature map slice sequence do not overlap with each other. This can reduce the computational load of subsequent similarity prediction and improve the accuracy of similarity prediction.

[0059] Furthermore, in some embodiments, the first feature map slice sequence contains feature map slices of different sizes, and the second feature map slice sequence contains feature map slices of different sizes. Therefore, embodiments of the present invention can achieve similarity prediction for multi-scale slices, resulting in better performance of subsequent 4D cost volume filtering.

[0060] Optionally, if the first image group and the second image group have different sizes, this embodiment of the invention requires that the image sizes of the first image group and the second image group be unified first. Specifically:

[0061] If the images in the first image group and the second image group are not the same size, take any one of the images in the first image group and the second image group as a reference, and adjust the height (H) and width (W) of the other images to be the same as the height (H) and width (W) of the reference image by cropping or upsampling / downsampling.

[0062] Optionally, in some embodiments, acquiring a first image group and a second image group obtained by capturing a target object with a preset binocular camera may include:

[0063] Images of the target are captured by a binocular camera to obtain a first image group and a second image group. The infrared spectral images in the first image group and the second image group are active infrared spectral images with pseudo-random infrared pattern projection.

[0064] In the embodiments described earlier in this example, the infrared spectral image can be a passive infrared spectral image or an active infrared spectral image with an attached infrared pattern projection. In this example, by attaching a pseudo-random infrared pattern to the target image captured by the infrared camera, an active infrared spectral image with a pseudo-random infrared pattern projection is obtained, enhancing the identification capability of the depth measurement position in the left and right view scenes and improving the overall accuracy of stereo matching. The effect of active infrared spectral stereo scanning is shown in the figure below. Figure 6 As shown.

[0065] Optionally, refer to Figure 7 In some embodiments, submodule 1 performs feature fusion on the first image group and the second image group respectively to obtain a first fused feature image and a second fused feature image, which may include:

[0066] The first image group and the second image group are fused using a preset feature fusion network, and the fused features are extracted using a preset 2D convolutional neural network to obtain the first fused feature image and the second fused feature image.

[0067] Specifically, in this embodiment of the invention, a 1D convolutional feature weighting fusion operation is first performed on the first image group / second image group through a feature fusion network, and then the fused features are extracted through a 2D convolutional neural network to obtain a first fused feature image and a second fused feature image. The feature fusion operation and the fused feature extraction operation of the first image group / second image group are independent of each other.

[0068] Furthermore, in some embodiments, the weights of the feature fusion network are the same as the weights of the 2D convolutional neural network.

[0069] It is understandable that by setting the weights of the feature fusion network to be the same as those of the 2D convolutional neural network, consistency between feature fusion and fusion feature extraction can be maintained.

[0070] Step S20: Predict the similarity of slices at the same position in the first feature map slice sequence and the second feature map slice sequence to obtain a similarity sequence, and generate a similarity filter map based on the similarity sequence.

[0071] Specifically, for N different slice sizes of the first feature map slice sequence and the second feature map slice sequence, a similarity sequence with N similarity values ​​can be generated. In this embodiment of the invention, after obtaining the similarity sequence through slice similarity prediction, the similarity sequence can be converted into a multi-scale similarity object through dimensionality transformation (reshape), and then a similarity filter map can be obtained by processing this multi-scale similarity object through convolution.

[0072] Optionally, refer to Figure 7 In some embodiments, submodule 2 may include step S20 as follows:

[0073] S21. The similarity of slices at the same position in the first feature map slice sequence and the second feature map slice sequence is predicted by the preset FlexiViT backbone network to obtain a similarity sequence.

[0074] S22. Generate multi-size similar objects based on similarity sequences;

[0075] S23. Process multi-size similar objects through 1D convolution to obtain a similarity filter map.

[0076] The FlexiViT backbone network includes an attention association head for cross-view feature maps, outputting a similarity sequence corresponding to the first and second feature map slice sequences. Furthermore, the weights of the FlexiViT backbone network and the attention association head are identical to maintain consistency in feature analysis during multi-geometric-scale similarity prediction, improving the accuracy and reliability of the prediction results. Then, in this embodiment, the similarity sequence is reshaped into multi-size similarities, and these similarities are processed by 1D convolution to obtain a similarity filter map with the same geometric scale as the first and second fused feature images, which serves as the weight for subsequent 4D cost volume filtering.

[0077] Understandably, referring to Figure 7 It can be seen that steps S21 to S23 are the multi-geometric scale similarity calculation process.

[0078] Step S30: Construct the 4D cost body to be filtered based on the first fused feature map and the second fused feature map.

[0079] In deep learning and computer vision, the cost volume is often used in stereo matching tasks. It is a multidimensional data structure that represents the correspondence between two images.

[0080] Specifically, step S30 may include:

[0081] The first and second fused feature maps are interleaved and stitched together at different disparity scales to obtain the 4D cost volume to be filtered; or

[0082] The differences between the first fused feature map and the second fused feature map at corresponding positions are calculated and aggregated at different disparity scales to obtain the 4D cost volume to be filtered.

[0083] Step S40: Filter the 4D cost volume to be filtered according to the similarity filtering map to obtain the filtered 4D cost volume.

[0084] Specifically, using the similarity filter map obtained in step S20, this similarity filter map is used as the weight for filtering the 4D cost body to be filtered, and a filtering operation is performed on the 4D cost body to be filtered, so as to further optimize the 4D cost body and obtain the filtered 4D cost body. It can be understood that, as shown in step S20, in this embodiment, for the filtering operation of the 4D cost body, the similarity filter map is an attention weight that integrates the similarity of feature slices at multiple geometric scales, which can suppress redundant spatial information in the 4D cost body and enhance its stereo matching related information.

[0085] Step S50: Generate a disparity regression prediction map based on the filtered 4D cost volume.

[0086] In this embodiment, the value of each pixel in the disparity regression prediction map is the predicted disparity value corresponding to the corresponding pixel in the reference image, and the reference image is any one of the first image group and the second image group.

[0087] Therefore, based on the filtered 4D cost volume obtained from the aforementioned steps, this embodiment of the invention can generate an accurate and reliable disparity regression prediction map with the same geometric dimensions as the reference image. Furthermore, this disparity regression prediction map is obtained based on the fused spectral features of RGB and infrared images, and the prediction process does not rely on manual design. This invention improves the stability of disparity calculation in binocular stereo systems, reduces manual costs in the process, and increases efficiency. It is understood that although this embodiment of the invention only involves the rapid construction of a single disparity map covering a local surface of the target object and does not currently involve the stereo stitching process of multiple disparity maps, the final disparity regression prediction map can serve as the basis for subsequently obtaining the depth corresponding to each pixel, reconstructing the point cloud of the target object, and triangulating the mesh.

[0088] Furthermore, referring to Figure 7 In some embodiments, submodule 3 may include step S50 as follows:

[0089] Based on the filtered 4D cost volume, an encoding / decoding network structure in the form of SegNet is used, employing 3D convolution for encoding and decoding operations, followed by upsampling and 1D convolutional feature weighting and fusion operations to obtain a disparity regression prediction map; or

[0090] Calculate the minimum cost disparity of the 3D slice at each pixel location in the filtered 4D cost volume to obtain the disparity regression prediction map.

[0091] As can be seen from steps S10 to S50, this embodiment of the invention employs a deep learning disparity map prediction method based on binocular stereo matching and 4D connectivity cost bodies. It creatively proposes a multi-geometric scale feature slice sequence approach, leveraging the FlexiViT visual backbone model to establish a cross-spatial scale fusion similarity filtering weight matrix, which is then used to filter the 4D connectivity cost body. This suppresses spatial redundancy and strengthens stereo matching-related information, thereby improving the accuracy and efficiency of subsequent cost body aggregation and disparity map regression prediction. Simultaneously, this embodiment uses a fusion approach of RGB and active infrared spectral modes to aggregate binocular left and right view features in the deep learning model. This serves as the basis for constructing the 4D cost body connectivity and the foundation for building multi-geometric scale correlation filtering weights. Binocular stereo matching is performed based on multimodal fusion features, rather than the traditional method of matching corresponding points using image pixels or their neighborhood statistics. This achieves a more accurate matching effect than a single RGB image or infrared spectral image.

[0092] In summary, this invention utilizes a binocular camera system consisting of two sets of RGB cameras / infrared cameras to capture two sets of RGB images and infrared spectral images. Feature fusion is then performed on these two sets of images to obtain corresponding fused feature images and a corresponding feature map slice sequence. By fusing the features of both RGB visible light and infrared spectra, a more accurate, reliable, and stable image output is achieved compared to a single RGB image or infrared spectral image, thereby improving the accuracy and reliability of subsequent similarity matching and disparity prediction. Furthermore, this invention performs phase fusion on slices at the same positions in the first and second feature map slice sequences. Similarity is predicted to obtain a similarity sequence, and a similarity filter map is generated based on the similarity sequence. This filter map is then used to filter the 4D cost volume constructed based on the first and second fusion feature maps, suppressing spatial redundancy information in the 4D cost volume and enhancing stereo matching-related information. Furthermore, this invention generates an accurate and reliable disparity regression prediction map based on the filtered 4D cost volume. Since this disparity regression prediction map is based on the fused spectral image features of RGB and infrared, and the prediction process does not rely on manual design, this invention improves the stability of disparity calculation in binocular stereo systems, reduces manual costs in the process, and improves efficiency.

[0093] The parallax prediction method of this invention uses low-cost equipment, is highly feasible and efficient, and the measurement and reconstruction scene model and volume matching hardware and operation methods are simple and easy to implement. It can effectively reduce the cost and implementation threshold of stereo measurement in related industrial application fields, and help overcome the problems of high labor costs and low implementation efficiency.

[0094] The parallax prediction method of this invention can be used in relatively common industrial scenarios / part local 3D reconstruction applications, such as typical application scenarios in industries like liquor, brewing, petroleum, chemicals, and food that require the measurement and 3D spatial modeling of the internal volume of a large number of large and medium-sized industrial tanks. It can also be extended to applications such as physical measurement and quality inspection of large industrial parts. Furthermore, it can be combined with intelligent industrial IoT and related platform technologies to help enterprises digitize industrial measurement assets and promote the exploration and promotion of new industrial applications such as intelligent volume measurement.

[0095] Continue to refer to Figure 7 A specific embodiment of the disparity prediction method of the present invention is given:

[0096] 1) The left and right RGB image channels and infrared spectral image channels of the registered binocular camera are independently fused to form a fused feature image. The infrared image can be an active infrared spectral image with pseudo-random infrared pattern projection or a passive infrared spectral image; the RGB image can be an RGB image converted to the left and right camera views through homography transformation, or an RGB image independently acquired from the left and right camera views. Both RGB and infrared spectral (plus active infrared structured light) channels serve as input signals for multimodal stereo matching to combat adverse environmental factors such as low light, reflection, and lack of texture, thereby improving the accuracy of feature matching in disparity analysis. The fusion method involves merging the two types of channels along the channel dimension into an input tensor of form (4, H, W), where the first dimension consists of three RGB pixel values ​​plus one infrared reflection pixel value, and H and W are the height and width of the RGB image and infrared spectral image, respectively. Each channel is normalized according to its mean and variance.

[0097] 2) Feature extraction is performed on the two fused input images. In this embodiment, EfficientNet is directly used as the feature extraction backbone network for the fused input images, and a necessary backend is added. Note that the two feature extraction operations are independent and parallel, and they use EfficientNet with shared weights for feature analysis. The final output is a 2-way fused feature map, which serves as the input to submodule 3. Its form is a three-dimensional tensor of (F, H / 2, W / 2), where F is the number of feature dimensions of the output image. This fused feature map can also be represented as a series of non-overlapping feature map slice sequences of size P. In this embodiment, the value of P can be one of eight geometric scales, such as [4, 8, 12, 16, 24, 32, 48, 64]. The fused feature map is represented as eight switching sequences with different switching sizes, where the feature slice shape in each sequence is (F, P, P), and the number of feature slices in the sequence is HW / (4*P). 2 ).

[0098] 3) ViT Encoder Stage. Eight different P-sized fused feature map slice sequences are each independently processed by the ViT encoder using autocorrelation sequence operations. In this embodiment, the FlexiViT encoder is used as the encoder backbone network to support input slice map sequences of different geometric sizes. Note that the processing of each of the eight fused feature map slice sequences from the left and right views constitutes two independent network branches, and the network structures traversed by these branches share weights.

[0099] 4) ViT Decoder Stage. The slice sequences at different geometric scales are again decoded by the ViT decoder. Note that the decoder has two inputs: self-focused encoding sequences from the left and right views, representing slices of the same size from the higher-level ViT encoder. Cross-correlation analysis is performed on the left and right feature sequences at different slice geometric scales, and the final output is a slice similarity metric sequence (i.e., a similarity sequence) of the same length as the input feature map slice sequences. There are a total of 8 slice similarity metric sequences, with the similarity metrics being scalar values.

[0100] 5) Each slice similarity measure sequence can be arranged in a similar order to the corresponding input feature slice sequence, and upsampled within the scale range of each slice, and rearranged into an image plane form to form 8 similarity measure plane tensors. Each tensor has a shape of (1, H / 2, W / 2). Finally, they are concatenated into a multi-scale similarity tensor of (8, H / 2, W / 2), and then a 1D convolution is applied to fuse the 8 components across multiple geometric scales in the first channel into a comprehensive similarity scalar. Finally, a fused similarity filter map W of shape (1, H / 2, W / 2) is output.

[0101] 6) In submodule 3, firstly, based on the left and right fused feature maps output by submodule 1, and according to the disparity offset range of D levels, the feature channels are misaligned and merged to generate a 4D connection cost body C with shape (2*F, D, H / 2, W / 2), representing the potential disparity deviation degree of the left and right fused feature views at a specific view position. In this embodiment, D is set to 64.

[0102] 7) Use the fused similarity filter map W to filter redundant information in the 4D cost body C, compress and enhance its similarity representation ability to obtain the filtered 4D cost body C'. The calculation method for the value of its i-th channel is as follows:

[0103] C'(i)=W⊙C(i)

[0104] The ⊙ symbol represents element-wise multiplication, which means applying the scale fusion similarity weights in W to all channels of the 4D cost volume before filtering.

[0105] 8) Based on the 4D cost volume C' obtained after filtering, a U-shaped network structure with a bypass branch mechanism in the form of SegNet is adopted. 3D convolution and deconvolution operations are used to perform encoding and decoding operations on C' to achieve cost aggregation.

[0106] 9) The result after cost aggregation is upsampled and enlarged to the same resolution as the original image of the reference view (i.e., the left view). The corresponding regression prediction operation is then performed, and the final regression output is the disparity prediction map of the reference view.

[0107] This embodiment also provides a parallax prediction device, which can be specifically integrated into a parallax prediction equipment. For example, the pixel array of an image sensor chip includes a photosensitive area and an optically dark area, such as... Figure 8 As shown, the disparity prediction device may include:

[0108] The feature fusion unit 801 is used to acquire a first image group and a second image group obtained by a preset binocular camera capturing a target object, and to perform feature fusion on the first image group and the second image group respectively to obtain a first fused feature image and a second fused feature image, as well as a first feature image slice sequence corresponding to the first fused feature image and a second feature image slice sequence corresponding to the second fused feature image. Each group of cameras in the binocular camera includes an RGB camera and an infrared camera. The first image group and the second image group both include RGB images and infrared spectral images. The slices at the same position in the first feature image slice sequence and the second feature image slice sequence are of equal size.

[0109] The similarity prediction unit 802 is used to predict the similarity of slices at the same position in the first feature map slice sequence and the second feature map slice sequence, obtain a similarity sequence, and generate a similarity filter map based on the similarity sequence.

[0110] The cost body construction unit 803 is used to construct the 4D cost body to be filtered based on the first fused feature map and the second fused feature map.

[0111] The filtering unit 804 is used to filter the 4D cost volume to be filtered according to the similarity filtering map to obtain the filtered 4D cost volume.

[0112] The prediction unit 805 is used to generate a disparity regression prediction map based on the filtered 4D cost volume. The value of each pixel in the disparity regression prediction map is the predicted disparity value corresponding to the corresponding pixel in the reference image. The reference image is any one of the first image group and the second image group.

[0113] like Figure 9 As shown, Figure 9This is a schematic diagram of the structure of a disparity prediction device provided in an embodiment of the present invention. The disparity prediction device 1100 includes a processor 1101 with one or more processing cores, a memory 1102 with one or more computer-readable storage media, and a computer program stored in the memory 1102 and executable on the processor. The processor 1101 and the memory 1102 are electrically connected. Those skilled in the art will understand that the structure of the disparity prediction device shown in the figure does not constitute a limitation on the disparity prediction device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0114] The processor 1101 is the control center of the disparity prediction device 1100. It connects to various parts of the disparity prediction device 1100 via various interfaces and lines. By running or loading software programs and / or units stored in the memory 1102, and by calling data stored in the memory 1102, it executes various functions and processes data of the disparity prediction device 1100, thereby providing overall monitoring of the disparity prediction device 1100. The processor 1101 can be a CPU, GPU, network processor (NP), etc., and can implement or execute the methods, steps, and logic diagrams disclosed in the embodiments of this invention.

[0115] In this embodiment of the invention, the processor 1101 in the parallax prediction device 1100 loads the instructions corresponding to the processes of one or more applications into the memory 1102 according to the following steps, and the processor 1101 runs the applications stored in the memory 1102 to realize various functions. See the previous embodiments for details, which will not be repeated here.

[0116] Optional, such as Figure 9 As shown, the parallax prediction device 1100 further includes: a touch display screen 1103, a radio frequency circuit 1104, an audio circuit 1105, an input unit 1106, and a power supply 1107. The processor 1101 is electrically connected to the touch display screen 1103, the radio frequency circuit 1104, the audio circuit 1105, the input unit 1106, and the power supply 1107. Those skilled in the art will understand that... Figure 9 The parallax prediction device structure shown does not constitute a limitation on the parallax prediction device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0117] The touch display screen 1103 can be used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. The touch display screen 1103 may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the parallax prediction device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Optionally, the display panel can be configured using a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar technologies. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), generate corresponding operation commands, and execute the corresponding program according to the operation commands. Optionally, the touch panel may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch location and the signal generated by the touch operation, transmitting the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 1101. It can also receive and execute commands from the processor 1101. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 1101 to determine the type of touch event. Subsequently, the processor 1101 provides corresponding visual output on the display panel based on the type of touch event. In this embodiment, the touch panel and the display panel can be integrated into the touch display screen 1103 to achieve input and output functions. However, in some embodiments, the touch panel and the touch display screen 1103 can be used as two independent components to achieve input and output functions. That is, the touch display screen 1103 can also be used as part of the input unit 1106 to achieve input functions.

[0118] The radio frequency circuit 1104 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other parallax prediction devices, and to transmit and receive signals with network devices or other parallax prediction devices.

[0119] Audio circuitry 1105 can be used to provide an audio interface between the user and the parallax prediction device via a speaker and a microphone. Audio circuitry 1105 can convert received audio data into electrical signals and transmit them to the speaker, where the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuitry 1105, converted back into audio data, and processed by processor 1101 before being transmitted via radio frequency circuitry 1104 to, for example, another parallax prediction device, or output to memory 1102 for further processing. Audio circuitry 1105 may also include an earphone jack to provide communication between external headphones and the parallax prediction device.

[0120] The input unit 1106 can be used to receive input numbers, characters, or user characteristic information (such as fingerprints, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0121] Power supply 1107 is used to power the various components of parallax prediction device 1100. Optionally, power supply 1107 can be logically connected to processor 1101 through power management system, thereby enabling functions such as charging, discharging, and power consumption management through power management system. Power supply 1107 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0122] although Figure 9 As not shown in the diagram, the parallax prediction device 1100 may also include a camera, a sensor, a wireless fidelity unit, a Bluetooth unit, etc., which will not be described in detail here.

[0123] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0124] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0125] To this end, embodiments of the present invention provide a computer-readable storage medium storing a plurality of computer programs that can be loaded by a processor to execute any of the disparity prediction methods provided in the embodiments of the present invention. The computer program can execute the steps of the aforementioned disparity prediction method, as described in the preceding embodiments, and will not be repeated here.

[0126] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0127] Since the computer program stored in the computer-readable storage medium can execute any of the disparity prediction methods provided in the embodiments of the present invention, it can achieve the beneficial effects that any of the disparity prediction methods provided in the embodiments of the present invention can achieve, as detailed in the preceding embodiments, and will not be repeated here.

[0128] In the above embodiments of the disparity prediction device, computer-readable storage medium, disparity prediction equipment, and computer program product, the descriptions of each embodiment have different focuses. Parts not described in detail in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes and beneficial effects of the disparity prediction device, computer-readable storage medium, computer program product, disparity prediction equipment, and their corresponding units described above can be referred to the description of the disparity prediction method in the above embodiments, and will not be repeated here.

[0129] The foregoing has provided a detailed description of the disparity prediction method, disparity prediction device, disparity prediction equipment, computer-readable storage medium, and computer program product provided by the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A parallax prediction method, characterized in that, The method includes: A first image group and a second image group obtained by capturing a target object with a preset binocular camera are acquired, and feature fusion is performed on the first image group and the second image group respectively to obtain a first fused feature image and a second fused feature image, as well as a first feature image slice sequence corresponding to the first fused feature image and a second feature image slice sequence corresponding to the second fused feature image. Each group of cameras in the binocular camera includes an RGB camera and an infrared camera. Both the first image group and the second image group include RGB images and infrared spectral images. The slices at the same position in the first feature image slice sequence and the second feature image slice sequence are of equal size. The similarity of slices at the same position in the first feature map slice sequence and the second feature map slice sequence is predicted to obtain a similarity sequence, and a similarity filter map is generated based on the similarity sequence. Construct the 4D cost body to be filtered based on the first fused feature map and the second fused feature map; The 4D cost volume to be filtered is filtered according to the similarity filtering map to obtain the filtered 4D cost volume. A disparity regression prediction map is generated based on the filtered 4D cost volume. The value of each pixel in the disparity regression prediction map is the predicted disparity value corresponding to the corresponding pixel in the reference image. The reference image is any one of the first image group and the second image group.

2. The disparity prediction method as described in claim 1, characterized in that, The acquisition of the first image group and the second image group obtained by capturing the target object with a preset binocular camera includes: The target object is captured by the binocular camera to obtain the first image group and the second image group, wherein the infrared spectral images in the first image group and the second image group are active infrared spectral images with pseudo-random infrared pattern projection.

3. The disparity prediction method as described in claim 2, characterized in that, The step of fusing features between the first image group and the second image group to obtain a first fused feature image and a second fused feature image includes: The first image group and the second image group are fused using a preset feature fusion network, and the fused features are extracted using a preset 2D convolutional neural network to obtain the first fused feature image and the second fused feature image.

4. The disparity prediction method as described in claim 3, characterized in that, The weights of the feature fusion network are the same as the weights of the 2D convolutional neural network.

5. The disparity prediction method as described in claim 1, characterized in that, The step of predicting the similarity of slices at the same position in the first feature map slice sequence and the second feature map slice sequence to obtain a similarity sequence, and generating a similarity filter map based on the similarity sequence, includes: The similarity sequence is obtained by predicting the similarity of slices at the same position in the first feature map slice sequence and the second feature map slice sequence through a preset FlexiViT backbone network. The FlexiViT backbone network is equipped with an attention association head for cross-view feature maps, and the weights of the FlexiViT backbone network and the attention association head are the same. Generate multi-size similar objects based on the similarity sequence; The similarity filter map is obtained by processing the multi-size similar objects through 1D convolution.

6. The disparity prediction method as described in claim 1, characterized in that, The step of constructing the 4D cost volume to be filtered based on the first fused feature map and the second fused feature map includes: The first fused feature map and the second fused feature map are interleaved and stitched together at different disparity scales to obtain the 4D cost volume to be filtered; or The differences between the first fused feature map and the second fused feature map at corresponding positions are calculated and aggregated at different parallax scales to obtain the 4D cost volume to be filtered.

7. The disparity prediction method as described in claim 1, characterized in that, The step of generating a disparity regression prediction map based on the filtered 4D cost volume includes: Based on the filtered 4D cost volume, using a SegNet-like encoder-decoder network structure, 3D convolution is used for encoding and decoding operations, followed by upsampling and 1D convolutional feature weighting fusion operations to obtain the disparity regression prediction map; or The minimum cost disparity of the 3D slice at each pixel location in the filtered 4D cost volume is calculated to obtain the disparity regression prediction map.

8. A parallax prediction device, characterized in that, The disparity prediction device includes: The feature fusion unit is used to acquire a first image group and a second image group obtained by a preset binocular camera capturing a target object, and to perform feature fusion on the first image group and the second image group respectively to obtain a first fused feature image and a second fused feature image, as well as a first feature image slice sequence corresponding to the first fused feature image and a second feature image slice sequence corresponding to the second fused feature image. Each group of cameras in the binocular camera includes an RGB camera and an infrared camera. The first image group and the second image group both include RGB images and infrared spectral images. The slices at the same position in the first feature image slice sequence and the second feature image slice sequence are of equal size. A similarity prediction unit is used to predict the similarity between slices at the same position in the first feature map slice sequence and the second feature map slice sequence to obtain a similarity sequence, and generate a similarity filter map based on the similarity sequence. The cost body construction unit is used to construct a 4D cost body to be filtered based on the first fused feature map and the second fused feature map. A filtering unit is used to filter the 4D cost body to be filtered according to the similarity filtering map to obtain a filtered 4D cost body. The prediction unit is used to generate a disparity regression prediction map based on the filtered 4D cost volume. The value of each pixel in the disparity regression prediction map is the predicted disparity value corresponding to the corresponding pixel in the reference image. The reference image is any one of the first image group and the second image group.

9. A parallax prediction device, characterized in that, It includes a processor and a memory, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of any of the disparity prediction methods of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of any of the disparity prediction methods of claims 1-7.

Images