Parallax prediction method and device, equipment and storage medium

Abstract

The invention discloses a parallax prediction method, device and equipment and a storage medium, and the method comprises the steps: obtaining two groups of RGB images and infrared spectrum images through the shooting of a binocular camera composed of two groups of RGB cameras / infrared cameras, and carrying out the feature fusion of the two groups of images, obtaining a corresponding fusion feature image and a corresponding feature image slice sequence; predicting the similarity of slices at the same position in the first feature map slice sequence and the second feature map slice sequence to obtain a similarity sequence, and generating a similarity filtering map according to the similarity sequence; constructing a to-be-filtered 4D cost body according to the first fusion feature map and the second fusion feature map; filtering the to-be-filtered 4D cost body according to the similarity filtering graph to obtain a filtered 4D cost body; and generating a parallax regression prediction map according to the filtered 4D cost body. According to the method, the stability of parallax calculation in the binocular stereo system can be improved, the labor cost in the process is reduced, and the efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a disparity prediction method, device, equipment and storage medium. Background Art

[0002] Disparity refers to the position difference between the projection points of the same point in the scene on the imaging planes of the two cameras when two cameras shoot the same scene from different angles in a binocular stereo vision system. The two cameras are usually placed horizontally to simulate human eyes. For a point in the scene, the greater the disparity between its projection points in the left and right camera images, the closer the point is to the camera. The disparity map is an important output in binocular stereo vision. It is an image with any image in the image pair output by the two cameras as the reference, the size of the reference image, and the value of each pixel as the disparity value. Specifically, the disparity map is a two-dimensional image, in which the position of each pixel corresponds to a pixel in the original image, and the value of the pixel represents the disparity value of the point in the left and right camera images. The importance of the disparity map lies in that it contains the distance information of each local point of the scene or object from the camera. By covering multiple disparity maps on the entire surface of the scanned object, the overall stereo model of the scanned object can be completely reconstructed, so that the scanned object can be accurately measured in three dimensions such as size and volume or other industrial quality inspection behaviors can be performed.

[0003] It can be seen that efficiently reconstructing local stereo modeling of actual industrial scenes / objects is the basis and key to quickly reconstructing the overall model and using the overall model for further operations. Traditional binocular stereo modeling based on computer vision mostly uses the left and right view disparity calculation method based on image features, and its pixel matching of same-name points mostly uses manually designed image features and field statistical methods, which is difficult to apply to various occasions such as complex lighting environments, reflective, dim, and texture-free surfaces. It has low stability, high labor costs, and low efficiency, which greatly restricts the application and development of binocular stereo reconstruction technology. Summary of the invention

[0004] The embodiments of the present invention provide a disparity prediction method, apparatus, device and storage medium, aiming to improve the stability of disparity calculation in a binocular stereo system, reduce the labor cost in the process and improve the efficiency.

[0005] In a first aspect, an embodiment of the present invention provides a disparity prediction method, the method comprising:

[0006] A first image group and a second image group are obtained by photographing a target object with a preset binocular camera, and feature fusion is performed on the first image group and the second image group respectively to obtain a first fused feature image and a second fused feature image, as well as a first feature map slice sequence corresponding to the first fused feature map and a second feature map slice sequence corresponding to the second fused feature map, wherein each camera group in the binocular camera includes an RGB camera and an infrared camera, and the first image group and the second image group both include an RGB image and an infrared spectrum image, and the slice sizes at the same position in the first feature map slice sequence and the second feature map slice sequence are equal;

[0007] Predicting the similarity of slices at the same position in the first feature map slice sequence and the second feature map slice sequence to obtain a similarity sequence, and generating a similarity filter graph according to the similarity sequence;

[0008] Construct a 4D cost volume to be filtered according to the first fused feature map and the second fused feature map;

[0009] Filter the 4D cost volume to be filtered according to the similarity filter graph to obtain a filtered 4D cost volume;

[0010] A disparity regression prediction map is generated according to the filtered 4D cost volume, wherein the value of each pixel in the disparity regression prediction map is a predicted disparity value corresponding to a corresponding pixel in a reference image, and the reference image is any image in the first image group and the second image group.

[0011] In a second aspect, an embodiment of the present invention provides a disparity prediction device, the disparity prediction device comprising:

[0012] A feature fusion unit is used to obtain a first image group and a second image group obtained by shooting a target object with a preset binocular camera, and perform feature fusion on the first image group and the second image group respectively to obtain a first fused feature image and a second fused feature image, as well as a first feature map slice sequence corresponding to the first fused feature map and a second feature map slice sequence corresponding to the second fused feature map, wherein each camera group in the binocular camera includes an RGB camera and an infrared camera, the first image group and the second image group both include an RGB image and an infrared spectrum image, and the slice sizes of the first feature map slice sequence and the second feature map slice sequence at the same position are equal;

[0013] A similarity prediction unit is used to predict the similarity of slices at the same position in the first feature map slice sequence and the second feature map slice sequence to obtain a similarity sequence, and generate a similarity filter map according to the similarity sequence;

[0014] A cost volume construction unit, used to construct a 4D cost volume to be filtered according to the first fused feature map and the second fused feature map;

[0015] A filtering unit, used for filtering the 4D cost volume to be filtered according to the similarity filtering graph to obtain a filtered 4D cost volume;

[0016] The prediction unit is used to generate a disparity regression prediction map according to the filtered 4D cost volume, wherein the value of each pixel in the disparity regression prediction map is a predicted disparity value corresponding to a corresponding pixel in a reference image, and the reference image is any image in the first image group and the second image group.

[0017] In a third aspect, an embodiment of the present invention further provides a disparity prediction device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any disparity prediction method provided by the embodiment of the present invention.

[0018] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, which includes a computer program. When the computer program runs on an electronic device, the computer program is used to enable the electronic device to execute the steps of any disparity prediction method provided by the embodiment of the present invention.

[0019] The beneficial effects of the present invention are:

[0020] The present invention obtains two groups of RGB images and infrared spectrum images by shooting with a binocular camera composed of two groups of RGB cameras / infrared cameras, and performs feature fusion on the two groups of images respectively to obtain corresponding fused feature images and corresponding feature map slice sequences. By fusing the two spectral picture features of RGB visible light and infrared, a more accurate, reliable and stable image output is achieved compared with a single RGB image or infrared spectrum image, so as to improve the accuracy and reliability of subsequent similarity matching and disparity prediction. Then, the present invention predicts the similarity of slices at the same position in the first feature map slice sequence and the second feature map slice sequence to obtain a similarity sequence, and generates a similarity filter map according to the similarity sequence for subsequent filtering of a 4D cost body constructed according to the first fused feature map and the second fused feature map, thereby suppressing spatial redundant information in the 4D cost body and strengthening stereo matching related information. Furthermore, the present invention generates an accurate and reliable disparity regression prediction map according to the filtered 4D cost body, and the disparity regression prediction map is obtained based on the fused spectrum picture features of RGB and infrared, and the prediction process does not need to rely on manual design. The present invention improves the stability of disparity calculation in a binocular stereo system, reduces the labor cost in the process, and improves efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 It is a schematic diagram of the left and right views and the disparity map taken by the binocular stereo vision system;

[0023] Figure 2 It is a schematic diagram of the parallax and depth calculation of the left and right cameras of the binocular stereo vision system;

[0024] Figure 3 It is a schematic diagram of the principle of three-dimensional reconstruction of an object;

[0025] Figure 4 This is a schematic diagram of the three-dimensional scanning of the inner wall of the wine tank and the local space modeling;

[0026] Figure 5 is a schematic flow chart of an embodiment of a disparity prediction method provided in an embodiment of the present invention;

[0027] Figure 6 This is a rendering of the active infrared spectrum stereo scanning in an embodiment of the present invention;

[0028] Figure 7 is a schematic flow chart of an embodiment of a disparity prediction method provided in an embodiment of the present invention;

[0029] Figure 8 is a schematic structural diagram of a disparity prediction device provided in an embodiment of the present invention;

[0030] Fig. 9 is a schematic diagram of the structure of a disparity prediction device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention. At the same time, in the description of the embodiments of the present invention, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0032] Embodiments of the present invention provide a disparity prediction method, apparatus, device and storage medium.

[0033] Specifically, this embodiment will be described from the perspective of a disparity prediction device, which can be integrated into a disparity prediction device. The disparity prediction device can be an image sensor chip, an image sensor, a camera, or an electronic device such as a mobile phone or a computer. That is, the disparity prediction method of the embodiment of the present invention can be executed by a disparity prediction device.

[0034] The following is a detailed description in conjunction with the accompanying drawings. In this embodiment, the execution subject is a disparity prediction device as an example. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments. Although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings.

[0035] Parallax refers to the position difference between the projection points of the same point in the scene on the imaging planes of the two cameras when the two cameras shoot the same scene from different angles in a binocular stereo vision system. The two cameras are usually placed horizontally to simulate human eyes. For a certain point in the scene, if the parallax between its projection points in the left and right camera images is large, it means that this point is closer to the camera; conversely, if the parallax is small, it means that this point is farther away from the camera. The disparity map is an important output in binocular stereo vision. It is an image that is based on any image in the image pair output by the two cameras, has the size of the reference image, and the value of each pixel is the disparity value, such as Figure 1 Specifically, the disparity map is a two-dimensional image in which the position of each pixel corresponds to a pixel in the original image, and the value of the pixel represents the disparity value of the point in the left and right camera images.

[0036] The importance of the disparity map lies in that it contains the distance information of each local point of the scene or object from the camera, such as Figure 2 See Figure 2 , there are two cameras O with position offset on the same horizontal plane l and O r , the pixel coordinates of the same object are different when they take pictures of it. l and O r The two upward arrows represent the main axes of the camera (parallel), and the two horizontal lines where the imaging points are located represent the imaging plane (equal length along the x direction). l and O rThe line between them is the baseline. Taking the left edge of the photo as the zero starting edge in the x direction, the x coordinates of point p and point p' in the corresponding phase plane are x l and x r Through parallax, the depth information of the space point P from the camera can be calculated using the principle of similar triangles. l , O r The distance between them is B, the distance between the imaging points p and p' is b = p'-p, the depth is z, the focal length is f, and the parallax d = x l -x r , then:

[0037] b / B=(zf) / z

[0038] (Bd) / B=(zf) / z

[0039] From this we can get:

[0040] z=fB / d

[0041] Therefore, for the registered binocular camera, the depth of the spatial position point from the camera baseline can be obtained based on the difference (disparity) between the positions of two pixels corresponding to the same spatial position point in the left and right views. Therefore, the problem of obtaining the depth of the spatial position point is transformed into the problem of finding the corresponding pixel points in the two image planes and calculating their disparity.

[0042] It is understandable that after obtaining the depth of each spatial position point of the object from the camera baseline, the object can be reconstructed in three dimensions. A disparity map can usually only cover part of the surface of the spatial object, such as Figure 3 As shown. By covering multiple disparity maps of the entire surface of the scanned object, the overall three-dimensional model of the scanned object can be completely reconstructed, so that the scanned object can be accurately measured in three-dimensional terms such as size and volume or other industrial quality inspection behaviors can be performed. Specifically, after obtaining the disparity maps of multiple angles corresponding to the object / scene, they can be converted into a series of point clouds in three-dimensional space, and then the point cloud collection is spliced ​​as a whole, and then triangulated and missing faces are filled, and finally meshed to form a continuous mesh surface that can represent the spatial shape of the object / scene, that is, a three-dimensional reconstruction of the scene / object. In industrial three-dimensional quality inspection, the acquisition of a single disparity map is the basis for complete three-dimensional reconstruction, and complete three-dimensional reconstruction is the basis for accurate three-dimensional measurement of the size and volume of the workpiece after three-dimensional scanning.

[0043] It can be seen that efficiently reconstructing local stereo modeling of actual industrial scenes / objects is the basis and key to quickly reconstructing the overall model and using the overall model for further operations.

[0044] For example, in the industrial field, the measurement and calibration of the volume capacity of industrial containers is the basis for their subsequent application, maintenance and monitoring, and is also a direct reflection of the digital application of industrial measurement assets. For example, before the storage tanks, reaction tanks, mixing tanks, etc. in the petroleum and chemical industries, as well as containers and vehicle-mounted transport tanks used in the industrial transportation industry are put into use, their volumes must be accurately measured and the corresponding tank calibration must be completed to estimate their available tank reserves; for another example, Figure 4 The example of three-dimensional scanning and local space modeling of the inner wall of a wine vat is shown. The large and medium-sized industrial containers used in the liquor / beer and brewing industries, especially the 1-2 ton ceramic jars, sauce jars, and wine barrels used for long-term fermentation and storage of base wine / original brew, are large in number, have high value of stored materials, have mixed sources of jars, and have large individual differences. For business management purposes such as accurate inventory, credit mortgage, entrusted storage, and real-time monitoring, it is also necessary to perform corresponding storage volume accurate measurement to meet the corresponding capacity estimation and inventory needs.

[0045] However, the current traditional large and medium-sized industrial container volume measurement still mostly uses the tank body geometry rough estimation method, water density measurement method, etc. Among them, the geometric rough estimation method usually has a large error range for large and medium-sized tanks, and is also not conducive to the effective measurement of tanks with irregular geometric shapes; and the water density measurement method usually requires adding a certain amount of water to the tank body and measuring the liquid level or the mass of the injected water to measure the tank volume, but this method is inconvenient to operate, has high manual operation costs, and cannot meet the rapid measurement of massive tanks in real situations (such as the number of wine tanks in wine warehouses is usually in the tens of thousands to hundreds of thousands). At present, accurate tank volume measurement is increasingly being completed using methods such as laser scanning. The laser scanning method uses a handheld portable scanner to scan the inner wall of the cylinder, and then forms a local grid through relative alignment, and then splices multiple local grids into an overall model by grid matching. However, the portable laser scanning method involves a series of steps such as sticking / tearing off reflective cursors on the inner wall, pre-scanning of calibration points, manual wall-sticking fine scanning, and confirmation of scanning results during operation. The method has high accuracy but significantly low efficiency, and is difficult to apply to efficient and low-cost measurement occasions involving massive industrial tanks such as liquor storage.

[0046] In the above example, in order to quickly reconstruct the overall container / tank inner wall volume model, it is necessary to first efficiently reconstruct the local three-dimensional modeling of the actual industrial scene / object such as the industrial container / tank inner wall.

[0047] At present, existing technologies mostly use binocular stereo modeling based on computer vision to calculate the disparity of each point in the space of the actual industrial scene / object in the left and right views, and calculate the depth based on the disparity to complete local stereo modeling. However, traditional binocular stereo modeling based on computer vision mostly uses the left and right view disparity calculation method based on image features, and its pixel matching of the same name points mostly uses artificially designed image features and field statistics methods, which is difficult to apply to various occasions such as complex lighting environments, reflective, dim, and texture-free surfaces. The stability is low, and the labor cost is high and the efficiency is low, which greatly restricts the application and development of binocular stereo reconstruction technology.

[0048] Glossary:

[0049] Industrial quality inspection: refers to the use of industrial cameras, lenses, light sources and other related visual equipment, using visible light, infrared, electromagnetic waves, lasers and other sensing technologies to automatically collect industrial device size, texture, texture and other attribute characteristics, and use machine vision software and artificial intelligence algorithms to detect its surface, size and other industrial qualities, and output statistics, alarms and feedback based on the test results, and guide the subsequent automated processing process. Common industrial quality inspection occasions include workpiece size measurement, surface defect detection, etc.

[0050] 3D scanning: It is a high-tech technology that integrates optical, mechanical, electrical and computer technologies. It is mainly used to scan the spatial shape, structure and color of an object to obtain a 3D model of the object surface. Common technologies include structured light, phase measurement, laser scanning, binocular vision technology, etc. In the field of industrial quality inspection, 3D scanning is widely used in the fields of workpiece size measurement, volume measurement, assembly inspection, etc., due to its advantages of high precision, non-contact measurement and fast data acquisition.

[0051] Stereo matching: Stereo matching is a technique for extracting depth information of an object / scene by comparing images of the same object / scene taken from different angles by two or more cameras (usually two horizontally placed cameras to simulate human eyes). The common input of stereo matching is the image of the object / scene from the left and right perspectives, and the output is the disparity map or depth map corresponding to the object / scene. Stereo matching is the basis for reconstructing a three-dimensional model of a scene or object space.

[0052] In order to solve the above problem, the present invention discloses a disparity prediction method. Figure 5 The specific process of the disparity prediction method can be as follows: Step S10 to Step S50, wherein:

[0053] Step S10, obtain a first image group and a second image group obtained by photographing the target object with a preset binocular camera, and perform feature fusion on the first image group and the second image group respectively to obtain a first fused feature image and a second fused feature image, as well as a first feature map slice sequence corresponding to the first fused feature map and a second feature map slice sequence corresponding to the second fused feature map.

[0054] Among them, each group of cameras in the binocular camera includes an RGB camera and an infrared (IR) camera, the first image group and the second image group both include an RGB image output by the RGB camera when photographing the target object and an infrared spectrum image output by the infrared camera when photographing the target object, and the size of the slices (patch) at the same position in the first feature map slice sequence is equal to that in the second feature map slice sequence.

[0055] In this embodiment, the target object is an object for which the parallax calculation of each point in space is to be performed, which may be the aforementioned industrial container or other object. The first feature map slice sequence corresponding to the first fused feature map refers to a feature map slice sequence composed of all slices of the first fused feature map, and the second feature map slice sequence corresponding to the second fused feature map refers to a feature map slice sequence composed of all slices of the second fused feature map.

[0056] It should be noted that the RGB camera captures the red, green and blue light in the scene through photosensitive elements (such as CCD or CMOS sensors), and then reconstructs the image by processing the different intensities of these colors, while the infrared camera forms an image by detecting the infrared radiation emitted or reflected by the object. All objects with a temperature above absolute zero will emit infrared radiation. The imaging of the infrared camera is not affected by unfavorable factors of RGB imaging such as complex lighting environment, reflection, dimness, and texture-free surface. Therefore, the embodiment of the present invention uses a binocular camera to capture two sets of RGB images / infrared spectrum images, and fuses the two spectral picture features of RGB visible light and infrared, which achieves more accurate, reliable and stable image output compared to a single RGB image or infrared spectrum image, so as to improve the accuracy and reliability of subsequent similarity matching and disparity prediction.

[0057] Specifically, for the fusion of RGB images and infrared spectrum images in the first image group, and the fusion of RGB images and infrared spectrum images in the second image group, the embodiment of the present invention can be implemented by deep learning methods such as multi-input networks, attention mechanisms or feature fusion networks, and the first fused feature image and the second fused feature image are obtained by fusion feature extraction. At the same time, the embodiment of the present invention also slices and segments the first fused feature image and the second fused feature image, respectively, to obtain a first feature map slice sequence and a second feature map slice sequence. Therefore, the embodiment of the present invention has two output forms for the fusion of RGB images and infrared spectrum images: a complete feature image (i.e., the first fused feature image and the second fused feature image) and a slice feature map sequence (i.e., the first feature map slice sequence and the second feature map slice sequence), which are used for subsequent 4D cost volume construction and multi-geometric scale similarity matrix calculation.

[0058] It should be noted that the feature map slices in the first feature map slice sequence do not overlap with each other, and the feature map slices in the second feature map slice sequence do not overlap with each other, which can reduce the amount of calculation for subsequent similarity prediction and improve the accuracy of similarity prediction.

[0059] Furthermore, in some embodiments, the first feature map slice sequence has feature map slices of different sizes, and the second feature map slice sequence has feature map slices of different sizes. Thus, the embodiment of the present invention can achieve similarity prediction of multi-scale slices, so that the subsequent 4D cost volume filtering has better effect.

[0060] Optionally, if the first image group and the second image group have different sizes, the embodiment of the present invention needs to unify the image sizes of the first image group and the second image group first. Specifically:

[0061] If the sizes of images in the first image group and the second image group are different, taking any image in the first image group and the second image group as a reference, the height (H) and width (W) of the other images are adjusted to be the same as the height (H) and width (W) of the reference image by cropping or upsampling / downsampling.

[0062] Optionally, in some embodiments, obtaining a first image group and a second image group obtained by photographing a target object with a preset binocular camera may include:

[0063] The target image is photographed by a binocular camera to obtain a first image group and a second image group, wherein the infrared spectrum images in the first image group and the second image group are active infrared spectrum images with pseudo-random infrared pattern projection.

[0064] In the embodiments described above, the infrared spectrum image can be a passive infrared spectrum image or an active infrared spectrum image with infrared pattern projection. In this embodiment, by attaching a pseudo-random infrared pattern when shooting a target image with an infrared camera, an active infrared spectrum image with pseudo-random infrared pattern projection is obtained to enhance the recognition of the depth measurement position of the left and right viewing angle scenes and improve the overall accuracy of stereo matching. The effect diagram of active infrared spectrum stereo scanning is shown in FIG. Figure 6 shown.

[0065] Optionally, refer to Figure 7 In some embodiments, the submodule 1 performs feature fusion on the first image group and the second image group respectively to obtain a first fused feature image and a second fused feature image, which may include:

[0066] The first image group and the second image group are respectively fused with features through a preset feature fusion network, and fused features are extracted through a preset 2D convolutional neural network to obtain a first fused feature image and a second fused feature image.

[0067] Specifically, the embodiment of the present invention first performs a 1D convolution feature weighted fusion operation on the first image group / the second image group through a feature fusion network, and then extracts fusion features through a 2D convolution neural network to obtain a first fusion feature image and a second fusion feature image. The feature fusion operation and fusion feature extraction operation of the first image group / the second image group are independent of each other.

[0068] Further, in some embodiments, the weights of the feature fusion network are the same as the weights of the 2D convolutional neural network.

[0069] It can be understood that by setting the weights of the feature fusion network to be the same as the weights of the 2D convolutional neural network, the consistency of feature fusion and fused feature extraction can be maintained.

[0070] Step S20, predicting the similarity of slices at the same position in the first feature map slice sequence and the second feature map slice sequence to obtain a similarity sequence, and generating a similarity filter graph according to the similarity sequence.

[0071] Among them, for the first feature map slice sequence and the second feature map slice sequence of N different slice sizes, a similarity sequence with N similarity values ​​can be generated. After the slice similarity prediction is performed to obtain the similarity sequence, the embodiment of the present invention can convert the similarity sequence into a multi-scale similar body by dimensional conversion (reshaping), and then the multi-scale similar body can be processed by convolution to obtain a similarity filter map.

[0072] Optionally, refer to Figure 7 Submodule 2, in some embodiments, step S20 may include:

[0073] S21, predicting the similarity of slices at the same position in the first feature map slice sequence and the second feature map slice sequence through a preset FlexiViT backbone network to obtain a similarity sequence;

[0074] S22, generating multi-size similar bodies according to the similarity sequence;

[0075] S23. Process multi-sized similar bodies through 1D convolution to obtain a similarity filter graph.

[0076] The FlexiViT backbone network is provided with an attention association head for cross-view feature maps to output a similarity sequence corresponding to the first feature map slice sequence and the second feature map slice sequence. In addition, the weights of the FlexiViT backbone network and the attention association head are the same to maintain the consistency of feature analysis in the multi-geometric scale similarity prediction process and improve the accuracy and reliability of the prediction results. Then, the embodiment of the present invention reshapes the similarity sequence into a multi-size similar body, and processes the multi-size similar body through 1D convolution to obtain a similarity filter map that is consistent with the geometric scale of the first fused feature image and the second fused feature image, which is used as the weight for subsequent 4D cost volume filtering.

[0077] It is understandable that, referring to Figure 7 It can be seen that steps S21 to S23 are a multi-geometric scale similarity calculation process.

[0078] Step S30: construct a 4D cost volume to be filtered according to the first fused feature map and the second fused feature map.

[0079] In deep learning and computer vision, the cost volume is often used in stereo matching tasks. It is a multidimensional data structure that represents the correspondence between two images.

[0080] Specifically, step S30 may include:

[0081] Interlacing and splicing the first fused feature map and the second fused feature map at different disparity scales to obtain a 4D cost volume to be filtered; or

[0082] The differences between the first fused feature map and the second fused feature map at corresponding positions are calculated and aggregated at different disparity scales to obtain a 4D cost volume to be filtered.

[0083] Step S40 , filtering the 4D cost volume to be filtered according to the similarity filter graph to obtain a filtered 4D cost volume.

[0084] Specifically, the similarity filter map obtained in step S20 is used as the weight for filtering the 4D cost volume to be filtered, and the 4D cost volume to be filtered is filtered to further optimize the 4D cost volume to be filtered to obtain the filtered 4D cost volume. It can be understood that according to step S20, in this embodiment, for the filtering operation of the 4D cost volume, the similarity filter map is an attention weight that integrates the similarity of multi-geometric scale feature slices, which can suppress redundant spatial information in the 4D cost volume and strengthen its stereo matching related information.

[0085] Step S50 , generating a disparity regression prediction map according to the filtered 4D cost volume.

[0086] In this embodiment, the value of each pixel in the disparity regression prediction map is the predicted disparity value corresponding to the corresponding pixel in the reference image, and the reference image is any image in the first image group and the second image group.

[0087] Therefore, based on the filtered 4D cost volume obtained in the above steps, the embodiment of the present invention can generate a disparity regression prediction map that is consistent with the geometric size of the reference image and is accurate and reliable. In addition, the disparity regression prediction map is obtained based on the fusion spectrum picture features of RGB and infrared, and the prediction process does not need to rely on manual design. The present invention improves the stability of disparity calculation in the binocular stereo system, reduces the labor cost in the process, and improves efficiency. It can be understood that although the embodiment of the present invention only involves the rapid construction of a single disparity map covering the local surface of the target object, and does not involve the stereo stitching process of multiple disparity maps, the disparity regression prediction map finally obtained can be used as the basis for subsequently obtaining the depth corresponding to each pixel, reconstructing the point cloud and triangulated mesh of the target object.

[0088] Further, refer to Figure 7 Submodule 3, in some embodiments, step S50 may include:

[0089] Based on the filtered 4D cost volume, a SegNet-style codec network structure is used to perform codec operations using 3D convolution, and upsampling and 1D convolution feature weighted fusion operations are performed to obtain a disparity regression prediction map; or

[0090] The minimum cost disparity of the 3D slice at each pixel position in the filtered 4D cost volume is calculated to obtain the disparity regression prediction map.

[0091] Combining steps S10 to S50, it can be seen that the embodiment of the present invention adopts a deep learning disparity map prediction method based on binocular stereo matching and 4D connection cost body, and creatively proposes to establish a cross-spatial scale fusion similarity filtering weight matrix by using the FlexiViT visual backbone large model in a multi-geometric scale feature slice sequence, and then use it to filter the 4D connection cost body, thereby suppressing the spatial redundant information and strengthening the stereo matching related information, so as to improve the accuracy and efficiency of subsequent cost body aggregation and disparity map regression prediction. At the same time, the embodiment of the present invention adopts the fusion method of RGB and active infrared spectrum mode to aggregate binocular left and right view features in the deep learning model as the basis for the 4D cost body connection construction, and as the basis for the construction of multi-geometric scale correlation filtering weights, to perform binocular stereo matching based on multi-modal fusion features, rather than the traditional method of matching the same-name points by image pixel points or their neighborhood statistics, thereby achieving a more accurate matching effect than a single RGB image or infrared spectrum image.

[0092] In summary, the embodiment of the present invention obtains two groups of RGB images and infrared spectrum images by shooting with a binocular camera composed of two groups of RGB cameras / infrared cameras, and performs feature fusion on the two groups of images respectively to obtain corresponding fused feature images and corresponding feature map slice sequences. By fusing the two spectral picture features of RGB visible light and infrared, a more accurate, reliable and stable image output is achieved compared with a single RGB image or infrared spectrum image, so as to improve the accuracy and reliability of subsequent similarity matching and disparity prediction; then, the present invention performs similarity matching on the slices at the same position in the first feature map slice sequence and the second feature map slice sequence. The similarity is predicted to obtain a similarity sequence, and a similarity filter map is generated according to the similarity sequence, which is used for subsequent filtering of a 4D cost body constructed according to the first fusion feature map and the second fusion feature map, thereby suppressing spatial redundant information in the 4D cost body and strengthening stereo matching related information; further, the present invention generates an accurate and reliable disparity regression prediction map according to the filtered 4D cost body, and the disparity regression prediction map is obtained based on the fusion spectrum picture features of RGB and infrared, and the prediction process does not need to rely on manual design. Therefore, the present invention improves the stability of disparity calculation in a binocular stereo system, reduces the labor cost in the process, and improves efficiency.

[0093] The equipment used in the disparity prediction method of the embodiment of the present invention is low-cost, highly feasible, and highly efficient. The hardware and operation methods for measuring and reconstructing the scene model and volume are simple and easy, which can effectively reduce the cost and feasibility threshold of stereo measurement in related industrial application fields, and help overcome the problems of high labor costs and low implementation efficiency.

[0094] The parallax prediction method of the embodiment of the present invention can be used in relatively common industrial scene / component local 3D reconstruction application scenarios, such as liquor, brewing, petroleum, chemical, food and other typical application scenarios that require internal volume measurement and 3D space modeling of massive large and medium-sized industrial tanks, etc. At the same time, it can also be expanded to be applied to physical measurement and quality inspection of large industrial parts, and can be further combined with intelligent industrial Internet of Things and related platform technologies to help enterprises realize the digitization of industrial measurement assets and promote the exploration and promotion of new industrial applications such as intelligent volume measurement.

[0095] Continue to refer to Figure 7 A specific embodiment of the disparity prediction method of the present invention is given:

[0096] 1) The left and right RGB image channels and the infrared spectrum image channel of the registered binocular camera are fused independently to form a fused feature image. The infrared image can be an active infrared spectrum image with a pseudo-random infrared pattern projection, or a passive infrared spectrum image; the RGB image can be an RGB image converted to the left and right camera perspectives through homography transformation, or an RGB image independently collected from the left and right perspectives. The two types of channels, RGB and infrared spectrum (plus active infrared structured light), are used as input signals for multimodal stereo matching to combat adverse environmental factors such as dimness, reflection, and texturelessness, and to jointly improve the accuracy of feature matching in disparity analysis. The fusion method is to merge the two types of channels together in the channel dimension to form an input tensor of the shape of (4, H, W), where the first dimension is 3 RGB pixel values ​​plus 1 infrared reflection pixel value, and H and W are the height and width of the RGB image and infrared spectrum image, respectively. Each channel is normalized according to its own mean and variance.

[0097] 2) Feature extraction is performed on the two-way fused input image. In this embodiment, EfficientNet is directly used as the feature extraction backbone network for the fused input image, and the necessary backend is added. Note that the two-way feature extraction operations are independent and parallel, and the weight-shared EfficientNet is used for feature analysis. The final output is in the form of a two-way fused feature map, which is used as the input of submodule 3 in the form of a three-dimensional tensor of (F, H / 2, W / 2), where F is the number of output image feature dimensions. The fused feature map can also be expressed as a series of non-overlapping feature map slice sequences of size P. In this embodiment, the value of P can be 8 geometric scales such as [4, 8, 12, 16, 24, 32, 48, 64]. The fused feature map is expressed as a switching sequence of 8 different switching sizes, where the shape of the feature slice in each sequence is (F, P, P), and the number of feature slices in the sequence is HW / (4*P 2 ).

[0098] 3) ViT encoder stage. The 8 fused feature map slice sequences of different P sizes are independently subjected to autocorrelation sequence operations by the ViT encoder. In this embodiment, the FlexiViT encoder is used as the encoder backbone network to support the input of slice map sequences of different geometric sizes. It is noted that the processing of the 8 fused feature map slice sequences from the left and right views constitutes two independent network branches, and the network structures through which the branches pass are all weight-shared.

[0099] 4) ViT decoder stage. Slice sequences of different geometric scales will be decoded again by the ViT decoder. Note that the decoder has two inputs, which are the self-attention coding sequences of the upper ViT encoder of the same slice size level from the left and right views. The left and right feature sequences of different slice geometric scales are cross-correlated, and finally a slice similarity metric sequence (i.e., similarity sequence) of the same length as the input feature map slice sequence is output. There are a total of 8 slice similarity metric sequences, and the similarity metric value is a scalar.

[0100] 5) Each slice similarity metric value sequence can adopt a slice arrangement order similar to the corresponding input feature slice sequence, and upsample within the scale range of each slice and rearrange it into an image plane form to form 8 similarity metric plane tensors, each of which has a shape of (1, H / 2, W / 2). Finally, they are spliced ​​into a multi-scale similarity volume tensor of (8, H / 2, W / 2), followed by a 1D convolution to fuse the 8 components across multiple geometric scales in the first channel into a comprehensive similarity scalar, and finally output a fused similarity filter map W of the shape of (1, H / 2, W / 2);

[0101] 6) In submodule 3, first, according to the left and right fusion feature maps output by submodule 1, according to the D-level disparity offset range, the feature channels are staggered and merged to generate a 4D connection cost volume C of shape (2*F, D, H / 2, W / 2), which represents the potential disparity deviation degree of the left and right fusion feature views at a specific view position. In this embodiment, the value of D is 64.

[0102] 7) Use the fused similarity filter graph W to filter the redundant information in the 4D cost volume C, compress and enhance its similarity expression ability, and obtain the filtered 4D cost volume C', whose i-th channel value is calculated as follows:

[0103] C'(i)=W⊙C(i)

[0104] The ⊙ represents element-by-element multiplication, that is, the scale fusion similarity weights in W are applied to all channels of the 4D cost volume before filtering.

[0105] 8) After filtering, a 4D cost volume C' is obtained. A U-shaped network structure with a bypass branch mechanism in the form of SegNet is adopted. 3D convolution and deconvolution operations are used to perform encoding and decoding operations on C' to achieve cost aggregation.

[0106] 9) The result after cost aggregation is upsampled, enlarged and adjusted to the same resolution as the original image of the reference perspective (i.e., left perspective), and the corresponding regression prediction operation is performed, and finally the corresponding disparity prediction map of the reference perspective is regressed and output.

[0107] This embodiment also provides a disparity prediction device, which can be integrated into a disparity prediction device. For example, a pixel array of an image sensor chip includes a photosensitive area and an optical dark area, such as Figure 8 As shown, the disparity prediction device may include:

[0108] A feature fusion unit 801 is used to obtain a first image group and a second image group obtained by shooting a target object with a preset binocular camera, and perform feature fusion on the first image group and the second image group respectively to obtain a first fused feature image and a second fused feature image, as well as a first feature map slice sequence corresponding to the first fused feature map and a second feature map slice sequence corresponding to the second fused feature map, wherein each camera group in the binocular camera includes an RGB camera and an infrared camera, the first image group and the second image group both include an RGB image and an infrared spectrum image, and the slice sizes of the first feature map slice sequence and the slice sizes of the second feature map slice sequence at the same position are equal;

[0109] A similarity prediction unit 802 is used to predict the similarity of slices at the same position in the first feature map slice sequence and the second feature map slice sequence to obtain a similarity sequence, and generate a similarity filter map according to the similarity sequence;

[0110] A cost volume construction unit 803 is used to construct a 4D cost volume to be filtered according to the first fused feature map and the second fused feature map;

[0111] A filtering unit 804 is used to filter the 4D cost volume to be filtered according to the similarity filtering graph to obtain a filtered 4D cost volume;

[0112] The prediction unit 805 is used to generate a disparity regression prediction map according to the filtered 4D cost volume, wherein the value of each pixel in the disparity regression prediction map is a predicted disparity value corresponding to a corresponding pixel in a reference image, and the reference image is any image in the first image group and the second image group.

[0113] like Fig. 9 As shown, Fig. 9A schematic diagram of the structure of a disparity prediction device provided by an embodiment of the present invention. The disparity prediction device 1100 includes a processor 1101 having one or more processing cores, a memory 1102 having one or more computer-readable storage media, and a computer program stored in the memory 1102 and executable on the processor. The processor 1101 is electrically connected to the memory 1102. Those skilled in the art will appreciate that the structure of the disparity prediction device shown in the figure does not constitute a limitation on the disparity prediction device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0114] The processor 1101 is the control center of the disparity prediction device 1100, and uses various interfaces and lines to connect various parts of the entire disparity prediction device 1100, and executes various functions and processes data of the disparity prediction device 1100 by running or loading software programs and / or units stored in the memory 1102, and calling data stored in the memory 1102, so as to monitor the disparity prediction device 1100 as a whole. The processor 1101 can be a processor CPU, a graphics processor GPU, a network processor (Network Processor, NP), etc., and can implement or execute various methods, steps and logic block diagrams disclosed in the embodiments of the present invention.

[0115] In an embodiment of the present invention, the processor 1101 in the disparity prediction device 1100 will load the instructions corresponding to the processes of one or more applications into the memory 1102 according to the following steps, and the processor 1101 will run the applications stored in the memory 1102 to implement various functions. Please refer to the previous embodiments and will not repeat them here.

[0116] Optional, such as Fig. 9 As shown, the disparity prediction device 1100 further includes: a touch screen 1103, a radio frequency circuit 1104, an audio circuit 1105, an input unit 1106, and a power supply 1107. The processor 1101 is electrically connected to the touch screen 1103, the radio frequency circuit 1104, the audio circuit 1105, the input unit 1106, and the power supply 1107, respectively. Those skilled in the art can understand that Fig. 9 The structure of the disparity prediction device shown in the figure does not constitute a limitation on the disparity prediction device, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0117] The touch display screen 1103 can be used to display a graphical user interface and receive operation instructions generated by the user acting on the graphical user interface. The touch display screen 1103 may include a display panel and a touch panel. Among them, the display panel can be used to display information input by the user or information provided to the user and various graphical user interfaces of the disparity prediction device, which can be composed of graphics, text, icons, videos and any combination thereof. Optionally, the display panel can be configured in the form of a liquid crystal display (LCD, Liquid Crystal Display), an organic light-emitting diode (OLED, Organic Light-EmittingDiode), etc. The touch panel can be used to collect the user's touch operation on or near it (such as the user using any suitable object or accessory such as a finger, stylus, etc. on the touch panel or near the touch panel), and generate corresponding operation instructions, and the operation instructions execute the corresponding program. Optionally, the touch panel may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 1101, and can receive the command sent by the processor 1101 and execute it. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it is transmitted to the processor 1101 to determine the type of touch event, and then the processor 1101 provides corresponding visual output on the display panel according to the type of touch event. In an embodiment of the present invention, the touch panel and the display panel can be integrated into the touch display screen 1103 to realize the input and output functions. However, in some embodiments, the touch panel and the touch panel can be used as two independent components to realize the input and output functions. That is, the touch display screen 1103 can also be used as a part of the input unit 1106 to realize the input function.

[0118] The radio frequency circuit 1104 may be used to transmit and receive radio frequency signals, so as to establish wireless communication with a network device or other disparity prediction devices through wireless communication, and to transmit and receive signals with the network device or other disparity prediction devices.

[0119] The audio circuit 1105 can be used to provide an audio interface between the user and the disparity prediction device through a speaker and a microphone. The audio circuit 1105 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 1105 and converted into audio data, and then the audio data is output to the processor 1101 for processing, and then sent to another disparity prediction device through the radio frequency circuit 1104, or the audio data is output to the memory 1102 for further processing. The audio circuit 1105 may also include an earphone jack to provide communication between an external earphone and the disparity prediction device.

[0120] The input unit 1106 may be used to receive input numbers, character information or user feature information (such as fingerprint, iris, facial information, etc.), and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0121] The power supply 1107 is used to supply power to various components of the disparity prediction device 1100. Optionally, the power supply 1107 can be logically connected to the processor 1101 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 1107 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0122] although Fig. 9 Not shown in the figure, the disparity prediction device 1100 may also include a camera, a sensor, a wireless fidelity unit, a Bluetooth unit, etc., which will not be repeated here.

[0123] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0124] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0125] To this end, an embodiment of the present invention provides a computer-readable storage medium, in which a plurality of computer programs are stored, and the computer program can be loaded by a processor to execute any disparity prediction method provided by the embodiment of the present invention. The computer program can execute the steps of the aforementioned disparity prediction method, which can be referred to the previous embodiment and will not be repeated here.

[0126] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0127] Since the computer program stored in the computer-readable storage medium can execute any disparity prediction method provided in the embodiments of the present invention, the beneficial effects that can be achieved by any disparity prediction method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0128] In the above-mentioned disparity prediction device, computer-readable storage medium, disparity prediction equipment, and computer program product embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process and beneficial effects of the above-mentioned disparity prediction device, computer-readable storage medium, computer program product, disparity prediction device, and corresponding units can refer to the description of the disparity prediction method in the above embodiment, and will not be repeated here.

[0129] The above is a detailed introduction to a disparity prediction method, a disparity prediction device, a disparity prediction apparatus, a computer-readable storage medium and a computer program product provided in an embodiment of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A disparity prediction method, characterized in that: The method comprises: Acquire a first image group and a second image group obtained by photographing a target object with a preset binocular camera, and perform feature fusion on the first image group and the second image group respectively to obtain a first fused feature image and a second fused feature image, as well as a first feature map slice sequence corresponding to the first fused feature map and a second feature map slice sequence corresponding to the second fused feature map, wherein each camera group in the binocular camera includes an RGB camera and an infrared camera, the first image group and the second image group both include an RGB image and an infrared spectrum image, and the slice sizes of the first feature map slice sequence and the second feature map slice sequence at the same position are equal; Predicting the similarity of slices at the same position in the first feature map slice sequence and the second feature map slice sequence to obtain a similarity sequence, and generating a similarity filter graph according to the similarity sequence; Constructing a 4D cost volume to be filtered according to the first fused feature map and the second fused feature map; Filtering the to-be-filtered 4D cost volume according to the similarity filter graph to obtain a filtered 4D cost volume; A disparity regression prediction map is generated according to the filtered 4D cost volume, wherein the value of each pixel in the disparity regression prediction map is a predicted disparity value corresponding to a corresponding pixel in a reference image, and the reference image is any image in the first image group and the second image group.

2. The disparity prediction method according to claim 1, wherein: The step of obtaining a first image group and a second image group obtained by photographing a target object with a preset binocular camera includes: The target image is photographed by the binocular camera to obtain the first image group and the second image group, wherein the infrared spectrum images in the first image group and the second image group are active infrared spectrum images with pseudo-random infrared pattern projection.

3. The disparity prediction method according to claim 2, wherein: The performing feature fusion on the first image group and the second image group respectively to obtain a first fused feature image and a second fused feature image comprises: The first image group and the second image group are respectively subjected to feature fusion through a preset feature fusion network, and fusion features are extracted through a preset 2D convolutional neural network to obtain the first fused feature image and the second fused feature image.

4. The disparity prediction method according to claim 3, wherein: The weight of the feature fusion network is the same as the weight of the 2D convolutional neural network.

5. The disparity prediction method according to claim 1, wherein: The predicting the similarity of slices at the same position in the first feature map slice sequence and the second feature map slice sequence to obtain a similarity sequence, and generating a similarity filter graph according to the similarity sequence includes: Predicting the similarity of slices at the same position in the first feature map slice sequence and the second feature map slice sequence through a preset FlexiViT backbone network to obtain the similarity sequence, wherein an attention association head of a cross-view feature map is provided in the FlexiViT backbone network, and the weights of the FlexiViT backbone network and the attention association head are the same; Generate multi-sized similar bodies according to the similarity sequence; The multi-sized similar bodies are processed by 1D convolution to obtain the similarity filter graph.

6. The disparity prediction method according to claim 1, wherein: The step of constructing a 4D cost volume to be filtered according to the first fused feature map and the second fused feature map includes: Interlacing and splicing the first fused feature map and the second fused feature map at different disparity scales to obtain the 4D cost volume to be filtered; or The differences between the first fused feature map and the second fused feature map at corresponding positions are calculated and aggregated at different disparity scales to obtain the 4D cost volume to be filtered.

7. The disparity prediction method according to claim 1, wherein: The generating a disparity regression prediction map according to the filtered 4D cost volume comprises: Based on the filtered 4D cost volume, a codec network structure in the form of SegNet is used to perform codec operations using 3D convolution, and upsampling and 1D convolution feature weighted fusion operations are performed to obtain the disparity regression prediction map; or The minimum cost disparity of the 3D slice at each pixel position in the filtered 4D cost volume is calculated to obtain the disparity regression prediction map.

8. A disparity prediction device, characterized in that: The disparity prediction device comprises: A feature fusion unit is used to obtain a first image group and a second image group obtained by shooting a target object with a preset binocular camera, and perform feature fusion on the first image group and the second image group respectively to obtain a first fused feature image and a second fused feature image, as well as a first feature map slice sequence corresponding to the first fused feature map and a second feature map slice sequence corresponding to the second fused feature map, wherein each camera group in the binocular camera includes an RGB camera and an infrared camera, the first image group and the second image group both include an RGB image and an infrared spectrum image, and the slice sizes of the first feature map slice sequence and the second feature map slice sequence at the same position are equal; A similarity prediction unit, used to predict the similarity of slices at the same position in the first feature map slice sequence and the second feature map slice sequence to obtain a similarity sequence, and generate a similarity filter graph according to the similarity sequence; A cost volume construction unit, configured to construct a 4D cost volume to be filtered according to the first fused feature map and the second fused feature map; A filtering unit, configured to filter the 4D cost volume to be filtered according to the similarity filtering graph to obtain a filtered 4D cost volume; A prediction unit is used to generate a disparity regression prediction map according to the filtered 4D cost volume, wherein the value of each pixel in the disparity regression prediction map is a predicted disparity value corresponding to a corresponding pixel in a reference image, and the reference image is any image in the first image group and the second image group.

9. A disparity prediction device, characterized in that The invention comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the disparity prediction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to perform the steps of the disparity prediction method according to any one of claims 1 to 7.

Images