Parallax prediction model optimization method and system based on similarity replacement, medium, program product and terminal

Abstract

According to the parallax prediction model optimization method and system based on similarity replacement, the medium, the program product and the terminal provided by the invention, the similarity replacement operation is added in the parallax prediction model to optimize and adjust the initial parallax label, so that the obtained target parallax label is closer to the real parallax. And retraining the parallax prediction model by taking the optimized high-quality parallax label as training data. Due to the fact that the quality of training data is improved, the model can better learn features and rules of parallax prediction, and therefore higher prediction performance and effect are shown in practical application.

Description

Technical Field

[0001] The present application relates to the field of disparity prediction technology, and in particular to a disparity prediction model optimization method, system, medium, program product, and terminal based on similarity replacement. Background Art

[0002] Binocular stereo matching is a hot topic in computer vision, with widespread applications in a wide range of scenarios, including distance perception, remote sensing, and autonomous driving. The basic principle of binocular vision is to use two parallel cameras to capture images. The difference (disparity) between the two images generates a depth map, which in turn provides information about the distance to obstacles. Data collected by binocular cameras in the wild lacks true disparity labels. This is often done by training large models on publicly available datasets to create pre-trained models. These pre-trained models are then used to predict and output disparity labels for the real-time field data.

[0003] However, due to the complexity of field scenes and the limitations of binocular camera data collection, the disparity labels generated by traditional pre-training models often have errors. For example, the public datasets used in the training process of large models cannot cover all possible scenes, and the generated disparity labels are difficult to achieve the accuracy of the labels generated by lidar in terms of details. Figure 1 In the field data collected, the disparity labels generated by the pre-trained model fluctuate in disparity values for objects at long distances with small disparity at the same depth. This indicates that the disparity labels are inaccurate. Using these poor-quality disparity labels to further train the disparity prediction model would significantly impact its performance and effectiveness. Summary of the Invention

[0004] In view of the above-mentioned shortcomings of the prior art, the present invention provides a disparity prediction model optimization method, system, medium, program product and terminal based on similarity replacement, which is used to solve the problem that the pseudo-disparity labels generated by the pre-training model of the prior art have errors and the training effect of the subsequent disparity prediction model is poor.

[0005] To achieve the above-mentioned and other related objectives, the first aspect of the present application provides a disparity prediction model optimization method based on similarity replacement, comprising: obtaining a left view and a right view captured by a binocular camera; utilizing a pre-trained disparity label generation model to obtain an initial disparity label based on the captured left and right views; obtaining a disparity prediction result based on the captured left and right views based on a constructed disparity prediction model; wherein the disparity prediction model includes a feature extraction module and a similarity replacement module; inputting the left and right views into the feature extraction module for feature extraction to obtain a left feature map and a right feature map; inputting the initial disparity label into the similarity replacement module, combining the left and right feature maps, and performing a similarity replacement operation to obtain a target disparity label; calculating a loss function based on the target disparity label and the disparity prediction result, and updating the parameters of the disparity prediction model according to the loss function value to obtain an optimized disparity prediction model.

[0006] In some embodiments of the first aspect of the present application, the initial disparity label is input into the similarity replacement module, and a similarity replacement operation is performed in combination with the left feature map and the right feature map. The process of obtaining the target disparity label includes: taking each pixel point of the left feature map as the first sample data set, and taking each pixel point of the right feature map as the second sample data set; calculating the similarity of the first sample data set and the second sample data set pixel by pixel based on a preset similarity search radius; and performing a similarity replacement operation on the initial disparity label according to the calculated similarity result to obtain the target disparity label.

[0007] In some embodiments of the first aspect of the present application, the process of calculating the similarity of a first sample data set and a second sample data set pixel by pixel based on a preset similarity search radius includes: obtaining any pixel point in the first sample data set, and finding a pixel point with the same name corresponding to the current pixel point in the second sample data set; obtaining pixels within the preset similarity search radius from the second sample data set with the pixel point with the same name in the second sample data set as the center; constructing a set of pixels in the second sample data set corresponding to the current pixel point in the first sample data set based on the pixel points with the same name in the second sample data set and the pixels within the preset similarity search radius; and using a dot product algorithm to calculate the similarity value between the current pixel point in the first sample data set and each pixel point in the corresponding pixel point set.

[0008] In some embodiments of the first aspect of the present application, a similarity replacement operation is performed on the initial disparity label based on the calculated similarity result, and the process of obtaining the target disparity label includes: obtaining the similarity calculation result between each pixel point in the first sample data set and each pixel point in the corresponding pixel point set in the second sample data set; comparing the similarity values between the current pixel point in the first sample data set and each pixel point in the corresponding pixel point set; performing a similarity replacement operation on the initial disparity label based on the current pixel point in the first sample data set and the pixel point in the pixel point set corresponding to the maximum value in the similarity value to obtain the target disparity label.

[0009] In some embodiments of the first aspect of the present application, a similarity replacement operation is performed on the initial disparity label based on the current pixel point in the first sample data set and the pixel point in the set of pixel points corresponding to the maximum value in the similarity value to obtain the target disparity label. The process includes: if the current pixel point in the first sample data set and the pixel point in the set of pixel points corresponding to the maximum value in the similarity value are the same-name points, then the label at the position corresponding to the current pixel point in the initial disparity label does not need to be replaced; if the current pixel point in the first sample data set and the pixel point in the set of pixel points corresponding to the maximum value in the similarity value are not the same-name points, then the label at the position corresponding to the current pixel point in the initial disparity label is replaced with the label at the position corresponding to the pixel point corresponding to the maximum value in the similarity value to obtain the target disparity label.

[0010] In some embodiments of the first aspect of the present application, the method further includes: optimizing parameters of the similarity replacement module based on a target loss function, and iteratively updating the similarity replacement module; the target loss function is calculated as follows: Among them, l is the left feature map, r is the right feature map, d is the disparity value, (u, v) is the coordinate position of the pixel point on the left feature map, (ud, v) is the coordinate position of the point on the right feature map corresponding to the pixel point (u, v) on the left feature map, and n is the number of samples in the right feature map.

[0011] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides a disparity prediction model optimization system based on similarity replacement, including: a left and right view acquisition module for acquiring the left view and the right view captured by a binocular camera; an initial disparity label acquisition module for using a pre-trained disparity label generation model to obtain an initial disparity label according to the captured left view and the right view; a disparity prediction model construction module for obtaining a disparity prediction result according to the captured left view and the right view based on the constructed disparity prediction model; wherein the disparity prediction model includes a feature extraction module and a similarity replacement module; a left and right feature map acquisition module for inputting the left view and the right view into the feature extraction module for feature extraction to obtain a left feature map and a right feature map; a target disparity label acquisition module for inputting the initial disparity label into the similarity replacement module, combining the left feature map and the right feature map, performing a similarity replacement operation, and obtaining a target disparity label; a disparity prediction model optimization module for calculating a loss function based on the target disparity label and the disparity prediction result, updating the parameters of the disparity prediction model according to the loss function value, and obtaining an optimized disparity prediction model.

[0012] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the disparity prediction model optimization method based on similarity replacement when executed by a processor.

[0013] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides a computer program product, which includes computer program code. When the computer program code is run on a computer, the computer implements the disparity prediction model optimization method based on similarity replacement.

[0014] To achieve the above-mentioned purpose and other related purposes, the fifth aspect of the present application provides an electronic terminal, including a memory, a processor and a computer program stored in the memory; the processor executes the computer program to implement the disparity prediction model optimization method based on similarity replacement.

[0015] As described above, the disparity prediction model optimization method, system, medium, program product, and terminal provided by this application based on similarity replacement have the following beneficial effects:

[0016] This application optimizes the initial disparity labels by adding a similarity replacement operation to the disparity prediction model, resulting in target disparity labels that are closer to the true disparity. The optimized, high-quality disparity labels serve as training data for retraining the disparity prediction model. This improved training data allows the model to better learn the characteristics and patterns of disparity prediction, resulting in higher prediction performance and effectiveness in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Shown is a disparity label prediction effect diagram in the prior art in one embodiment of the present application.

[0018] Figure 2 Shown is a flow chart of a disparity prediction model optimization method based on similarity replacement according to an embodiment of the present application.

[0019] Figure 3 Shown is a schematic diagram of a process for obtaining a target disparity label according to an embodiment of the present application.

[0020] Figure 4 Shown is a flowchart of calculating similarity pixel by pixel in one embodiment of the present application.

[0021] Figure 5 Shown is a flowchart of a similarity replacement operation in one embodiment of the present application.

[0022] Figure 6 FIG. 1 is a schematic diagram showing the structure of a disparity prediction model according to an embodiment of the present application.

[0023] Figure 7 Shown is a structural diagram of a disparity prediction model optimization system based on similarity replacement in one embodiment of the present application.

[0024] Figure 8 Shown is a structural schematic diagram of an electronic terminal in one embodiment of the present application. DETAILED DESCRIPTION

[0025] The following describes the embodiments of the present application through specific examples. Those skilled in the art can easily understand the other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0026] Before further explaining the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are subject to the following interpretations:

[0027] Binocular stereo vision: A method based on the principle of parallax that uses imaging equipment to obtain two images of the object being measured from different positions, and then calculates the positional deviation between corresponding points in the images to obtain the three-dimensional geometric information of the object.

[0028] To facilitate understanding of the embodiments of this application, first Figure 2 and Figure 6 Detailed description. Figure 2 A schematic flow chart of a method for optimizing a disparity prediction model based on similarity replacement in an embodiment of the present invention is shown. The method in this embodiment includes:

[0029] Step S11: Obtain the left view and right view captured by the binocular camera.

[0030] It should be noted that this embodiment uses a binocular camera to capture left and right views of the object or scene to be measured based on the principle of binocular stereo vision. A binocular camera consists of two parallel cameras that capture the same scene from different positions, producing two slightly different images (left and right). The left and right views refer to images of the same scene captured from two different perspectives (usually horizontal). The difference between the left and right views is the parallax, which can be calculated to subsequently infer depth information of objects in the scene.

[0031] Step S12: using a pre-trained disparity label generation model, an initial disparity label is obtained according to the collected left view and right view.

[0032] like Figure 6 As shown, a large model or neural network is used to construct a disparity label generation model, which is pre-trained to obtain a pre-trained disparity label generation model. The pre-trained disparity label generation model inputs the left and right views captured in real time by the binocular camera, and outputs the initial disparity labels corresponding to the left and right views.

[0033] Step S13: Based on the constructed disparity prediction model, a disparity prediction result is obtained from the captured left and right views. The disparity prediction model includes a feature extraction module and a similarity replacement module. A deep learning neural network is used to construct the disparity prediction model. The type of deep learning neural network is not limited in this embodiment. The pre-trained disparity label generation model can have the same or different structure as the disparity prediction model, and the present invention is not limited to this.

[0034] Combine Figure 6 The disparity prediction model includes a feature extraction module and a similarity replacement module. The feature extraction module is used to extract features from the input left and right views, providing a basis for subsequent disparity calculation. The feature extraction module performs feature extraction through a weight-shared twin convolutional neural network.

[0035] In some examples, such as Figure 6As shown in the figure, the feature extraction module consists of one convolutional layer and four residual layers, capable of outputting multi-scale feature maps. The convolutional layer transforms the number of channels in the input image to extract basic image features. The convolutional layer is set to a stride of 1 and a channel count of 32. The residual layer performs convolution, normalization, and regularization operations for deeper feature extraction.

[0036] Step S14: Input the left view and the right view into the feature extraction module for feature extraction to obtain a left feature map and a right feature map.

[0037] Specifically, if Figure 6 As shown, the feature extraction module is used to transform the number of channels of the input left view and right view through one convolution layer, and then perform convolution, normalization and regularization operations through four residual layers, and finally output a left feature map corresponding to the left view and a right feature map corresponding to the right view. The left feature map and the right feature map are multi-scale feature maps. For example, the left feature map and the right feature map both include feature maps of three different resolutions of downsampling 4 times, 8 times and 16 times. The left feature map and the right feature map can also be feature maps of other sizes, which is not limited in this embodiment.

[0038] Step S15: inputting the initial disparity label into the similarity replacement module, combining the left feature map and the right feature map, performing a similarity replacement operation, and obtaining a target disparity label.

[0039] The initial disparity label is a preliminary disparity map output by the disparity label generation model. Although the initial disparity label already contains certain depth information, it may contain noise or inaccurate disparity values, especially in areas with unclear texture or occlusion. The initial disparity label is fine-tuned through the similarity replacement operation of the similarity replacement module to generate a more accurate target disparity label. The target disparity label has higher accuracy and can more accurately reflect the depth information of objects in the scene. The target disparity label is used for subsequent training of the disparity prediction model, which helps to improve the performance and generalization ability of the model.

[0040] In one embodiment, if Figure 3 As shown, the initial disparity label is input into the similarity replacement module, and the similarity replacement operation is performed on the left feature map and the right feature map. The process of obtaining the target disparity label includes:

[0041] Step S151: each pixel point of the left feature map is used as the first sample data set, and each pixel point of the right feature map is used as the second sample data set.

[0042] It should be explained that if the accuracy of the obtained disparity label is high, the similarity of the feature maps corresponding to the left view and the right view is also high, that is, the similarity between the same-name points of the left feature map and the right feature map is high, and the same-name points of the left view and the right view also match each other. Therefore, if it is necessary to determine the accuracy of the disparity label, it is confirmed by calculating the similarity of the left feature map and the right feature map. Calculating the similarity of the left feature map and the right feature map is essentially calculating the similarity of each pixel of the left feature map and each pixel of the right feature map. Taking each pixel of the left feature map as the first sample data set and each pixel of the right feature map as the second sample data set, the similarity between the first sample data set and the second sample data set is calculated.

[0043] Step S152: Calculating similarity between the first sample data set and the second sample data set pixel by pixel based on a preset similarity search radius.

[0044] Furthermore, calculating the similarity between the first sample data set and the second sample data set is to calculate the similarity between each pixel of the first sample data set and each pixel of the right feature map. However, if the similarity is calculated in sequence between each pixel of the first sample data set and each pixel of the second sample data set, specifically, a pixel of the first sample data set is randomly selected, and the similarity is calculated in sequence between the pixel and each pixel of the second sample data set, and the above operation is performed on each pixel of the first sample data set, the amount of calculation is large and the efficiency is low. The similarity between the current pixel of the first sample data set and a non-identical point in the second sample data set that is particularly far away is definitely low, and there is no need to calculate the similarity.

[0045] Therefore, it is only necessary to filter out the pixel points around the point with the same name as the current pixel point in the first sample data set in the second sample data set and perform similarity calculation. Figure 4 As shown, the specific process is as follows:

[0046] Step S1521: obtaining any pixel point in the first sample data set, and finding a pixel point with the same name as the current pixel point in the second sample data set;

[0047] Step S1522: taking the pixel point with the same name in the second sample data set as the center, obtaining the pixel points within the preset similarity search radius from the second sample data set;

[0048] Step S1523: constructing a set of pixels in the second sample data set corresponding to the current pixel in the first sample data set based on the pixels with the same name in the second sample data set and the pixels within the preset similarity search radius;

[0049] Step S1524: using a dot product algorithm to calculate similarity values between the current pixel point in the first sample data set and each pixel point in the corresponding pixel point set.

[0050] For example, for any pixel point A in the first sample data set, find the pixel point A1 with the same name corresponding to the current pixel point A in the second sample data set. Then, with the pixel point A1 with the same name in the second sample data set as the center, search for other pixel points (such as B1, C1, D1) within the radius according to the preset similarity search radius r in the second sample data set. A pixel point set is formed based on the pixel point A1 with the same name in the second sample data set and other pixel points within the preset similarity search radius r. The pixel point set is (A1, B1, C1, D1). Calculate the similarity values between the current pixel point A in the first sample data set and each pixel point (A1, B1, C1, D1) in the pixel point set, that is, the similarity value between A and A1, the similarity value between A and B1, the similarity value between A and C1, and the similarity value between A and D1. The dot product algorithm is preferably used as the similarity calculation method.

[0051] For each pixel in the first sample dataset, a preset similarity search radius is used to filter out pixels in the second sample dataset that are within a radius of each pixel in the first sample dataset. Similarity is calculated for each pixel in the first sample dataset with the selected set of pixels in the second sample dataset. This effectively calculates the similarity between the first and second sample datasets, while avoiding unnecessary calculations and improving computational efficiency.

[0052] Combine Figure 2 Note: The left feature map and the right feature map extracted by the feature extraction module are multi-scale feature maps. When the preset similarity search radius is used to search for pixels, feature maps of different scales are searched according to the preset similarity search radius r. When the left feature map and the right feature map are both feature maps of three different scales, downsampled by 4 times, 8 times, and 16 times, the feature maps of different scales in the right feature map are searched based on the same preset similarity search radius r. At this time, the number of pixels searched for feature maps of different scales is different. For example, in feature maps with higher resolution, the number of pixels covered by the preset similarity search radius r will be more than that in feature maps with lower resolution. Due to the different resolutions of feature maps of different scales, the number of pixels obtained during the search process is different. By iteratively searching feature maps of different scales, the number of pixels is reduced, that is, the number of pixels that need to be processed is reduced, thereby optimizing the efficiency of subsequent similarity calculations.

[0053] Step S153: performing a similarity replacement operation on the initial disparity label according to the calculated similarity result to obtain a target disparity label.

[0054] Based on the similarity calculation results between the first sample data set and the second sample data set, the feature consistency between the left feature map and the right feature map can be determined. According to the relevance sorting of the similarity calculation results, the wrong labels in the initial disparity labels can be found and replaced to obtain accurate labels. Figure 5 As shown, the specific similarity replacement operation process is as follows:

[0055] Step S1531: obtaining similarity calculation results between each pixel point in the first sample data set and each pixel point in the corresponding pixel point set in the second sample data set;

[0056] Step S1532: comparing similarity values between the current pixel in the first sample data set and each pixel in the corresponding pixel set;

[0057] Step S1533: performing a similarity replacement operation on the initial disparity label according to the current pixel in the first sample data set and the pixel in the pixel set corresponding to the maximum similarity value to obtain a target disparity label.

[0058] It should be noted that, based on a preset similarity search radius, a set of pixel points corresponding to each pixel point in the first sample data set can be screened in the second sample data set, and a similarity value is further calculated between each pixel point in the first sample data set and each pixel point in its corresponding set of pixel points. The similarity values between the current pixel point in the first sample data set and each pixel point in its corresponding set of pixel points are compared, i.e., the similarity values are sorted to determine a maximum similarity value, and then, based on the maximum similarity value, the pixel point in the pixel set corresponding to the current pixel point in the first sample data set is determined to be the pixel point corresponding to the maximum value obtained by calculating the maximum similarity value.

[0059] For example, compare the similarity values between the current pixel point A in the first sample data set and each pixel point (A1, B1, C1, D1) in the pixel point set, that is, the similarity value between A and A1, the similarity value between A and B1, the similarity value between A and C1, and the similarity value between A and D1. Assuming that the similarity value between A and B1 is the largest after sorting, the pixel point in the pixel point set corresponding to the maximum value in the similarity value is B1.

[0060] Furthermore, based on the current pixel in the first sample data set and the pixel in the pixel set corresponding to the maximum value of the similarity value obtained by comparison, a similarity replacement operation is performed on the initial disparity label to obtain the target disparity label. The specific process includes:

[0061] If the current pixel in the first sample data set and the pixel in the pixel set corresponding to the maximum value in the similarity value have the same name, the label at the position corresponding to the current pixel in the initial disparity label does not need to be replaced.

[0062] For example, suppose that the similarity values between the current pixel point A in the first sample data set and each pixel point (A1, B1, C1, D1) in the pixel point set are compared, that is, the similarity value between A and A1, the similarity value between A and B1, the similarity value between A and C1, and the similarity value between A and D1. If the similarity value between A and A1 is the largest after sorting, then the pixel point in the pixel point set corresponding to the maximum value in the similarity value is A1. Pixel point A1 is the same-name point in the right feature map that corresponds to the current pixel point A in the left feature map, indicating that the features of the left feature map and the right feature map are consistent at this time, and the label corresponding to the position of pixel point A in the initial disparity label is accurate, and no replacement operation is required.

[0063] A homonymous point is a pixel that corresponds to the same point in physical space in both the left and right views. For example, an object may appear at one location in the left view and at another location in the right view due to the change in viewing angle. However, these are projections of the same object and are therefore called homonymous points.

[0064] If the current pixel point in the first sample data set and the pixel point in the set of pixel points corresponding to the maximum value in the similarity value are not points with the same name, the label of the position corresponding to the current pixel point in the initial disparity label is replaced with the label of the position corresponding to the pixel point corresponding to the maximum value in the similarity value to obtain the target disparity label.

[0065] For example, suppose that the similarity values between the current pixel point A in the first sample data set and each pixel point (A1, B1, C1, D1) in the pixel point set are compared, that is, the similarity value between A and A1, the similarity value between A and B1, the similarity value between A and C1, and the similarity value between A and D1. If the similarity value between A and B1 is the largest after sorting, then the pixel point in the pixel point set corresponding to the maximum value in the similarity value is B1. At this time, the current pixel point A in the left feature map and the pixel point B1 in the right feature map are not homonymous points, then the label of the corresponding position of the current pixel point A in the initial disparity label is replaced with the label of the corresponding position of the pixel point B1 corresponding to the maximum value in the similarity value. Specifically, the corresponding position of pixel point A and the corresponding position of pixel point B1 are found in the initial disparity label, and the label of the corresponding position of pixel point A in the initial disparity label is replaced with the label of the corresponding position of pixel point B1.

[0066] It should be noted that the similarity calculation between the left feature map and the right feature map is performed for each pixel in the initial disparity label, and then whether to perform a similarity replacement operation is determined based on the calculation result. After completing the similarity replacement operation, the target disparity label after the initial disparity label is optimized is obtained.

[0067] Step S16: Calculating a loss function based on the target disparity label and the disparity prediction result, and updating the parameters of the disparity prediction model according to the loss function value to obtain an optimized disparity prediction model. The loss function adopts a smooth L1 loss function.

[0068] During disparity prediction model training, the target disparity label is obtained through the similarity replacement module. This label provides a more accurate disparity value for each pixel. Therefore, it serves as supervisory information for disparity prediction and is used to train the disparity prediction model. A loss function measures the difference between the target disparity label and the disparity prediction result. A smaller loss function value indicates that the disparity prediction model's prediction is closer to the true value.

[0069] In this embodiment, a smooth L1 loss function is used to calculate the loss function value between the target disparity label and the disparity prediction result. Based on the calculated loss function value, the model parameters are updated through the back propagation algorithm. After multiple iterative updates, the model parameters are gradually optimized and the loss function value gradually decreases, and finally an optimized disparity prediction model is obtained. The optimized disparity prediction model can predict the disparity value more accurately.

[0070] In one embodiment, the method further includes step S17: optimizing the parameters of the similarity replacement module based on a target loss function, and iteratively updating the similarity replacement module; the target loss function is calculated as follows:

[0071]

[0072] Among them, l is the left feature map, r is the right feature map, d is the disparity value, (u, v) is the coordinate position of the pixel point on the left feature map, (ud, v) is the coordinate position of the point on the right feature map corresponding to the pixel point (u, v) on the left feature map, and n is the number of samples in the right feature map.

[0073] It should be understood that during the training process of the similarity replacement module, the similarity value between the current pixel point of the left feature map and the point of the same name in the right feature map is as large as possible and the distance is as close as possible, and the similarity value between the current pixel point of the left feature map and the non-homologous point of the right feature map is as small as possible and the distance is as far as possible, so as to ensure the consistency of the left feature map and the right feature map.

[0074] During the optimization of the disparity prediction model, the similarity replacement module in the disparity prediction model is also iteratively updated synchronously. The parameters of the similarity replacement module are optimized based on the calculation of the target loss function. The similarity replacement module finally obtained has a higher accuracy in the similarity replacement operation of the initial disparity label, thereby obtaining a more accurate target disparity label, further improving the prediction ability of the disparity prediction model.

[0075] In some examples, the disparity prediction model further includes a disparity estimation module. Figure 6 Description: After the left view and the right view are input into the feature extraction module, the left feature map and the right feature map are output respectively; the left feature map and the right feature map are input into the disparity estimation module, and the disparity prediction result is output after matching cost calculation, cost aggregation and disparity calculation.

[0076] Specifically, in the disparity estimation module, the left feature map and the right feature map are input, and the matching cost is first calculated by correlation to obtain the correlation between the left feature map and the right feature map, and a three-dimensional matrix of size w*h*d is obtained. The specific calculation formula is as follows:

[0077]

[0078] Where s is the scale of the feature map, w and h are the width and height of the left or right feature map, respectively, and D is the disparity search range. This three-dimensional matrix stores the matching cost value corresponding to each disparity within the disparity range for each pixel in the feature map.

[0079] Cost aggregation is performed according to the three-dimensional matrix to obtain aggregated matching costs, wherein the cost aggregation includes matching cost aggregation at the same resolution and matching cost aggregation at different resolutions.

[0080] A softmax operation is performed on the aggregated matching cost (cost_volume) within the disparity range D to obtain the disparity value of each pixel position in the three-dimensional matrix and obtain a disparity prediction map, that is, a disparity prediction result.

[0081] It's important to emphasize that this application optimizes the initial disparity labels by adding a similarity replacement operation to the disparity prediction model, resulting in target disparity labels that are closer to the true disparity. The optimized, high-quality disparity labels serve as training data for retraining the disparity prediction model. This improved training data allows the model to better learn the characteristics and patterns of disparity prediction, resulting in higher prediction performance and effectiveness in practical applications.

[0082] In the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects, and do not limit their order. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity or execution order, and words such as "first" and "second" do not necessarily mean different.

[0083] It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" represent examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0084] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, ab, ac, bc or abc, where a, b, c can be single or multiple.

[0085] Figure 7 Schematic diagram of a disparity prediction model optimization system based on similarity replacement provided by an embodiment of the present application. Figure 7 As shown, the system 700 includes:

[0086] The left and right view acquisition module 710 is used to acquire the left view and the right view captured by the binocular camera;

[0087] An initial disparity label acquisition module 720 is configured to obtain an initial disparity label based on the collected left and right views using a pre-trained disparity label generation model;

[0088] A disparity prediction model construction module 730 is configured to obtain a disparity prediction result based on the constructed disparity prediction model and the collected left and right views; wherein the disparity prediction model includes a feature extraction module and a similarity replacement module;

[0089] A left and right feature map acquisition module 740 is configured to input the left view and the right view into the feature extraction module for feature extraction to obtain a left feature map and a right feature map;

[0090] A target disparity label acquisition module 750 is configured to input the initial disparity label into the similarity replacement module, perform a similarity replacement operation on the left feature map and the right feature map, and obtain a target disparity label;

[0091] The disparity prediction model optimization module 760 is configured to calculate a loss function based on the target disparity label and the disparity prediction result, and update the parameters of the disparity prediction model according to the loss function value to obtain an optimized disparity prediction model.

[0092] It should be understood that the specific process of each module executing the above corresponding steps has been described in detail in the above method embodiment, and for the sake of brevity, it will not be repeated here.

[0093] It should also be understood that the division of modules in the embodiments of the present application is illustrative and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the present application may be integrated into a single processor, or may exist physically separately, or two or more modules may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules.

[0094] Figure 8 : is a schematic block diagram of an electronic terminal provided in an embodiment of the present application. Figure 8 As shown, the electronic terminal includes: at least one processor 801, a memory 802, at least one network interface 803 and a user interface 805. The various components in the device are coupled together via a bus system 804. It is understood that the bus system 804 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 804 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 8 Various buses are labeled as bus systems.

[0095] The user interface 805 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0096] It will be appreciated that the memory 802 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM) or a programmable read-only memory (PROM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0097] The memory 802 in the embodiment of the present invention is used to store various categories of data to support the operation of the electronic terminal 800. Examples of such data include: any executable program for operating on the electronic terminal 800, such as an operating system 8021 and an application 8022; the operating system 8021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 8022 can include various applications, such as a media player (Media Player), a browser (Browser), etc., for implementing various application services. The disparity prediction model optimization method based on similarity replacement provided in the embodiment of the present invention can be included in the application 8022.

[0098] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 801. Processor 801 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in processor 801 or by software instructions. The above processor 801 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 801 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor 801 can be a microprocessor or any conventional processor. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory. The processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0099] In an exemplary embodiment, the electronic terminal 800 may be configured to execute the aforementioned method using one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs).

[0100] According to the method provided in the embodiments of the present application, the present application also provides a computer program product, which includes: computer program code, which, when running on a computer, enables the computer to execute the disparity prediction model optimization method based on similarity replacement of any embodiment of the illustrated embodiments.

[0101] According to the method provided in the embodiments of the present application, the present application also provides a computer-readable storage medium, which stores program code. When the program code is run on a computer, the computer executes the disparity prediction model optimization method based on similarity replacement of any embodiment of the illustrated embodiments.

[0102] As used in this specification, the terms "component," "module," "system," and the like are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. By way of illustration, both an application running on a computing device and a computing device can be a component. One or more components can reside in a process and / or an execution thread, and a component can be located on a computer and / or distributed between two or more computers. In addition, these components can be executed from various computer-readable media having various data structures stored thereon. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component on a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0103] Those skilled in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0104] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0105] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0106] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0107] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0108] In the above embodiments, the functions of each functional unit can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When the computer program instructions (program) are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. Available media may be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., high-density digital video discs (DVDs), or semiconductor media (e.g., solid state disks (SSDs)).

[0109] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other media that can store program codes.

[0110] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0111] In summary, the present application provides a method, system, medium, program product, and terminal for optimizing a disparity prediction model based on similarity replacement. The present application optimizes and adjusts the initial disparity label by adding a similarity replacement operation to the disparity prediction model, so that the obtained target disparity label is closer to the real disparity. The optimized high-quality disparity label is used as training data to retrain the disparity prediction model. Due to the improvement in the quality of the training data, the model can better learn the characteristics and laws of disparity prediction, thereby showing higher prediction performance and effects in practical applications. Therefore, the present application effectively overcomes the various shortcomings of the existing technology and has a high industrial utilization value.

[0112] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical concepts disclosed in this application shall be covered by the claims of this application.

Claims

1. A disparity prediction model optimization method based on similarity replacement, characterized in that: include: Get the left view and right view captured by the binocular camera; Using the pre-trained disparity label generation model, the initial disparity label is obtained based on the collected left and right views; Based on the constructed disparity prediction model, a disparity prediction result is obtained according to the collected left view and right view; wherein the disparity prediction model includes a feature extraction module and a similarity replacement module; Inputting the left view and the right view into the feature extraction module to perform feature extraction to obtain a left feature map and a right feature map; Inputting the initial disparity label into the similarity replacement module, combining the left feature map and the right feature map, performing a similarity replacement operation, and obtaining a target disparity label; A loss function is calculated based on the target disparity label and the disparity prediction result, and the parameters of the disparity prediction model are updated according to the loss function value to obtain an optimized disparity prediction model.

2. The method for optimizing a disparity prediction model based on similarity replacement according to claim 1, wherein: The process of inputting the initial disparity label into the similarity replacement module, combining the left feature map and the right feature map, performing a similarity replacement operation, and obtaining the target disparity label includes: Each pixel point of the left feature map is used as the first sample data set, and each pixel point of the right feature map is used as the second sample data set; Calculating similarity pixel by pixel between the first sample data set and the second sample data set based on a preset similarity search radius; A similarity replacement operation is performed on the initial disparity label according to the calculated similarity result to obtain the target disparity label.

3. The method for optimizing a disparity prediction model based on similarity replacement according to claim 2, wherein: The process of calculating the similarity of the first sample data set and the second sample data set pixel by pixel based on the preset similarity search radius includes: Obtain any pixel point in the first sample data set, and find a pixel point with the same name as the current pixel point in the second sample data set; Taking the pixel with the same name in the second sample data set as the center, obtaining the pixel within the preset similarity search radius from the second sample data set; Based on the pixels of the same name in the second sample data set and the pixels within the preset similarity search radius, a set of pixels in the second sample data set corresponding to the current pixel in the first sample data set is constructed; A dot product algorithm is used to calculate the similarity value between the current pixel point in the first sample data set and each pixel point in the corresponding pixel point set.

4. The method for optimizing a disparity prediction model based on similarity replacement according to claim 2, wherein: The process of performing similarity replacement operation on the initial disparity label based on the calculated similarity result and obtaining the target disparity label includes: Obtaining similarity calculation results between each pixel point in the first sample data set and each pixel point in the corresponding pixel point set in the second sample data set; Comparing similarity values between a current pixel point in the first sample data set and each pixel point in the corresponding pixel point set; A similarity replacement operation is performed on the initial disparity label according to the current pixel in the first sample data set and the pixel in the pixel set corresponding to the maximum similarity value to obtain a target disparity label.

5. The method for optimizing a disparity prediction model based on similarity replacement according to claim 4, wherein: The process of performing a similarity replacement operation on the initial disparity label according to the current pixel in the first sample data set and the pixel in the pixel set corresponding to the maximum similarity value to obtain the target disparity label includes: If the current pixel in the first sample data set and the pixel in the set of pixels corresponding to the maximum value of the similarity value are points with the same name, the label at the position corresponding to the current pixel in the initial disparity label does not need to be replaced; If the current pixel point in the first sample data set and the pixel point in the set of pixel points corresponding to the maximum value in the similarity value are not points with the same name, the label of the position corresponding to the current pixel point in the initial disparity label is replaced with the label of the position corresponding to the pixel point corresponding to the maximum value in the similarity value to obtain the target disparity label.

6. The method for optimizing a disparity prediction model based on similarity replacement according to claim 1, wherein: The method further includes: optimizing parameters of the similarity replacement module based on a target loss function, and iteratively updating the similarity replacement module; the target loss function is calculated as follows: Among them, l is the left feature map, r is the right feature map, d is the disparity value, (u, v) is the coordinate position of the pixel point on the left feature map, (ud, v) is the coordinate position of the point on the right feature map corresponding to the pixel point (u, v) on the left feature map, and n is the number of samples in the right feature map.

7. A disparity prediction model optimization system based on similarity replacement, characterized in that: include: Left and right view acquisition module, used to obtain the left view and right view captured by the binocular camera; An initial disparity label acquisition module is used to obtain an initial disparity label based on the collected left and right views using a pre-trained disparity label generation model; A disparity prediction model construction module, configured to obtain a disparity prediction result based on the constructed disparity prediction model and the collected left and right views; wherein the disparity prediction model includes a feature extraction module and a similarity replacement module; A left and right feature map acquisition module, configured to input the left view and the right view into the feature extraction module for feature extraction to obtain a left feature map and a right feature map; a target disparity label acquisition module, configured to input the initial disparity label into the similarity replacement module, perform a similarity replacement operation on the left feature map and the right feature map, and obtain a target disparity label; The disparity prediction model optimization module is used to calculate a loss function based on the target disparity label and the disparity prediction result, and update the parameters of the disparity prediction model according to the loss function value to obtain an optimized disparity prediction model.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the disparity prediction model optimization method based on similarity replacement according to any one of claims 1 to 6 is implemented.

9. A computer program product, characterized in that The computer program product includes computer program codes, and when the computer program codes are executed on a computer, the computer is enabled to implement the disparity prediction model optimization method based on similarity replacement according to any one of claims 1 to 6.

10. An electronic terminal comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the disparity prediction model optimization method based on similarity replacement according to any one of claims 1 to 6.

Images