Binocular camera stereo matching method, system and intelligent terminal based on loss function

Abstract

The present invention discloses a binocular camera stereo matching method, system and intelligent terminal based on loss function, the method comprising: obtaining the left eye image and the right eye image of the binocular camera; extracting the pixel points of the left eye image and the pixel points of the right eye image, and calculating the single point similarity of each pixel point, and taking the single point similarity as the first loss function; extracting the overall similarity of the left eye image and the right eye image, and taking the maximum value of the overall similarity as the second loss function; performing weighted summation of the first loss function and the second loss function to obtain a joint loss function; and performing stereo matching of the binocular camera based on the joint loss function. The method solves the technical problem of poor result accuracy and generalization ability of the binocular matching method in the prior art due to strong dependence on true value acquisition.

Description

Technical Field

[0001] The present invention relates to the technical field of stereo matching methods, and in particular to a binocular camera stereo matching method, system and intelligent terminal based on a loss function. Background Art

[0002] A binocular camera refers to two cameras that are hard-wired to meet human eye ergonomics. During operation, the binocular camera captures the left and right viewpoint images of the same scene. It is necessary to use the stereo matching method to obtain the disparity map and then the depth map. Therefore, the accuracy of the output results of the stereo matching method of the binocular camera directly affects the effect and accuracy of the depth map.

[0003] In the prior art, a stereo matching method is implemented using a supervised learning deep learning method. This method requires the true distance value of a data set during the training process. In theory, obtaining enough true distance values ​​can implement the supervision scheme. However, first of all, since the method is highly dependent on the scene (including the actual physical scene captured by the camera and the parameters of the lens itself) when obtaining the true distance value, it does not have the ability to generalize. At the same time, since it is necessary to measure the true distance value of each pixel on the image, it is difficult to obtain the true distance value. In addition, during the working process, it is necessary to use sensors such as laser radar to collect true values, which cannot meet the requirements of obtaining the true distance value of each pixel corresponding to the camera. Manual processing needs to be added later, and the fusion of multiple sensors (laser, camera) is also unstable, resulting in poor accuracy of data collection. Summary of the invention

[0004] To this end, the embodiments of the present invention provide a binocular camera stereo matching method, system and intelligent terminal based on a loss function, so as to at least partially solve the technical problem in the prior art that the results of the binocular matching method are poor in accuracy due to the strong dependence on true value acquisition.

[0005] In order to achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A binocular camera stereo matching method based on a loss function, the method comprising:

[0007] Get the left eye image and the right eye image of the binocular camera;

[0008] Extracting pixel points of the left-eye image and pixel points of the right-eye image, and calculating the single-point similarity of each pixel point, and using the single-point similarity as the first loss function;

[0009] Extracting the overall similarity between the left-eye image and the right-eye image, and taking the maximum value of the overall similarity as the second loss function;

[0010] Performing a weighted summation on the first loss function and the second loss function to obtain a joint loss function;

[0011] Stereo matching is performed on the binocular cameras based on the joint loss function.

[0012] Furthermore, a residual network is used as a first layer network, and the first loss function is calculated through the first layer network.

[0013] Further, extracting the pixel points of the left-eye image and the pixel points of the right-eye image, and calculating the single-point similarity of each pixel point, and using the single-point similarity as the first loss function, specifically includes:

[0014] Extracting a feature vector of each pixel of the left-eye image and the right-eye image;

[0015] For each possible disparity of a pixel in the left image, find the position of the corresponding pixel in the right image, and calculate the cosine similarity of the feature vectors of the group of corresponding pixels;

[0016] The single-point similarity is calculated based on the cosine similarity of the feature vector, and the single-point similarity is used as the first loss function.

[0017] Further, the calculating of single point similarity based on the cosine similarity of the feature vector specifically includes:

[0018] The single point similarity is calculated using the following formula:

[0019]

[0020] in:

[0021]

[0022] Among them, X, Y are the pixel values ​​in the left and right images, R, C are the neighborhood sizes, then u x ,u y is the average gray value of the left and right images, is the gray value variance of the left and right images, σ xy is the covariance of the left and right figures, C1 and C2 are empirical parameters.

[0023] Furthermore, the single point similarity is used as the first loss function, specifically including:

[0024] The single point similarity is used as the first loss function.

[0025] Furthermore, a residual network with deformable convolution is used as a second layer network, and the second loss function is calculated through the second layer network.

[0026] Further, performing weighted summation on the first loss function and the second loss function to obtain a joint loss function specifically includes:

[0027] The joint loss function is calculated using the following formula:

[0028] L3=α·L1+β·L2

[0029] Among them, α and β are the weights of the joint loss function, and in the initial stage of model training, α>β, and in the tuning stage of model training, α<β.

[0030] The present invention also provides a binocular camera stereo matching system based on a loss function, the system comprising:

[0031] An image acquisition unit, used to acquire a left-eye image and a right-eye image of a binocular camera;

[0032] A first loss function acquisition unit, used for extracting pixel points of the left-eye image and pixel points of the right-eye image, and calculating the single-point similarity of each pixel point, and using the single-point similarity as the first loss function;

[0033] A second loss function acquisition unit, used for extracting the overall similarity between the left-eye image and the right-eye image, and taking the maximum value of the overall similarity as a second loss function;

[0034] A joint loss function acquisition unit, configured to perform a weighted summation of the first loss function and the second loss function to obtain a joint loss function;

[0035] The image stereo matching unit is used to perform stereo matching on the binocular cameras based on the joint loss function.

[0036] The present invention also provides an intelligent terminal, which includes: a data acquisition device, a processor and a memory;

[0037] The data acquisition device is used to acquire data; the memory is used to store one or more program instructions; and the processor is used to execute one or more program instructions to execute the method as described above.

[0038] The present invention also provides a computer-readable storage medium, wherein the computer storage medium contains one or more program instructions, and the one or more program instructions are used to execute the method as described above.

[0039] The binocular camera stereo matching method, system and intelligent terminal based on loss function provided by the present invention extract the pixel points of the left eye image and the pixel points of the right eye image, and calculate the single point similarity of each pixel point, use the single point similarity as the first loss function, extract the overall similarity of the left eye image and the right eye image, and use the maximum value of the overall similarity as the second loss function, perform weighted summation on the first loss function and the second loss function to obtain a joint loss function, and perform stereo matching on the binocular camera based on the joint loss function. The method realizes binocular stereo matching by cascading a two-layer deep learning network, does not rely on the true value of the distance of the data set, reduces the need for manual annotation, and improves the accuracy of the method. Moreover, this loss function focuses on the overall matching degree of the data, does not rely on the detection accuracy of the true value of the distance, and has strong generalization ability. The technical problem of poor result accuracy and generalization ability of the binocular matching method in the prior art due to strong dependence on true value acquisition is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0041] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0042] Figure 1 A flowchart of a specific implementation of the binocular camera stereo matching method based on loss function provided by the present invention;

[0043] Figure 2 for Figure 1 The structural block diagram of the first layer network residual block in the method shown;

[0044] Figure 3 for Figure 1 The structural block diagram of the residual block of the second layer network in the method shown;

[0045] Figure 4 A structural block diagram of a specific implementation of the binocular camera stereo matching method based on loss function provided by the present invention;

[0046] Figure 5 The original image in the initial stage of model training;

[0047] Figure 6 This is the corresponding effect diagram of the initial stage of model training;

[0048] Figure 7 This is the corresponding effect diagram of the model training and tuning stage. DETAILED DESCRIPTION

[0049] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The binocular camera stereo matching method provided by the invention is based on a loss function, which uses a joint loss function as a parameter, so that the self-supervised deep learning solution can realize the stereo matching method, which meets the needs of the automotive market, and proves that the binocular vision solution is suitable for the self-supervised deep learning solution to realize the stereo matching method. More importantly, this method does not rely on the true distance value of the data set, reduces the need for manual annotation, simplifies the task difficulty, avoids the problem of poor method accuracy caused by dependence on the true distance value, and this loss function focuses on the overall matching degree of the data, has a strong generalization ability, basically does not need to modify the reasoning logic of deep learning, and has a strong landing ability based on deep learning chips.

[0051] Please refer to Figure 1 , Figure 1 The present invention provides a flowchart of a specific implementation of the binocular camera stereo matching method based on the loss function.

[0052] In a specific embodiment, the present invention provides a binocular stereo matching method using a two-layer deep learning network cascade, wherein the first layer extracts the similarity of a single pixel on the left and right camera images, and the second layer extracts the probability of overall similarity of pixels on the image.

[0053] like Figure 1 As shown, the binocular camera stereo matching method based on loss function provided by the present invention includes the following steps:

[0054] S1: Obtain the left eye image and the right eye image of the binocular camera.

[0055] S2: extracting pixel points of the left-eye image and pixel points of the right-eye image, and calculating the single-point similarity of each pixel point, and using the single-point similarity as the first loss function.

[0056] Specifically, in this embodiment, a residual network is used as a first layer network, and the first loss function is calculated through the first layer network.

[0057] When the residual network is used to calculate the first loss function, the pixel points of the left eye image and the pixel points of the right eye image are extracted, and the single point similarity of each pixel point is calculated, and the single point similarity is used as the first loss function, which specifically includes the following steps:

[0058] S21: Extracting a feature vector of each pixel of the left-eye image and the right-eye image.

[0059] In a specific usage scenario, the residual network (ResNet) can be selected as the first layer network. The block diagram of the residual block is as follows Figure 2 As shown, 50 residual blocks are cascaded to extract the feature vector of each pixel in the left and right images.

[0060] S22: for each possible disparity of a pixel in the left-eye image, find the position of a corresponding pixel in the right-eye image, and calculate the cosine similarity of the feature vectors of the group of corresponding pixels.

[0061] Still taking the above usage scenario as an example, for each possible disparity (d) of each pixel in the left image, find the position of the corresponding point in the right image, and calculate the cosine similarity (CS) of the feature vector using formula (1):

[0062]

[0063] Among them, x, y are the eigenvectors of the left and right images, and i is the vector subscript.

[0064] S23: Calculate the single point similarity based on the cosine similarity of the feature vector, and use the single point similarity as the first loss function. Specifically, in order to improve the calculation accuracy, the maximum value of the single point similarity is used as the first loss function, that is, the structural similarity of a single pixel (SSIM) is used as the similarity loss function (L1), as shown in formula (2):

[0065] L1=max∑SSIM (2)

[0066] In step S22, the single point similarity is calculated using the following formula:

[0067]

[0068] Among them, X, Y are the pixel values ​​in the left and right images, R, C are the neighborhood sizes, then u x ,u y is the average gray value of the left and right images, is the gray value variance of the left and right images, σ xy is the covariance of the left and right figures, C1 and C2 are empirical parameters, generally C1 is taken as 0.02 and C2 is taken as 0.04.

[0069] S3: extracting the overall similarity between the left-eye image and the right-eye image, and taking the maximum value of the overall similarity as the second loss function.

[0070] In this specific implementation, a residual network with deformable convolution is used as the second layer network, and the second loss function is calculated through the second layer network.

[0071] In the above specific usage scenario, the second layer network uses a residual network with deformable convolution to extract the overall similarity between the left and right images. The block diagram of the residual block is as follows: Figure 4 As shown in Figure 1, adding deformable convolution makes the similarity have a certain propagation effect in the neighborhood. The loss function (L2) of the second network is to maximize the similarity of the overall pixels of the image, as shown in formula (3):

[0072] L2=max∑CS (3)

[0073] S4: Perform a weighted summation on the first loss function and the second loss function to obtain a joint loss function.

[0074] Specifically, the joint loss function is calculated using the following formula:

[0075] L3=α·L1+β·L2

[0076] Among them, α and β are the weights of the joint loss function. By setting different weight values, the optimal loss function value can be provided. It is recommended that α>β in the initial stage of model training and α<β in the tuning stage of model training, that is, according to the different characteristics of the training stage, such as Figure 5 As shown in the original image, in the initial stage of training, α is set to 0.8 and β is set to 0.2. The corresponding effect is as follows Figure 6 As shown; during the training tuning phase, α is set to 0.2 and β is set to 0.8. The effect is as follows Figure 7 shown.

[0077] S5: Perform stereo matching on the binocular cameras based on the joint loss function.

[0078] In the above specific implementation, the binocular camera stereo matching method based on loss function provided by the present invention extracts the pixel points of the left eye image and the pixel points of the right eye image, and calculates the single point similarity of each pixel point, takes the single point similarity as the first loss function, extracts the overall similarity of the left eye image and the right eye image, and takes the maximum value of the overall similarity as the second loss function, performs weighted summation on the first loss function and the second loss function to obtain a joint loss function, and performs stereo matching on the binocular camera based on the joint loss function. This method realizes binocular stereo matching by cascading a two-layer deep learning network, does not rely on the true value of the distance of the data set, reduces the need for manual annotation, improves the accuracy of the method, and this loss function focuses on the overall matching degree of the data, does not rely on the detection accuracy of the true value of the distance, and has a strong generalization ability. It solves the technical problem of poor result accuracy and generalization ability of the binocular matching method in the prior art due to the strong dependence on the true value acquisition.

[0079] In addition to the above method, the present invention also provides a binocular camera stereo matching system based on a loss function, such as Figure 4 As shown, the system comprises:

[0080] The image acquisition unit 100 is used to acquire the left eye image and the right eye image of the binocular camera;

[0081] A first loss function acquisition unit 200, used to extract pixel points of the left-eye image and pixel points of the right-eye image, and calculate the single-point similarity of each pixel point, and use the single-point similarity as the first loss function;

[0082] A second loss function acquisition unit 300 is used to extract the overall similarity between the left-eye image and the right-eye image, and use the maximum value of the overall similarity as a second loss function;

[0083] A joint loss function acquisition unit 400, configured to perform a weighted summation of the first loss function and the second loss function to obtain a joint loss function;

[0084] The image stereo matching unit 500 is used to perform stereo matching on the binocular cameras based on the joint loss function.

[0085] In a specific embodiment, the binocular camera stereo matching system based on loss function provided by the present invention extracts the pixel points of the left eye image and the pixel points of the right eye image, and calculates the single point similarity of each pixel point, takes the single point similarity as the first loss function, extracts the overall similarity of the left eye image and the right eye image, and takes the maximum value of the overall similarity as the second loss function, performs weighted summation on the first loss function and the second loss function to obtain a joint loss function, and performs stereo matching on the binocular camera based on the joint loss function. This method realizes binocular stereo matching by cascading a two-layer deep learning network, does not rely on the true value of the distance of the data set, reduces the need for manual annotation, improves the accuracy of the method, and this loss function focuses on the overall matching degree of the data, does not rely on the detection accuracy of the true value of the distance, and has a strong generalization ability. It solves the technical problem of poor result accuracy and generalization ability of the binocular matching method in the prior art due to strong dependence on true value acquisition.

[0086] The present invention also provides an intelligent terminal, which includes: a data acquisition device, a processor and a memory;

[0087] The data acquisition device is used to acquire data; the memory is used to store one or more program instructions; and the processor is used to execute one or more program instructions to execute the method as described above.

[0088] Corresponding to the above embodiment, an embodiment of the present invention further provides a computer storage medium, which includes one or more program instructions. The one or more program instructions are used to be executed by a binocular camera depth calibration system as described above.

[0089] In the embodiment of the present invention, the processor may be an integrated circuit chip having the signal processing capability. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0090] The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware.

[0091] The storage medium may be a memory, which may be, for example, a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory.

[0092] Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.

[0093] The volatile memory may be a random access memory (RAM) which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus DRAM (DRRAM).

[0094] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0095] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention can be implemented using a combination of hardware and software. When software is used, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. Storage media can be any available media that can be accessed by general or special-purpose computers.

[0096] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. A binocular camera stereo matching method based on loss function, characterized in that: The method comprises: Get the left eye image and the right eye image of the binocular camera; Extracting pixel points of the left-eye image and pixel points of the right-eye image, and calculating the single-point similarity of each pixel point, and using the single-point similarity as the first loss function; Extracting the overall similarity between the left-eye image and the right-eye image, and taking the maximum value of the overall similarity as the second loss function; Performing a weighted summation on the first loss function and the second loss function to obtain a joint loss function; Performing stereo matching on binocular cameras based on the joint loss function; Using the residual network as the first layer network, and calculating the first loss function through the first layer network; When the residual network is used to calculate the first loss function, the pixel points of the left eye image and the pixel points of the right eye image are extracted, and the single point similarity of each pixel point is calculated, and the single point similarity is used as the first loss function, which specifically includes the following steps: S21: extracting the feature vector of each pixel of the left image and the right image, selecting a residual network as the first layer network, and cascading 50 residual blocks to extract the feature vector of each pixel of the left image and the right image; S22: For each possible disparity of a pixel in the left image, find the position of the corresponding pixel in the right image, and calculate the cosine similarity of the feature vector of the corresponding pixel. For each possible disparity d of a pixel in the left image, find the position of the corresponding point in the right image, and calculate the cosine similarity CS of the feature vector by formula (1): Among them, x, y are the eigenvectors of the left and right images, and i is the vector subscript; S23: Calculate the single point similarity based on the cosine similarity of the feature vector, and use the single point similarity as the first loss function, and use the structural similarity SSIM of a single pixel as the similarity loss function L1, as shown in formula (2): L1=max∑SSIM(2) In step S22, the single point similarity is calculated using the following formula: Among them, X, Y are the pixel values ​​in the left and right images, R, C are the neighborhood sizes, then u x ,u y is the average gray value of the left and right images, is the gray value variance of the left and right images, σ xy is the covariance of the left and right figures, C1 and C2 are empirical parameters, C1 is taken as 0.02, and C2 is taken as 0.04; Using a residual network with deformable convolution as a second layer network, and calculating the second loss function through the second layer network; The weighted summing of the first loss function L1 and the second loss function L2 to obtain a joint loss function L3 specifically includes: The joint loss function is calculated using the following formula: L3=α·L1+β·L2 Among them, α and β are the weights of the joint loss function, and in the initial stage of model training, α>β, and in the tuning stage of model training, α<β.

2. A binocular camera stereo matching system based on a loss function, used to implement the method as claimed in claim 1, characterized in that: The system comprises: An image acquisition unit, used to acquire a left-eye image and a right-eye image of a binocular camera; A first loss function acquisition unit, used for extracting pixel points of the left-eye image and pixel points of the right-eye image, and calculating the single-point similarity of each pixel point, and using the single-point similarity as the first loss function; A second loss function acquisition unit, used for extracting the overall similarity between the left-eye image and the right-eye image, and taking the maximum value of the overall similarity as a second loss function; A joint loss function acquisition unit, configured to perform a weighted summation of the first loss function and the second loss function to obtain a joint loss function; The image stereo matching unit is used to perform stereo matching on the binocular cameras based on the joint loss function.

3. An intelligent terminal, characterized in that: The intelligent terminal includes: a data acquisition device, a processor and a memory; The data acquisition device is used to acquire data; the memory is used to store one or more program instructions; and the processor is used to execute one or more program instructions to execute the method according to claim 1.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium contains one or more program instructions, and the one or more program instructions are used to execute the method according to claim 1.

Images