Training method of disparity estimation model, image processing method and related equipment

Abstract

The present application provides a method for training a disparity estimation model, an image processing method and related equipment, and relates to the field of image technology. The training method includes: obtaining training data, the training data including a first training image, a second training image and camera parameters; inputting the training data into a disparity estimation model to obtain a disparity training image, the disparity training image is used to indicate the disparity between the first training image and the second training image, the disparity estimation model is used to determine the disparity of the image captured by the first camera and the image captured by the second camera in combination with the camera parameters; training the disparity estimation model based on the training data and the disparity training image to obtain a trained disparity estimation model. Based on the technical solution of the present application, when the disparity estimation model is used to perform disparity estimation on the images respectively captured by the dual cameras, the accuracy of the estimated disparity can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a disparity estimation model training method, an image processing method and related equipment. Background Art

[0002] With the widespread use of electronic devices, more and more cameras are integrated in electronic devices, and multiple cameras can meet the needs of users in various shooting scenes. Taking the electronic device as a mobile phone as an example, the electronic device can obtain a frame of shooting image through dual cameras.

[0003] In the prior art, when acquiring a captured image based on two frames of images collected by dual cameras, the electronic device can use a binocular depth network combined with the principle of binocular stereo vision to first determine the disparity corresponding to the same target in the two frames of images, and then perform processing based on the disparity to obtain the captured image; however, for images with repetitive textures or large motion amplitudes, the above method has the problem of a large error between the determined disparity and the actual disparity.

[0004] Therefore, a new method is urgently needed to effectively improve the accuracy of the obtained parallax, thereby improving the quality of the obtained captured images. Summary of the invention

[0005] The present application provides a disparity estimation model training method, an image processing method and related equipment, which can improve the accuracy of disparity and thus improve the quality of captured images obtained using disparity.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions:

[0007] In a first aspect, a network model training method is provided, the method comprising:

[0008] Acquire training data, wherein the training data includes a first training image, a second training image and camera parameters; for the same photographed object, the first training image is an image captured by a first camera, and the second training image is an image captured by a second camera; the camera parameters include camera intrinsic parameters and binocular extrinsic parameters, the camera intrinsic parameters include camera intrinsic parameters of the first camera and camera intrinsic parameters of the second camera, and the binocular extrinsic parameters indicate extrinsic parameters between the first camera and the second camera; input the training data into a disparity estimation model to obtain a disparity training image, the disparity training image is used to indicate the disparity between the first training image and the second training image, and the disparity estimation model is used to determine the disparity of the image captured by the first camera and the image captured by the second camera in combination with the camera parameters; train the disparity estimation model based on the training data and the disparity training image to obtain a trained disparity estimation model.

[0009] Optionally, the photographed objects may include but are not limited to: people, animals, plants, cars, buildings, etc.

[0010] Optionally, the first camera is an ultra-wide-angle camera, and the second camera is a wide-angle camera; or, the first camera is a wide-angle camera, and the second camera is a telephoto camera.

[0011] Optionally, the first camera and the second camera are both wide-angle cameras, or both are telephoto cameras.

[0012] Optionally, the first training image and the second training image may also be high-resolution color training images in the RGB domain.

[0013] In the training method provided in the embodiment of the present application, on the one hand, in the process of calculating the disparity between two frames of training images, the correlation between the hardware parameters of the two cameras corresponding to the two frames of training images is taken into account, and the error of the correlation calculation is reduced and the accuracy is improved by increasing the camera intrinsic parameters and the binocular extrinsic parameters; on the other hand, the disparity with epipolar constraint is added to the loss value in the training process, so that the loss value in the training process can minimize the residual and maximize the likelihood estimate, thereby making the trained disparity estimation model more accurate, and a more accurate disparity can be obtained in the subsequent use process.

[0014] In combination with the first aspect, in some implementations of the first aspect, the disparity estimation model includes a first convolutional network model, a second convolutional network model, a correlation search module, and a fusion model;

[0015] Inputting training data into a disparity estimation model to obtain disparity training images, including: inputting a first training image into a first convolutional network model to obtain a first group of multi-scale feature images, the first group of multi-scale feature images including a plurality of first training feature images with different resolutions; inputting a second training image into a second convolutional network model to obtain a second group of multi-scale feature images, the second group of multi-scale feature images including a plurality of second training feature images with different resolutions; inputting the first group of multi-scale feature images and the second group of multi-scale feature images into a correlation search module to obtain a third group of multi-scale feature images, the third group of multi-scale feature images including a plurality of third training feature images with different resolutions; inputting the third group of multi-scale feature images into a fusion model to obtain disparity training images; wherein the first training feature images and the second training feature images include semantic information, and the third training feature images include correlation values.

[0016] In this implementation, the first convolutional network model and the second convolutional network model are used for feature extraction, and the weights of the first convolutional network model and the second convolutional network model are shared.

[0017] In this implementation, the fusion model is a GRU model.

[0018] In combination with the first aspect, in some implementations of the first aspect, the correlation search module is used to perform a correlation search in a preset mapping area on a first training feature image and a second training feature image of the same resolution in the first group of multi-scale feature images and the second group of multi-scale feature images in combination with a camera parameter to obtain a third training feature image corresponding to the same resolution;

[0019] Among them, the correlation search is used to indicate that for each pixel in the first training feature image, the feature vector corresponding to the pixel is inner-producted with the feature vector corresponding to each pixel in the preset mapping area in the second training feature image, and the maximum value in the inner product result is determined as the corresponding correlation value, and the preset mapping area is determined in combination with the camera parameters.

[0020] In this implementation, when performing a correlation search, the present application takes into account the positional relationship between the first camera and the second camera, and maps the pixel points in the second training feature image to the first training feature image through camera parameters. When calculating the correlation value, only the correlation value within the preset mapping area is calculated, rather than the entire image. This improves processing efficiency and avoids redundant calculations, while also avoiding the problem of large correlation calculation errors caused by repeated textures in the entire image.

[0021] In combination with the first aspect, in certain implementations of the first aspect, the parameters of the disparity estimation model are obtained by iterating through a back-propagation algorithm based on the difference between the disparity training image and the disparity true value image, combined with the disparity with epipolar constraints; the disparity true value image is used to indicate the actual disparity between the first training image and the second training image.

[0022] In this implementation, a disparity with epipolar constraints is added to the loss value, so that during the training process, the loss value can minimize the residual and maximize the likelihood estimate, thereby making the determined disparity more precise, and a more accurate disparity can be obtained during subsequent use.

[0023] In combination with the first aspect, in some implementations of the first aspect, the first training image and the second training image have the same resolution.

[0024] In this implementation, the first training image and the second training image have the same resolution, which can facilitate the first convolutional network model and the second convolutional network model to perform the same convolution processing, thereby obtaining multiple pairs of first training feature images and second training feature images with the same resolution, making the processing process faster.

[0025] In combination with the first aspect, in certain implementations of the first aspect, the resolution of multiple first training feature images included in the first group of multi-scale feature images gradually decreases at a magnification, and the resolution of multiple second training feature images included in the second group of multi-scale feature images gradually decreases at a magnification.

[0026] In this implementation, the resolution of the plurality of third training feature images included in the third group of multi-scale feature images obtained based on the first group of multi-scale feature images and the second group of multi-scale feature images gradually decreases according to the magnification.

[0027] In this implementation, the original semantic information included in the low-resolution, dense first training feature image is more concentrated, and can be used to solve large-scale parallax. Similarly, the original semantic information included in the low-resolution, dense second training feature image is more concentrated, and can be used to solve large-scale parallax.

[0028] In combination with the first aspect, in certain implementations of the first aspect, the first group of multi-scale feature images includes a first training feature image having the same resolution as the first training image; and the second group of multi-scale feature images includes a second training feature image having the same resolution as the second training image.

[0029] In this implementation, the first group of multi-scale feature images obtained includes a first training feature image corresponding to the original resolution, so as to retain relatively fine details through the first training feature image of the original resolution without losing detail information; similarly, the second group of multi-scale feature images obtained includes a second training feature image corresponding to the original resolution, so as to retain relatively fine details through the second training feature image of the original resolution without losing detail information.

[0030] In a second aspect, an image processing method is provided, which is applied to an electronic device, and the method includes:

[0031] A first interface is displayed, wherein the first interface includes a first control; a first operation on the first control is detected; in response to the first operation, a first image, a second image and camera parameters are acquired; for the same photographed object, the first image is an image captured by a first camera, and the second image is an image captured by a second camera, and the camera parameters include camera intrinsic parameters and binocular extrinsic parameters, wherein the camera intrinsic parameters include camera intrinsic parameters of the first camera and camera intrinsic parameters of the second camera, and the binocular extrinsic parameters indicate extrinsic parameters between the first camera and the second camera; according to the disparity estimation model trained in the above first aspect or any possible implementation of the first aspect, the first image and the second image are processed in combination with the camera parameters to obtain a disparity image, wherein the disparity estimation model is used to determine the disparity of the image captured by the first camera and the image captured by the second camera in combination with the camera parameters; image processing is performed on the disparity image to obtain a captured image.

[0032] In the image processing method provided in the embodiment of the present application, since the correlation between the hardware parameters of the two cameras corresponding to the two frames of images is combined during the disparity calculation process for the two frames of images in the initial image pair through the disparity estimation model, and data such as camera intrinsic parameters and binocular extrinsic parameters are added, the error of the correlation calculation can be reduced, thereby improving the calculation accuracy; on the other hand, since the loss value of the disparity estimation model during the training process adds the disparity with epipolar constraints, the loss value can minimize the residual and maximize the likelihood estimate during the training process, thereby allowing the trained disparity estimation model to obtain more accurate disparity when processing the initial image pair during use.

[0033] In conjunction with the second aspect, in some implementations of the second aspect, the method further includes:

[0034] A second operation is detected, and the second operation is used to instruct to turn on a large aperture mode of the electronic device. The large aperture mode refers to a shooting mode that highlights the subject and blurs or defocuses the background.

[0035] In combination with the second aspect, in certain implementations of the second aspect, the first interface refers to a main screen interface of the electronic device, the main screen interface includes a camera application, and the first control refers to a control corresponding to the camera application.

[0036] In combination with the second aspect, in some implementations of the second aspect, the first interface refers to a photo taking interface, and the first control refers to a control for instructing photo taking.

[0037] In conjunction with the second aspect, in certain implementations of the second aspect, the first interface refers to a video recording interface, and the first control refers to a control for indicating video recording.

[0038] In a third aspect, a training device for a disparity estimation model is provided, the training device comprising one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store a computer program code, the computer program code comprises a computer instruction, and the one or more processors call the computer instruction to cause the training device to execute:

[0039] Acquire training data, wherein the training data includes a first training image, a second training image and camera parameters; for the same photographed object, the first training image is an image captured by a first camera, and the second training image is an image captured by a second camera; the camera parameters include camera intrinsic parameters and binocular extrinsic parameters, the camera intrinsic parameters include camera intrinsic parameters of the first camera and camera intrinsic parameters of the second camera, and the binocular extrinsic parameters indicate extrinsic parameters between the first camera and the second camera; input the training data into a disparity estimation model to obtain a disparity training image, the disparity training image is used to indicate the disparity between the first training image and the second training image, and the disparity estimation model is used to determine the disparity of the image captured by the first camera and the image captured by the second camera in combination with the camera parameters; train the disparity estimation model based on the training data and the disparity training image to obtain a trained disparity estimation model.

[0040] According to a fourth aspect, an image processing apparatus is provided, the image processing apparatus comprising one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store a computer program code, the computer program code comprises a computer instruction, and the one or more processors call the computer instruction to cause the image processing apparatus to execute:

[0041] A first interface is displayed, wherein the first interface includes a first control; a first operation on the first control is detected; in response to the first operation, a first image, a second image and camera parameters are acquired; for the same photographed object, the first image is an image captured by a first camera, and the second image is an image captured by a second camera, and the camera parameters include camera intrinsic parameters and binocular extrinsic parameters, wherein the camera intrinsic parameters include camera intrinsic parameters of the first camera and camera intrinsic parameters of the second camera, and the binocular extrinsic parameters indicate extrinsic parameters between the first camera and the second camera; according to the disparity estimation model trained in the above first aspect or any possible implementation of the first aspect, the first image and the second image are processed in combination with the camera parameters to obtain a disparity image, wherein the disparity estimation model is used to determine the disparity of the image captured by the first camera and the image captured by the second camera in combination with the camera parameters; image processing is performed on the disparity image to obtain a captured image.

[0042] In a fifth aspect, a training device for a disparity estimation model is provided, comprising a module / unit for executing the first aspect or any one of the methods in the first aspect.

[0043] In a sixth aspect, an image processing device is provided, comprising a module / unit for executing the second aspect or any one of the methods in the second aspect.

[0044] In the seventh aspect, a training device for a disparity estimation model is provided, the training device comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions so that the training device executes any one of the training methods in the first aspect.

[0045] In an eighth aspect, an image processing device is provided, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions to enable the image processing device to perform any one of the image processing methods in the second aspect.

[0046] In a ninth aspect, a chip system is provided, which is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes any one of the methods in the first aspect or the second aspect.

[0047] In a tenth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program code. When the computer program code is executed by an electronic device, the electronic device executes any one of the methods in the first aspect or the second aspect.

[0048] In an eleventh aspect, a computer program product is provided, the computer program product comprising: a computer program code, when the computer program code is executed by an electronic device, the electronic device executes any one of the methods in the first aspect or the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of the hardware architecture of an electronic device provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of the software architecture of an electronic device provided in an embodiment of the present application;

[0051] Figure 3 A schematic diagram of obtaining parallax for the prior art;

[0052] Figure 4 A schematic diagram of a process flow of a method for training a disparity estimation model provided in an embodiment of the present application;

[0053] Figure 5 A diagram showing an example process of training a disparity estimation model provided in an embodiment of the present application;

[0054] Figure 6 A schematic diagram of performing correlation search in combination with camera parameters provided in an embodiment of the present application;

[0055] Figure 7 A schematic diagram of a correlation pyramid provided in an embodiment of the present application;

[0056] Figure 8 A schematic diagram of a fusion model provided in an embodiment of the present application;

[0057] Fig. 9 A flowchart of an image processing method provided in an embodiment of the present application;

[0058] Fig.10 An exemplary process diagram of an image processing method provided in an embodiment of the present application;

[0059] Fig.11 A schematic diagram of a user interface in a large aperture mode provided in an embodiment of the present application;

[0060] Fig.12 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0061] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0063] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or. For example, A / B can mean A or B. The “and / or” in this article is merely a way to describe the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0064] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0065] First, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0066] 1. Binocular stereo vision: Binocular stereo vision is a method based on the principle of parallax and uses imaging equipment to obtain two frames of images of the object under test from different positions, and obtains the three-dimensional geometric information of the object by calculating the position deviation between corresponding points of the images.

[0067] In an embodiment of the present application, the imaging device may indicate two cameras arranged on the electronic device. Since the two cameras are arranged at a certain distance in physical space, they may be used as imaging devices at different positions to respectively acquire a frame of image.

[0068] 2. Neural network. A neural network refers to a network formed by connecting multiple single neural units together, that is, the output of one neural unit can be the input of another neural unit; the input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0069] 3. Deep neural network (DNN), deep neural network is also called multi-layer neural network, which can be understood as a neural network with multiple hidden layers. According to the position of different layers, DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers; the layers can be fully connected, that is, any neuron in the i-th layer can be connected to any neuron in the i+1-th layer.

[0070] 4. Convolutional neural network (CNN), a convolutional neural network is a deep neural network with a convolutional structure. A convolutional neural network contains a feature extractor consisting of a convolutional layer and a subsampling layer, which can be regarded as a filter; a convolutional layer refers to a neuron layer in a convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of a convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units.

[0071] 5. Gated recurrent unit (GRU), a neural network model for processing sequence data, can be used to solve long-term dependency problems. The core of GRU is to transfer the hidden state information of the previous moment to the next moment. GRU has two gating structures, namely the reset gate and the update gate. The reset gate is used to control which information in the hidden state of the previous moment needs to be retained, while the update gate is used to control the weight between the hidden state of the previous moment and the input of the current moment; GRU also has a memory unit called the candidate state, which has the function of storing and transmitting information. In the GRU network, the candidate state is calculated by the input of the current moment and the hidden state of the previous moment, and is weighted by the reset gate and the update gate; finally, the hidden state of the current moment is obtained by weighted merging of the candidate state and the hidden state of the previous moment.

[0072] 6. Back propagation algorithm. The neural network can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which is used to obtain the optimal parameters of the neural network model; for example, the weight matrix.

[0073] 7. Resolution. The resolution of an image indicates the amount of information stored in the image, such as how many pixels are there per inch. The unit of resolution is PPI (pixels per inch), usually called pixels per inch. Image resolution can be expressed as horizontal pixels × vertical pixels.

[0074] The above is a brief introduction to the terms involved in the embodiments of the present application, which will not be repeated below.

[0075] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0076] Figure 1 A structural schematic diagram of an electronic device provided by the present application is shown.

[0077] The electronic device 100 can be a mobile phone, a smart screen, a tablet computer, a wearable electronic device, an in-vehicle electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a projector, etc. The embodiment of the present application does not impose any restrictions on the specific type of the electronic device 100.

[0078] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0079] It should be noted that Figure 1 The structure shown does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include Figure 1 More or fewer components than those shown, or the electronic device 100 may include Figure 1 Combinations of some of the components shown, or the electronic device 100 may include Figure 1 Subassemblies of some of the components shown. Figure 1 The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0080] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0081] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0082] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0083] Exemplarily, the processor 110 can run the training method of the disparity estimation model provided by the embodiment of the present application, for example, obtaining training data, wherein the training data includes a first training image, a second training image and camera parameters; for the same shooting object, the first training image is an image captured by the first camera, and the second training image is an image captured by the second camera; the camera parameters include camera intrinsic parameters and binocular extrinsic parameters, the camera intrinsic parameters include camera intrinsic parameters of the first camera and camera intrinsic parameters of the second camera, and the binocular extrinsic parameters indicate the extrinsic parameters between the first camera and the second camera; the training data is input into the disparity estimation model to obtain a disparity training image, the disparity training image is used to indicate the disparity between the first training image and the second training image, and the disparity estimation model is used to determine the disparity of the image captured by the first camera and the image captured by the second camera in combination with the camera parameters; the disparity estimation model is trained based on the training data and the disparity training image to obtain a trained disparity estimation model.

[0084] Exemplarily, the processor 110 can run the image processing method provided by the embodiment of the present application, for example, displaying a first interface, the first interface including a first control; detecting a first operation on the first control; in response to the first operation, acquiring a first image, a second image and camera parameters; for the same photographed object, the first image is an image captured by the first camera, the second image is an image captured by the second camera, the camera parameters include camera intrinsic parameters and binocular extrinsic parameters, the camera intrinsic parameters include camera intrinsic parameters of the first camera and camera intrinsic parameters of the second camera, and the binocular extrinsic parameters indicate the extrinsic parameters between the first camera and the second camera; according to the disparity estimation model trained in the above training method, combined with the camera parameters, the first image and the second image are processed to obtain a disparity image, wherein the disparity estimation model is used to determine the disparity of the image captured by the first camera and the image captured by the second camera in combination with the camera parameters; image processing is performed on the disparity image to obtain a captured image. Here, the photographed object in the captured image is prominent and has a high definition, and the background is blurred or blurred.

[0085] The wireless communication function of the electronic device 100 can be implemented through components such as the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor.

[0086] The electronic device 100 can realize the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0087] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel.

[0088] The camera 193 is used to capture images. It can be triggered to start by application instructions to realize the photo function, such as taking pictures to obtain images of any scene. The camera may include components such as imaging lenses, filters, and image sensors. The light emitted or reflected by the object enters the imaging lens, passes through the filter, and finally converges on the image sensor. The image sensor is mainly used to converge the light emitted or reflected by all objects in the camera angle (also known as the scene to be photographed, the target scene, and can also be understood as the scene image that the user expects to shoot) to form an image; the filter is mainly used to filter out the redundant light waves in the light (for example, light waves other than visible light, such as infrared); the image sensor is mainly used to perform photoelectric conversion on the received light signal, convert it into an electrical signal, and input it into the processor 110 for subsequent processing. Among them, the camera 193 can be located in front of the electronic device 100, or on the back of the electronic device 100. The specific number and arrangement of the cameras can be set according to needs, and this application does not impose any restrictions.

[0089] Exemplarily, the electronic device 100 includes a front camera and a rear camera. For example, the rear camera includes a plurality of cameras, and the plurality of cameras may include a first camera and a second camera.

[0090] For example, the electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0091] Exemplarily, the touch sensor 180K is also referred to as a touch control device. The touch sensor 180K may be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, which is also referred to as a touch control screen. The touch sensor 180K is used to detect a touch operation acting on or near it. The touch sensor 180K may pass the detected touch operation to the application processor to determine the type of touch event. A visual output related to the touch operation may be provided through the display screen 194. In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100, and may be disposed at a different position from the display screen 194.

[0092] The hardware system of the electronic device 100 is described in detail above, and the software system of the electronic device 100 is introduced below.

[0093] like Figure 2 As shown, the system architecture may include an application layer 210 , an application framework layer 220 , a hardware abstraction layer 230 , a driver layer 240 and a hardware layer 250 .

[0094] The application layer 210 may include camera application, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0095] The application framework layer 220 provides an application programming interface (API) and a programming framework for the application programs of the application layer; the application framework layer may include some predefined functions.

[0096] For example, the application framework layer 220 may include a camera access interface, which may include camera management and camera equipment. The camera management may be used to provide an access interface for managing the camera, and the camera equipment may be used to provide an interface for accessing the camera.

[0097] The hardware abstraction layer 230 is used to abstract the hardware. For example, the hardware abstraction layer may include a camera abstraction layer and other hardware device abstraction layers; the camera hardware abstraction layer may call algorithms in a camera algorithm library.

[0098] For example, a camera algorithm library may include software algorithms for image processing.

[0099] The driver layer 240 is used to provide drivers for different hardware devices. For example, the driver layer may include a camera device driver, a digital signal processor driver, a graphics processor driver, or a central processing unit driver.

[0100] The hardware layer 250 may include camera devices and other hardware devices.

[0101] For example, the hardware layer 250 includes a camera device, a digital signal processor, a graphics processor, or a central processing unit; exemplarily, in an embodiment of the present application, the camera device may include a first camera and a second camera, and an image signal processor, and the image signal processor may be used for image processing.

[0102] Exemplarily, the disparity estimation model training method and image processing method provided in the embodiments of the present application may be executed in a digital signal processor, a graphics processor, or a central processing unit; or in other computing hardware of an electronic device.

[0103] At present, with the widespread use of electronic devices, more and more cameras are integrated into electronic devices, and multiple cameras are used to meet users' photography needs in various shooting scenes. Taking a mobile phone with dual cameras as an example, in the related technology, when obtaining a captured image based on two frames of images captured by the dual cameras, it is usually necessary to use a binocular depth network to determine the depth information corresponding to the same target in the two frames of images, and then combine the depth information to process and obtain the captured image.

[0104] Among them, when the binocular depth network calculates the disparity of two frames of images, it matches a small piece of one image with the entirety of the other image. In this way, when matching images with repetitive textures or large motion amplitudes, it is possible to match the wrong area, resulting in inaccurate disparity determined later, and thus inaccurate depth information. The depth map restored using this depth information will have large errors.

[0105] For example, Figure 3 FIG. 1 shows a schematic diagram of the processing process using a binocular depth network in the prior art. Figure 3 The image a shown in (a) is Figure 3 The image b shown in (b) in the figure has repeated stripes. When the binocular depth network calculates the disparity between the two frames, it is possible to match the wrong area. For example, it mistakenly matches the stripes in the upper right corner of image a with the stripes in the lower right corner of image b. This results in the determination of Figure 3 The information indicated in the lower right corner of the disparity map shown in (c) is inaccurate.

[0106] After analyzing the existing problems, it can be seen that this is because the input data of the binocular deep network is only two frames of images. In the process of disparity calculation, the correlation between the two frames of images is not taken into account, or in other words, the correlation between the hardware parameters of the two cameras corresponding to the two frames of images is not considered. When the two frames of images include a large number of repeated textures or the repeated textures are relatively concentrated, exceeding the tolerance range of the internal links of the binocular deep network, the disparity determined by the binocular deep network will have a large error with the actual disparity.

[0107] In view of this, the image processing method provided in the embodiment of the present application, on the one hand, takes into account the correlation between the two frames of images or the correlation between the hardware parameters of the two cameras corresponding to the two frames of images in the process of calculating the disparity between the two frames of images, and reduces the error of the correlation calculation and improves the accuracy by increasing the camera intrinsic parameters and the binocular extrinsic parameters; on the other hand, adds the disparity with epipolar constraint to the loss value in the training process, so that the loss value in the training process can minimize the residual and maximize the likelihood estimate, thereby making the trained disparity estimation model more accurate, and more accurate disparity can be obtained in the subsequent use process.

[0108] Exemplarily, the image processing method provided in the present application can be applied to face-related application scenarios, such as face unlocking scenarios, face payment scenarios, background blur, portrait lighting effects, animated expressions, three-dimensional beauty, liveness detection, vision correction, etc.

[0109] Exemplarily, the image processing method provided in the present application can also be applied to intelligent human-computer interaction scenarios, such as skeleton tracking, gesture recognition, spatial mapping, three-dimensional reconstruction, map reconstruction, autonomous navigation, augmented reality (AR) scenarios, etc.

[0110] It should be understood that the above is an example of an application scenario and does not limit the application scenario of the present application.

[0111] Through the above analysis, it can be known that the main reason for the large error between the parallax of the captured image and the actual parallax is that in the related technology, the binocular depth network used only processes two frames of images, without considering the correlation between the hardware parameters of the two cameras that obtain the two frames of images. In this regard, the present application has made improvements to this reason. In the process of determining the parallax, the correlation between the hardware parameters of the two cameras is adaptively increased. For example, the error is reduced by increasing the camera intrinsic parameters and the binocular extrinsic parameters to achieve the purpose of improving the accuracy of the parallax.

[0112] On this basis, this application also adds epipolar constraints on binocular internal and external parameters to further improve the accuracy of parallax.

[0113] The following describes in detail the training method and image processing method of the disparity estimation model provided by the present application in conjunction with the accompanying drawings.

[0114] First, the training method of the disparity estimation model provided in the embodiment of the present application is described in detail; the disparity estimation model obtained by the training method provided in the embodiment of the present application can be used to perform disparity estimation on images acquired by two cameras, thereby improving the accuracy of the disparity while satisfying the operating performance of the electronic device.

[0115] Figure 4 It is a flowchart of a method for training a disparity estimation model provided in an embodiment of the present application.

[0116] It should be understood that Figure 4 The training method of the disparity estimation model shown can be performed by a training device, which can be Figure 1 Electronic devices in the Figure 2 The digital signal processor, graphics processor or central processing unit shown is executed; Figure 4 The training method 300 shown includes S310 to S360, and S310 to S360 are described in detail below.

[0117] S310: Acquire training data, where the training data includes a training image pair and camera parameters, and the training image pair includes a first training image and a second training image.

[0118] The training image pairs correspond to the camera parameters one by one, and the camera parameters at least include the camera intrinsic parameters corresponding to the two cameras for obtaining the first training image and the second training image in the training image pair and the corresponding extrinsic parameters between the two cameras.

[0119] It should be understood that the first training image and the second training image correspond to different cameras respectively, and the first training image and the second training image include the same shooting object, which may indicate a person, an animal, a plant, a car, a building, etc., and the present application does not impose any limitation on this.

[0120] In addition, the first training image and the second training image have the same resolution.

[0121] Exemplarily, the first training image corresponds to the first camera, the second training image corresponds to the second camera, the first training image and the second training image include the same person; the resolutions of the first training image and the second training image are both 1920×1080, in pixels.

[0122] Optionally, the first camera is an ultra-wide-angle camera, and the second camera is a wide-angle camera; or, the first camera is a wide-angle camera, and the second camera is a telephoto camera.

[0123] Optionally, the first camera and the second camera are both wide-angle cameras, or both are telephoto cameras.

[0124] The above are only two examples. The camera types of the first camera and the second camera can be the same or different. The first camera and the second camera can also be other cameras, such as an ultra-telephoto camera, an ultra-wide-angle camera, etc. This application does not impose any other limitations on this.

[0125] Exemplarily, the first training image and the second training image may also be high-resolution (HR) high-definition color training images in the RGB domain, and each pixel in the image includes three primary color pixel values.

[0126] It should be understood that the training image pair can be an image captured by the electronic device through the first camera and the second camera, or an image obtained from the inside of the electronic device (for example, an image stored in the photo album of the electronic device, or an image obtained by the electronic device from the cloud), and different images come from different cameras; or the training image pair can also be an image obtained from other devices, which is collected by the first camera and the second camera in the other device. The source of the training image pair can be obtained as needed, and the embodiments of the present application do not impose any restrictions on this.

[0127] It should also be understood that when the training image pair is taken by a camera of an electronic device, the corresponding camera parameters can be obtained from the electronic device, and the camera parameters are generally stored in the electronic device before leaving the factory; when the training image pair is an image obtained from other devices, the camera parameters can also be provided by other devices, and the embodiments of the present application do not impose any restrictions on this.

[0128] It should be noted that the camera intrinsics are used to indicate parameters related to the camera's own properties, such as the focal length and pixel size corresponding to the camera. For example, the first camera corresponds to the first camera intrinsics, and the first camera intrinsics include the focal length and pixel size corresponding to the first camera; the second camera corresponds to the second camera intrinsics, and the second camera intrinsics include the focal length and pixel size corresponding to the second camera. In the embodiment of the present application, the first camera is different from the second camera, and the first camera intrinsics are also different from the second camera intrinsics.

[0129] Binocular extrinsic parameters are used to indicate the rotation and translation transformations that one camera needs to make when the imaging planes of the two cameras coincide. For example, assuming that the imaging plane of the second camera coincides with the imaging plane of the first camera, the rotation and translation transformations that the first camera needs to make are the extrinsic parameters between the first camera and the second camera.

[0130] S320: Perform convolution processing on the training image pair to obtain two groups of multi-scale feature images.

[0131] Each group of multi-scale feature images includes multiple feature images with different resolutions, but different groups may include feature images with the same resolution. The resolution of any feature image in the multi-scale feature images is less than or equal to the resolution of the original corresponding training image.

[0132] Optionally, convolution processing can be implemented using a convolutional neural network.

[0133] Specifically, the above S320 includes: performing feature extraction on the first training image in the training image pair using a convolutional neural network to obtain a first group of multi-scale feature images, where the first group of multi-scale feature images includes a plurality of first training feature images with different resolutions.

[0134] Similarly, a convolutional neural network is also used to perform feature extraction on the second training image in the training image pair to obtain a second group of multi-scale feature images, which includes a plurality of second training feature images with different resolutions.

[0135] It should be noted that the feature extraction performed on the first training image and the second training image in the training image pair in the embodiment of the present application refers to the extraction of pixel-level features; in addition, the convolutional neural network for feature extraction on the first training image and the convolutional neural network for feature extraction on the second training image can share weights, or in other words, the two convolutional neural networks are the same, so that multiple groups of first training feature images and second training feature images with the same resolution can be obtained, and then, the first training feature images with different resolutions are used to form a first group of multi-scale feature images, and the second training feature images with different resolutions are used to form a second group of multi-scale feature images.

[0136] For example, Figure 5As shown, by performing feature extraction on the first training image X1, a first training feature image X11 corresponding to the original resolution, a first training feature image X12 of 1 / 2 resolution, a first training feature image X13 of 1 / 4 resolution, etc. can be obtained. The first training feature image X11, the first training feature image X12, and the first training feature image X13 can be collectively referred to as a first group of multi-scale feature images. It can be understood that the 1 / 2 resolution is used to indicate that the resolution of the first training feature image X12 is 1 / 2 of the resolution of the first training feature image X11, and the 1 / 4 resolution is used to indicate that the resolution of the first training feature image X13 is 1 / 4 of the resolution of the first training feature image X11.

[0137] like Figure 5 As shown, feature extraction is performed on the second training image X2, and a second training feature image X21 corresponding to the original resolution, a second training feature image X22 of 1 / 2 resolution, a second training feature image X23 of 1 / 4 resolution, etc. can be obtained. The second training feature image X21, the second training feature image X22, and the second training feature image X23 can be collectively referred to as a second group of multi-scale feature images. It can be understood that the 1 / 2 resolution is used to indicate that the resolution of the second training feature image X22 is 1 / 2 of the resolution of the second training feature image X21, and the 1 / 4 resolution is used to indicate that the resolution of the second training feature image X23 is 1 / 4 of the resolution of the second training feature image X21. In addition, since the resolution of the first training image X1 is the same as that of the second training image X2, the resolution of the first training feature image X11 is the same as that of the second training feature image X21, the resolution of the first training feature image X12 is the same as that of the second training feature image X22, and the resolution of the first training feature image X13 is the same as that of the second training feature image X23.

[0138] It should be understood that the above only uses three feature images with different resolutions as an example, and a set of multi-scale feature images may include two, three or more feature images with different resolutions, which is not limited in the present application.

[0139] In the embodiment of the present application, the first group of multi-scale feature images obtained includes a first training feature image X11 corresponding to the original resolution, in order to retain relatively fine details through the first training feature image X11 with the original resolution without losing detail information; in addition, it includes a low-resolution, dense first training feature image X12, a first training feature image X13, etc., because the original semantic information included in the low-resolution, dense first training feature image X12 and the first training feature image X13 is more concentrated, which can be used to solve large-scale parallax.

[0140] Similarly, the second set of multi-scale feature images obtained includes the second training feature image X21 corresponding to the original resolution, in order to retain relatively fine details through the second training feature image X21 of the original resolution without losing detail information; in addition, the low-resolution, dense second training feature image X22, the second training feature image X23, etc. are included because the original semantic information included in the low-resolution, dense second training feature image X22 and the second training feature image X23 is more concentrated, which can be used to solve large-scale parallax. Here, the lower the resolution, the more concentrated the original semantic information included in the image, and the two are associated.

[0141] S330 , for the first training feature image and the second training feature image of the same resolution in the two groups of multi-scale feature images, perform a correlation search in a preset mapping area in combination with camera parameters to obtain a corresponding third training feature image.

[0142] The features in the third training feature image are used to represent the correlation value (correlation volume), or correlation tensor, visual similarity, between the pixels in the first training feature image and the second training feature image.

[0143] It should be noted that performing correlation search on the first training feature image and the second training feature image of the same resolution in combination with camera parameters means performing inner product of the feature vector of the first training feature image of the same resolution with the feature vector within the preset mapping area of ​​the second training feature image to generate a corresponding third training feature image.

[0144] For example, in combination Figure 5 The example shown is Figure 6 As shown in (a), the first training feature image X11 and the second training feature image X21 are taken as examples. Assume that the pixel position of the same point P in X11 is p1, and the pixel position in X21 is p2, and the camera intrinsic parameter corresponding to the first camera is K1, the camera intrinsic parameter corresponding to the second camera is K2, and the external parameter from the second camera to the first camera is R+t, where R is used to indicate the rotation relationship and t is used to indicate the translation relationship.

[0145] From the constraints of camera intrinsic parameters and binocular extrinsic parameters, we know that: p1 = K1P; p2 = K2(RP+t); Based on the above two formulas, we can get the relationship between p1 and p2, such as p2 = K2(RK1 -1 p1+t). Thus, according to the derived relationship between p1 and p2, a correlation search can be performed in a preset mapping area near the coordinate values ​​after the camera parameter mapping.

[0146] For example, when p1 is a pixel point in the first training feature image, based on the relationship between p1 and p2, the pixel coordinate value corresponding to p2 in the second training feature image can be obtained; then, p2 can be taken as the pixel center position, and the surrounding 5×5 area can be determined as the preset mapping area; further, the inner product of the feature vector corresponding to p1 and the feature vector corresponding to each pixel in the 5×5 preset mapping area can be calculated respectively, and the maximum value of the inner product can be determined as the corresponding correlation value.

[0147] It should be understood that the above 5×5 is only an example, and the size of the preset mapping area can be set and adjusted based on needs, and this application does not impose any restrictions on this. The correlation search process for other pixels in the first training feature image X11 is similar to the above, and will not be repeated here.

[0148] It should be noted that when performing a correlation search between X11 and X21, the present application takes into account the positional relationship between the first camera and the second camera, and maps the pixel points in X21 to X11 through the camera parameters. When calculating the correlation value, only the correlation value within the preset mapping area is calculated, rather than the entire image. This improves processing efficiency and avoids redundant calculations. It also avoids the problem of large correlation calculation errors caused by repeated textures in the entire image.

[0149] On this basis, since there are multiple first training feature images with different resolutions and multiple second training feature images with different resolutions, combined with the camera parameters, after performing a correlation search on each pair of first training feature images and second training feature images belonging to the same resolution, a third training feature image can be obtained; after performing a correlation search on multiple pairs of first training feature images and second training feature images, multiple third training feature images can be obtained.

[0150] For example, in combination Figure 5 The example shown is Figure 6 As shown in (b), the first set of multi-scale feature images includes three first training feature images X11, X12 and X13 with different resolutions, and the second set of multi-scale feature images includes three second training feature images X21, X22 and X23 with different resolutions. Among them, the resolution of X11 and X21 is the same, which is the original resolution; the resolution of X12 and X22 is the same, which is 1 / 2 of the original resolution; the resolution of X13 and X23 is the same, which is 1 / 4 of the original resolution.

[0151] Then, if Figure 6As shown in (b), for the first training feature image X11 and the second training feature image X21 belonging to the same resolution, after performing a correlation search in combination with the camera parameters, a third training feature image X31 corresponding to the same resolution can be generated; each pixel in the third training feature image X31 includes a correlation value indicating the feature vectors in the preset mapping area in the first training feature image X11 and the second training feature image X21.

[0152] Similarly, the process of obtaining the third training feature image X32 and the third training feature image X33 is similar to the above, and the details may refer to the above description, which will not be repeated here.

[0153] It should be understood that in the process of correlation search, the embodiment of the present application can accurately calculate the correlation between a pixel and multiple pixels within the corresponding preset mapping area by adding camera intrinsic parameters and binocular extrinsic parameters, rather than using the entire image to calculate the correlation, which relatively saves the calculation range, improves processing efficiency, avoids redundant calculations, and thus avoids the problem of large errors caused by full-image correlation calculations. That is, by adding camera intrinsic parameters and binocular extrinsic parameters, the present application can reduce the error of correlation calculation and be more accurate.

[0154] S340: Fusing multiple third training feature images to obtain a disparity training image.

[0155] Optionally, a plurality of third training feature images may be overlapped to form a correlation pyramid.

[0156] For example, Figure 7 As shown, it is a correlation pyramid provided by an embodiment of the present application. The correlation pyramid is composed of Figure 6 The third training feature image X33, the third training feature image X32 and the third training feature image X31 in the acquired third group of multi-scale feature images are composed of three layers stacked from top to bottom.

[0157] The plurality of third training feature images may also be collectively referred to as a third group of training feature images, and each third training feature image constituting the correlation pyramid may also be referred to as a feature correlation layer.

[0158] Optionally, a plurality of third training feature images or correlation pyramids are fused using a fusion model to obtain a disparity training image.

[0159] Exemplarily, the fusion model may be a GRU model, and the resolution of the disparity training image is the same as the maximum resolution in the third training feature image, such as the same as the resolution of the third training feature image X31. The pixels in the disparity training image correspond to the disparity information at all positions.

[0160] For example, Figure 8 As shown in Figure 1, it is a schematic diagram of the structure of the GRU model. Figure 5 In the example shown, when the third group of multi-scale feature images includes three third training feature images of different resolutions, the GRU model may include three sub-processing modules, four input terminals and one output terminal. The four input terminals are respectively used to input the disparity training images determined for the kth time, and the other three input terminals are used to respectively input the third training feature image X31, the third training feature image X32 and the third training feature image X33.

[0161] The three sub-processing modules can be respectively called the first sub-processing module, the second sub-processing module and the third sub-processing module from bottom to top; the first sub-processing module is used to combine the disparity training image determined for the kth time and the third training feature image X33 after downsampling for processing, and then upsample the first processing result and provide it to the second sub-processing module; the second sub-processing module is used to combine the disparity training image determined for the kth time, the downsampled third training feature image X31, the unsampled third training image X32, and the first processing result for processing, and then upsample the second processing result and provide it to the third sub-processing module; the third sub-processing module is used to combine the loss value determined for the kth time, the third training feature image X31 and the second processing result for processing to obtain the third processing result, and then, the third processing result is fused with the disparity training image determined for the kth time (such as addition operation) to obtain the disparity training image determined for the k+1th time.

[0162] Among them, the first sub-processing module, the second sub-processing module and the third sub-processing module can be used to perform weighted operation processing, and the weights can be adjusted and set based on needs, and this application does not impose any restrictions on this. For example, the weights of the first sub-processing module, the second sub-processing module and the third sub-processing module when performing weighted operation can be the same.

[0163] In addition, it should be understood that the disparity training image determined for the kth time is the disparity training image obtained in the previous training; when k is 0, that is, when GRU fusion is used for the first time, the disparity training image for the first time can be set as needed. The disparity training image determined for the k+1th time is the disparity training image determined in the current training process. When the training is repeated multiple times, k increases by 1 for each training cycle.

[0164] S350: Determine a loss value based on the disparity training image and the disparity true value image.

[0165] S360 , adjusting parameters based on the loss value, and looping through steps S310 to S350 until the loss value converges to obtain a disparity estimation model.

[0166] The disparity true value image is used to indicate the real disparity determined based on the first training image and the second training image. The disparity estimation model includes a convolutional neural network, a correlation search module for realizing correlation search in combination with camera parameters, and a GRU model. The disparity estimation model is used to indicate the trained convolutional neural network, correlation search module, and GRU model when the loss value converges, and the adjustment parameters refer to the adaptive adjustment of the parameters in the convolutional neural network, the correlation search module, and the GRU model.

[0167] Among them, the loss value can be determined based on the following formula:

[0168]

[0169] Where L is the loss value, γ is a hyperparameter (or preset value, which can be adjusted based on needs); d gt represents the true disparity, d i,j It represents the disparity at the pixel position of the i-th row and the j-th column calculated based on the above S310 to S350, and the image has a total of n rows and m columns.

[0170] Optionally, in S360, the embodiment of the present application may add a disparity with epipolar constraints for the loss value.

[0171] It should be understood that the camera extrinsic parameters are the essential matrix E = t^R, and the corresponding basic matrix is ​​F = K2 -T t^RK1 -1 The fundamental matrix is ​​used to describe the transformation relationship between the image captured by the first camera and the image captured by the second camera. The essential matrix is ​​based on the fundamental matrix, excluding the influence of the camera's intrinsic parameters and only retaining the geometric relationship between the two frames of images. The essential matrix is ​​obtained by the fundamental matrix and the camera's intrinsic parameters.

[0172] Recall the definition of ^: do the outer product with t at the same time.

[0173] Combination Figure 6 As shown in (a), for the same point P at pixel position p1 in X11 and pixel position p2 in X21, there is an epipolar constraint p2 T Fp1 = 0; for parallax d, p2 + d = p1, that is, p2 = p1 - d, and thus 0 = p2 T Fp1=(p1-d) T Fp1.

[0174] And find the parallax d (i.e. Figure 8 d in k+1 ), d is not necessarily accurate, so the disparity with epipolar constraints can be added to the loss value to minimize the residual and maximize the likelihood estimate.

[0175] like:

[0176] Among them, minL is used to indicate the minimum value of the loss value, and the second term in the above formula is used to indicate the added epipolar constraint term.

[0177] It should be understood that the embodiment of the present application adds a disparity with epipolar constraints to the loss value, so that during the training process, the loss value can minimize the residual and maximize the likelihood estimate, thereby making the determined disparity more precise, and a more accurate disparity can be obtained during subsequent use.

[0178] In the training method provided in the embodiment of the present application, on the one hand, in the process of calculating the disparity between two frames of training images, the correlation between the hardware parameters of the two cameras corresponding to the two frames of training images is taken into account, and the error of the correlation calculation is reduced and the accuracy is improved by increasing the camera intrinsic parameters and the binocular extrinsic parameters; on the other hand, the disparity with epipolar constraint is added to the loss value in the training process, so that the loss value in the training process can minimize the residual and maximize the likelihood estimate, thereby making the trained disparity estimation model more accurate, and a more accurate disparity can be obtained in the subsequent use process.

[0179] Next, we combine the above-mentioned trained disparity estimation model and the attached Fig. 9 and Fig.10 The image processing method provided in the embodiment of the present application is described in detail.

[0180] Fig. 9 FIG. 1 is a flow chart of an image processing method shown in an embodiment of the present application. The image processing method is applied to an electronic device including a first camera and a second camera. Fig. 9 As shown, the image processing method 400 includes: S410 to S450.

[0181] S410: Start a camera application and display a first interface, where the first interface includes a first control.

[0182] Exemplarily, the first interface may be a photographing interface in a photographing mode, or a recording interface in a recording mode, or may also be an interface in other modes.

[0183] When the first interface is a photographing interface, the first control may be a control for instructing photographing, such as a shooting key; when the first interface is a video recording interface, the first control may be a control for instructing video recording, such as a video recording key.

[0184] S420: Detect a first operation on a first control.

[0185] Exemplarily, the first operation may be a click operation on a shooting button or a video recording button, or may be a gesture operation, a voice operation, or the like.

[0186] S430. In response to the first operation, an initial image pair and camera parameters are obtained, where the initial image pair includes a first image and a second image, where the first image is acquired by a first camera for an object, and the second image is acquired by a second camera for the same object, and the camera parameters include camera intrinsic parameters and binocular extrinsic parameters.

[0187] Exemplarily, the color space of the first image and the second image is the same, and the first image and the second image may be Raw domain images, YUV domain images, RGB domain images, or the like.

[0188] It should be understood that the Raw domain image may refer to an image acquired in the Raw color space; that is, the Raw domain image may refer to an image in the Raw color space, and the rest is similar and will not be elaborated herein.

[0189] S440: Process the initial image pair according to the disparity estimation model and in combination with the camera parameters to obtain a disparity image. The disparity image is used to indicate the disparity information between two initial image pairs for the same photographed object.

[0190] For example, in combination Fig.10 As shown, the initial image pair is input Figure 5 After the disparity estimation model is trained, the first image in the initial image pair is first convolved to obtain a set of multi-scale feature images, such as three first feature images with different resolutions. Similarly, the second image in the initial image pair is convolved to obtain a set of multi-scale feature images, such as three second feature images with different resolutions. The first image and the second image have the same resolution and include the same photographed object. The resolutions of the three first feature images with different resolutions and the three second feature images with different resolutions are the same for each other.

[0191] Then, for the two sets of multi-scale feature images, the first feature image and the second feature image of the same resolution are combined with the camera parameters to perform a correlation search within the preset mapping area to obtain the corresponding third feature images. The features in the third feature image are used to represent the correlation value between the semantic information included in the pixels of the first feature image and the second feature image. The preset mapping area can be set based on needs. The specific processing process can refer to the aforementioned Figure 6 Next, the third feature images with different resolutions are input into the fusion model for processing to obtain the corresponding disparity images. The processing process can refer to the above Figure 8 The introduction of , I will not go into details here.

[0192] Optionally, before S440, the method may further include: preprocessing.

[0193] Among them, preprocessing may include cropping, rotation, flipping, scaling, etc., which is not limited in this application.

[0194] It should be understood that after preprocessing, the resolutions of the first image and the second image can be made the same, and then, the preprocessed image is input into the disparity estimation model to obtain a disparity image.

[0195] S450: Perform image processing on the parallax image to obtain a processed captured image.

[0196] Exemplarily, the image processing may include: image conversion, blur processing, blur processing, etc.

[0197] Image conversion may include but is not limited to: conversion from a parallax image to a depth image, or conversion from a parallax image to a color image; accordingly, the captured image is a depth image or a color image. The depth image is used to represent the distance information from the object in the captured scene to the camera.

[0198] The blurring and virtualizing processes may be implemented based on algorithms provided in related technologies, and this application does not limit this.

[0199] Optionally, Fig. 9 The image processing method shown may be an image processing method independent of the ISP; for example, Fig. 9 The image processing method shown can be used in Figure 2 The camera algorithms shown are implemented in the library.

[0200] Optionally, Fig. 9 The image processing method shown can be executed in an ISP; for example, the ISP may include a first module, the first module is used to execute the following Fig. 9 The image processing method shown; for example, a training method of a disparity estimation model can be executed in the NPU to obtain parameters of the disparity estimation model; the first module in the ISP can obtain the parameters of the disparity estimation model to obtain the disparity estimation model, and the initial image pair is processed by the disparity estimation model.

[0201] In the image processing method provided in the embodiment of the present application, since the correlation between the hardware parameters of the two cameras corresponding to the two frames of images is combined during the disparity calculation process for the two frames of images in the initial image pair through the disparity estimation model, and data such as camera intrinsic parameters and binocular extrinsic parameters are added, the error of the correlation calculation can be reduced, thereby improving the calculation accuracy; on the other hand, since the loss value of the disparity estimation model during the training process adds the disparity with epipolar constraints, the loss value can minimize the residual and maximize the likelihood estimate during the training process, thereby allowing the trained disparity estimation model to obtain more accurate disparity when processing the initial image pair during use.

[0202] Optionally, the image processing method provided by the present application can also be applied to a large aperture mode in a camera application.

[0203] The large aperture mode is a shooting mode that highlights the subject and blurs or defocuses the background.

[0204] Fig.11 FIG. 4 shows a schematic diagram of a user interface involved in the large aperture mode. Fig.11 As shown in (a) of FIG. 1 , a graphical user interface (GUI) of an electronic device is shown. When the electronic device detects that a user clicks on the icon of a camera application on the interface, the camera application can be started and a display such as Fig.11 Another GUI shown in (b) in the figure may be referred to as a first interface.

[0205] The first interface may include a viewfinder 10. In the preview state, the preview image may be displayed in real time in the viewfinder 10. The first interface may also include multiple shooting mode options and a first control, i.e., a shooting key 11. The multiple shooting mode options include, for example, a photo mode, a video mode, a large aperture mode, etc. The shooting key 11 is used to indicate that the current shooting mode is a photo mode, a video mode, or other modes. Among them, the camera application is generally in the photo mode by default when it is opened.

[0206] For example, Fig.11 As shown in (b), after the electronic device starts the camera application, the electronic device runs a program corresponding to the image processing method, and switches to the large aperture mode in response to the user's sliding operation on the shooting mode option.

[0207] like Fig.11In the large aperture mode, in response to the user's click operation on the shooting key 11, the first camera and the second camera included in the electronic device are used to obtain images respectively, and the two frames of images obtained are processed using the disparity estimation model to obtain corresponding disparity images, and then image processing is performed based on the disparity image to obtain a captured image with higher foreground clarity and better quality, which can be used for display or storage.

[0208] The image processing method provided in the embodiment of the present application can estimate accurate disparity by processing images collected by two cameras using a disparity estimation model, and then based on the disparity, can process a captured image with high foreground clarity and good quality in a large aperture mode.

[0209] Combination of the above Figures 1 to 11 , describes in detail the training method and image processing method of the disparity estimation model of the embodiment of the present application, and will now be combined with Figure 12 to Figure 13 , describing in detail the software system, hardware system, device and chip of the electronic device to which the present application is applicable. It should be understood that the software system, hardware system, device and chip in the embodiments of the present application can execute the training methods and image processing methods of various network models in the aforementioned embodiments of the present application, that is, the specific working processes of the following various products can refer to the corresponding processes in the aforementioned method embodiments.

[0210] Fig.12 It is a schematic diagram of an image processing device provided in an embodiment of the present application.

[0211] It should be understood that the image processing apparatus 500 may execute Fig. 9 The image processing method shown in FIG. 5 is as follows: the image processing apparatus 500 includes: a display unit 510 and a processing unit 520. It should also be understood that the disparity estimation model can be deployed in the image processing apparatus 500.

[0212] The display unit 510 is used to display a first interface, and the first interface includes a first control.

[0213] The processing unit 520 is used to detect a first operation on a first control; in response to the first operation, a first image, a second image and camera parameters are obtained; for the same photographed object, the first image is an image captured by a first camera, and the second image is an image captured by a second camera, and the camera parameters include camera intrinsic parameters and binocular extrinsic parameters, the camera intrinsic parameters include camera intrinsic parameters of the first camera and camera intrinsic parameters of the second camera, and the binocular extrinsic parameters indicate extrinsic parameters between the first camera and the second camera; according to the disparity estimation model trained in the above training method, combined with the camera parameters, the first image and the second image are processed to obtain a disparity image, wherein the disparity estimation model is used to determine the disparity of the image captured by the first camera and the image captured by the second camera in combination with the camera parameters; image processing is performed on the disparity image to obtain a captured image.

[0214] It should be noted that the above-mentioned image processing device 500 is embodied in the form of a functional unit. The term "unit" here can be implemented in the form of software and / or hardware, and is not specifically limited to this.

[0215] For example, a "unit" may be a software program, a hardware circuit, or a combination of the two that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (e.g., a shared processor, a dedicated processor, or a group processor, etc.) and a memory for executing one or more software or firmware programs, a combined logic circuit, and / or other suitable components that support the described functions.

[0216] Therefore, the units of each example described in the embodiments of the present application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0217] Fig.13 A structural schematic diagram of an electronic device provided by the present application is shown. Fig.13 The dotted line in the figure indicates that the unit or the module is optional; the electronic device 600 can be used to implement the method described in any of the above embodiments.

[0218] The electronic device 600 includes one or more processors 601, which can support the electronic device 600 to implement the time-lapse photography method in the method embodiment. The processor 601 can be a general-purpose processor or a special-purpose processor. For example, the processor 601 can be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, such as discrete gates, transistor logic devices, or discrete hardware components.

[0219] Optionally, the processor 601 may be used to control the electronic device 600, execute software programs, and process data of the software programs. The electronic device 600 may also include a communication unit 605 to implement input (reception) and output (transmission) of signals.

[0220] For example, the electronic device 600 may be a chip, the communication unit 605 may be an input and / or output circuit of the chip, or the communication unit 605 may be a communication interface of the chip, and the chip may be a component of a terminal device or other electronic devices.

[0221] For another example, the electronic device 600 may be a terminal device, the communication unit 605 may be a transceiver of the terminal device, or the communication unit 605 may be one or more memories 602 in the terminal device, on which a program 604 is stored. The program 604 may be executed by the processor 601 to generate instructions 603, so that the processor 601 executes the method described in any of the above embodiments according to the instructions 603.

[0222] Optionally, data may also be stored in the memory 602 .

[0223] Optionally, the processor 601 may also read data stored in the memory 602 . The data may be stored at the same storage address as the program 604 , or may be stored at a different storage address from the program 604 .

[0224] Optionally, the processor 601 and the memory 602 may be provided separately or integrated together, for example, integrated on a system on chip (SOC) of the terminal device.

[0225] Optionally, the present application further provides a computer program product, which, when executed by the processor 601, implements the method in any embodiment of the present application.

[0226] For example, the computer program product may be stored in the memory 602 , such as a program 604 , which is finally converted into an executable target file that can be executed by the processor 601 after preprocessing, compiling, assembling, and linking.

[0227] Optionally, the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a computer, the method in any embodiment of the present application is implemented. The computer program can be a high-level language program or an executable target program.

[0228] For example, the computer-readable storage medium is, for example, a memory 602. The memory 602 may be a volatile memory or a non-volatile memory, or the memory 602 may include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0229] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0230] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0231] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the embodiments of the electronic device described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0232] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0233] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0234] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0235] In addition, the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0236] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0237] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims. In short, the above are only preferred embodiments of the technical solution of the present application, and are not used to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for training a disparity estimation model, characterized in that: include: Acquire training data, wherein the training data includes a first training image, a second training image, and camera parameters; for the same photographed object, the first training image is an image captured by a first camera, and the second training image is an image captured by a second camera; the camera parameters include camera intrinsic parameters and binocular extrinsic parameters, the camera intrinsic parameters include camera intrinsic parameters of the first camera and camera intrinsic parameters of the second camera, and the binocular extrinsic parameters indicate extrinsic parameters between the first camera and the second camera; Inputting the training data into a disparity estimation model to obtain a disparity training image, wherein the disparity training image is used to indicate the disparity between the first training image and the second training image, and the disparity estimation model is used to determine the disparity of the image captured by the first camera and the image captured by the second camera in combination with the camera parameters; The disparity estimation model is trained based on the training data and the disparity training image to obtain a trained disparity estimation model; the parameters of the disparity estimation model are obtained by iterating a back-propagation algorithm based on a loss value between the disparity training image and a true disparity image, adding a disparity with an epipolar constraint to the loss value; the true disparity image is used to indicate a real disparity between the first training image and the second training image.

2. The training method according to claim 1, characterized in that: The disparity estimation model includes a first convolutional network model, a second convolutional network model, a correlation search module and a fusion model; Inputting the training data into a disparity estimation model to obtain a disparity training image includes: Inputting the first training image into the first convolutional network model to obtain a first set of multi-scale feature images, wherein the first set of multi-scale feature images includes a plurality of first training feature images with different resolutions; Inputting the second training image into the second convolutional network model to obtain a second set of multi-scale feature images, wherein the second set of multi-scale feature images includes a plurality of second training feature images with different resolutions; Inputting the first group of multi-scale feature images and the second group of multi-scale feature images into the correlation search module to obtain a third group of multi-scale feature images, wherein the third group of multi-scale feature images includes a plurality of third training feature images with different resolutions; Inputting the third group of multi-scale feature images into the fusion model to obtain the disparity training image; The first training feature image and the second training feature image include semantic information, and the third training feature image includes a correlation value.

3. The training method according to claim 2, characterized in that: The correlation search module is used to perform a correlation search in a preset mapping area on the first training feature image and the second training feature image of the same resolution in the first group of multi-scale feature images and the second group of multi-scale feature images in combination with the camera parameters to obtain a third training feature image corresponding to the same resolution; Among them, the correlation search is used to indicate that for each pixel in the first training feature image, the feature vector corresponding to the pixel is inner-producted with the feature vector corresponding to each pixel in the preset mapping area in the second training feature image, and the maximum value in the inner product result is determined as the corresponding correlation value, and the preset mapping area is determined in combination with the camera parameters.

4. The training method according to any one of claims 1 to 3, characterized in that: The first training image and the second training image have the same resolution.

5. The training method according to claim 2 or 3, characterized in that: The resolutions of the multiple first training feature images included in the first group of multi-scale feature images gradually decrease according to a magnification; the resolutions of the multiple second training feature images included in the second group of multi-scale feature images gradually decrease according to a magnification.

6. The training method according to claim 2 or 3, characterized in that: The first group of multi-scale feature images includes a first training feature image with the same resolution as the first training image; the second group of multi-scale feature images includes a second training feature image with the same resolution as the second training image.

7. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: Display a first interface, wherein the first interface includes a first control; detecting a first operation on the first control; In response to the first operation, a first image, a second image and camera parameters are acquired; for the same photographed object, the first image is an image captured by a first camera, the second image is an image captured by a second camera, the camera parameters include camera intrinsic parameters and binocular extrinsic parameters, the camera intrinsic parameters include camera intrinsic parameters of the first camera and camera intrinsic parameters of the second camera, and the binocular extrinsic parameters indicate extrinsic parameters between the first camera and the second camera; Processing the first image and the second image according to the disparity estimation model trained according to any one of claims 1 to 6 in combination with the camera parameters to obtain a disparity image, wherein the disparity estimation model is used to estimate the disparity of the image captured by the first camera and the image captured by the second camera in combination with the camera parameters; The parallax image is processed to obtain a captured image.

8. The method according to claim 7, characterized in that The method further comprises: A second operation is detected, where the second operation is used to instruct to turn on a large aperture mode of the electronic device, where the large aperture mode refers to a shooting mode that highlights the shooting object and blurs or defocuses the background.

9. The method according to claim 7 or 8, characterized in that: The first interface refers to a main screen interface of the electronic device, the main screen interface includes a camera application, and the first control refers to a control corresponding to the camera application.

10. The method according to claim 7 or 8, characterized in that: The first interface refers to a photo taking interface, and the first control refers to a control for instructing photo taking.

11. The method according to claim 7 or 8, characterized in that: The first interface refers to a video recording interface, and the first control refers to a control for instructing video recording.

12. A training device for a disparity estimation model, characterized in that: The training device comprises: one or more processors and memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the training device to perform the training method as described in any one of claims 1 to 6.

13. An image processing device, characterized in that: The image processing device comprises: one or more processors and memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the image processing method as described in any one of claims 7 to 11.

14. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 6, or 7 to 11.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 6, or 7 to 11.

Images