Image processing method, apparatus, device, and medium

Abstract

This application discloses an image processing method, apparatus, device, and medium, belonging to the field of artificial intelligence technology. The image processing method includes: acquiring a first image and a second image; concatenating a first vector of the first image and a second vector of the second image to obtain a third vector; inputting the third vector into a pre-trained optical flow model to obtain an offset vector of the first image relative to the second image, wherein the optical flow model is used to determine the offset vectors of the two images; and registering the first image and the second image according to the offset vector.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to an image processing method, apparatus, device, and medium. Background Technology

[0002] Image registration (also known as image alignment) is the process of matching two or more images acquired at different times, by different sensors (imaging devices), or under different conditions (weather, illumination, camera position and angle, etc.).

[0003] In related technologies, image registration mainly utilizes Fourier transform to calculate image correlation, thereby obtaining offset coordinates for image registration. However, determining the offset coordinates for image registration through Fourier transform involves a large amount of computation, resulting in low image registration efficiency. Summary of the Invention

[0004] The purpose of this application is to provide an image processing method, apparatus, device, and medium that can solve the problem of low image registration efficiency.

[0005] In a first aspect, embodiments of this application provide an image processing method, including:

[0006] Acquire the first image and the second image;

[0007] The first vector from the first image and the second vector from the second image are concatenated to obtain the third vector;

[0008] The third vector is input into the pre-trained optical flow model to obtain the offset vector of the first image relative to the second image, where the optical flow model is used to determine the offset vector between the two images;

[0009] The first and second images are registered based on the offset vector.

[0010] Secondly, embodiments of this application provide an image processing apparatus, including:

[0011] The acquisition module is used to acquire the first image and the second image;

[0012] The stitching module is used to stitch together the first vector of the first image and the second vector of the second image to obtain the third vector;

[0013] The first determining module is used to input the third vector into the pre-trained optical flow model to obtain the offset vector of the first image relative to the second image, wherein the optical flow model is used to determine the offset vector of the two images;

[0014] The registration module is used to register the first image and the second image based on the offset vector.

[0015] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory, wherein the memory stores a program or instructions that can run on the processor, and when the program or instructions are executed by the processor, they implement the steps of the method described in the first aspect.

[0016] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0017] Fifthly, embodiments of this application provide a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the steps of the method described in the first aspect.

[0018] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0019] In this embodiment, by concatenating the vectors of two images and then inputting the concatenated vector into a pre-trained optical flow model, the offset vector of one image relative to the other can be obtained. The two images can then be registered based on this offset vector. Compared to related technologies, this method eliminates the need for Fourier transform to obtain the offset vector for image registration, resulting in lower computational complexity and improved image registration efficiency. Attached Figure Description

[0020] Figure 1 This is a schematic flowchart of the image processing method provided in the embodiments of this application;

[0021] Figure 2 This is a schematic diagram of the optical flow model provided in the embodiments of this application;

[0022] Figure 3 This is a schematic diagram illustrating the process of obtaining the offset vector using an optical flow model, as provided in an embodiment of this application.

[0023] Figure 4 This is a schematic diagram illustrating the process of synthesizing images from a photographed scene according to an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application;

[0025] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0026] Figure 7 This is a schematic diagram of the hardware structure of an electronic device that implements the embodiments of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0028] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0029] The image processing methods, apparatus, devices, and media provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0030] Figure 1 This is a schematic flowchart of the image processing method provided in an embodiment of this application. Figure 1 As shown, image processing methods may include:

[0031] S101: Acquire the first image and the second image;

[0032] S102: Concatenate the first vector of the first image and the second vector of the second image to obtain the third vector;

[0033] S103: Input the third vector into the pre-trained optical flow model to obtain the offset vector of the first image relative to the second image, wherein the optical flow model is used to determine the offset vector between the two images;

[0034] S104: Register the first image and the second image according to the offset vector.

[0035] The specific implementation methods of the above steps will be described in detail below.

[0036] In this embodiment, by concatenating the vectors of two images and then inputting the concatenated vector into a pre-trained optical flow model, the offset vector of one image relative to the other can be obtained. The two images can then be registered based on this offset vector. Compared to related technologies, this method eliminates the need for Fourier transform to obtain the offset vector for image registration, resulting in lower computational complexity and improved image registration efficiency.

[0037] In some possible implementations of the embodiments of this application, optical flow in these embodiments refers to the instantaneous velocity of pixel movement of a spatially moving object on the observation imaging plane, specifically the amount of movement of a pixel of the same object in the previous frame to the next frame in two consecutive frames. In computer vision, optical flow refers to the movement of an object in an image, which is caused by camera movement, the movement of the actual object corresponding to the object in the image, or both.

[0038] In some possible implementations of the embodiments of this application, the activation function of the optical flow model in the embodiments of this application is a non-saturating activation function.

[0039] By using non-saturating activation functions, the problem of "vanishing gradients" can be solved, and the convergence speed of optical flow models can be accelerated.

[0040] In some possible implementations of the embodiments of this application, the non-saturated activation function includes, but is not limited to: rectified linear unit (ReLU), exponential linear unit (ELU), leaky ReLU, parametric ReLU (PReLU), randomized ReLU (RReLU), etc.

[0041] In some possible implementations of the embodiments of this application, preferably, the non-saturated activation function is Leaky ReLU and the negative slope parameter of Leaky ReLU is 0.1.

[0042] In some possible implementations of the embodiments of this application, the optical flow model in the embodiments of this application includes: a feature extraction layer and a feature recovery layer, wherein the feature extraction layer includes at least one first convolutional layer, used to extract the image feature vector corresponding to the third vector through different convolutional kernel sizes and strides; the feature recovery layer includes at least one second convolutional layer, at least one deconvolutional layer, at least one feature concatenation layer and a sampling layer; the second convolutional layer is used to convolve the image feature vector output by the previous layer; the deconvolutional layer is used to deconvolve the image feature vector output by the previous layer; the feature concatenation layer is used to concatenate the image feature vector output by the deconvolutional layer with the image feature vector of the same dimension extracted by at least one first convolutional layer; and the sampling layer is used to adjust the image feature vector output by the previous layer to have the same dimension as the third vector to obtain an offset vector.

[0043] In some possible implementations of the embodiments of this application, the optical flow model in the embodiments of this application may include 2N network layers, wherein the first N network layers are feature extraction layers, the last N network layers are feature recovery layers, the first to the Nth network layers are convolutional layers, the N+1 to 2N-1 network layers include convolutional layers, deconvolutional layers and feature concatenation layers, and the 2Nth network layer is a sampling layer.

[0044] For example, such as Figure 2 As shown. Figure 2 This is a schematic diagram of the optical flow model provided in an embodiment of this application. Figure 2 The optical flow model shown consists of 16 network layers. The first 8 layers are feature extraction layers, and the last 8 layers are feature recovery layers. Layers 1 to 8 are convolutional layers, layers 9 to 15 include convolutional layers, deconvolutional layers, and feature concatenation layers, and layer 16 is an upsampling layer used to output the offset vector between the two images.

[0045] The first layer of the network has a convolutional layer + Leaky ReLU structure, with an input dimension of (6, h, w), where 6 represents the 6 channels of the two images, with each image having 3 channels (red, green, and blue components of the RGB image). h represents the height of the input image, and w represents its width. h and w must be set to integer multiples of 32 based on the total number of layers and the convolutional sampling stride; for example, h = 640, w = 832. The network parameters for the first layer are: 6 input channels, 32 output channels, a kernel size of 7, and a stride of 2.

[0046] The network structure of the second layer is a convolutional layer + LeakyReLU. The input of the second layer is the output of the first layer. The network parameters of the second layer are: 32 input channels, 64 output channels, 5 convolutional kernels, and 2 strides.

[0047] The network structure of the third layer is a convolutional layer + LeakyReLU. The input of the third layer is the output of the second layer. The network parameters of the third layer are: 64 input channels, 128 output channels, 5 convolutional kernels, and 2 strides.

[0048] The network structure of the 4th layer is a convolutional layer + LeakyReLU. The input of the 4th layer is the output of the 3rd layer. The network parameters of the 4th layer are: 128 input channels, 128 output channels, 3 convolutional kernels, and 1 stride.

[0049] The network structure of the 5th layer is a convolutional layer + LeakyReLU. The input of the 5th layer is the output of the 4th layer. The network parameters of the 5th layer are: 128 input channels, 256 output channels, 3 convolutional kernels, and 2 strides.

[0050] The network structure of the 6th layer is a convolutional layer + LeakyReLU. The input of the 6th layer is the output of the 5th layer. The network parameters of the 6th layer are: 256 input channels, 256 output channels, 3 convolutional kernels, and 1 stride.

[0051] The network structure of the 7th layer is a convolutional layer + LeakyReLU. The input of the 7th layer is the output of the 6th layer. The network parameters of the 7th layer are: 256 input channels, 256 output channels, 3 convolutional kernels, and 2 strides.

[0052] The network structure of the 8th layer is a convolutional layer + LeakyReLU. The input of the 8th layer is the output of the 7th layer. The network parameters of the 8th layer are: 256 input channels, 256 output channels, 3 convolutional kernels, and 1 stride.

[0053] The 9th network layer includes the 9a, 9b, and 9c network layers.

[0054] The network structure of the 9a layer is a convolutional layer. The input of the 9a layer is the output of the 8th layer. The network parameters of the 9a layer are: 256 input channels, 2 output channels, 3 convolutional kernels, and 1 stride.

[0055] The network structure of the 9b layer is a deconvolution layer. The input of the 9b layer is the output of the 9a layer. The network parameters of the 9b layer are: 2 input channels, 2 output channels, 4 kernel size, and 2 stride.

[0056] The network structure of the 9c layer is a deconvolutional layer + LeakyReLU. The input of the 9c layer is the output of the 8th layer. The network parameters of the 9c layer are: 256 input channels, 128 output channels, 4 kernel size, and 2 stride.

[0057] The network structure of the 10th layer is a splicing layer. The input of the 10th layer is the output of the 6th layer, the output of the 9c layer, and the output of the 9b layer. The function of the 10th layer is to merge the outputs of the 6th layer, the 9c layer, and the 9b layer.

[0058] The 11th network layer includes the 11a, 11b, and 11c network layers.

[0059] The network structure of the 11a layer is a convolutional layer. The input of the 11a layer is the output of the 10th layer. The network parameters of the 11a layer are: 386 input channels, 2 output channels, 3 convolutional kernels, and 1 stride.

[0060] The network structure of the 11b layer is a deconvolution layer. The input of the 11b layer is the output of the 11a layer. The network parameters of the 11b layer are: 2 input channels, 2 output channels, 4 kernel size, and 2 stride.

[0061] The network structure of the 11c layer is a deconvolutional layer + LeakyReLU. The input of the 11c layer is the output of the 10th layer. The network parameters of the 11c layer are: 386 input channels, 64 output channels, 4 kernels, and 2 strides.

[0062] The network structure of the 12th layer is a splicing layer. The input of the 12th layer is the output of the 4th layer, the output of the 11c layer, and the output of the 11b layer. The function of the 12th layer is to merge the outputs of the 4th layer, the 11c layer, and the 11b layer.

[0063] The 13th network layer includes the 13a, 13b, and 13c network layers.

[0064] The network structure of the 13a layer is a convolutional layer. The input of the 13a layer is the output of the 12th layer. The network parameters of the 13a layer are: 194 input channels, 2 output channels, 3 convolutional kernels, and 1 stride.

[0065] The network structure of layer 13b is a deconvolution layer. The input of layer 13b is the output of layer 13a. The network parameters of layer 13b are: 2 input channels, 2 output channels, 4 kernel size, and 2 stride.

[0066] The network structure of the 13c layer is a deconvolutional layer + LeakyReLU. The input of the 13c layer is the output of the 12th layer. The network parameters of the 11c layer are: 194 input channels, 32 output channels, 4 kernels, and 2 strides.

[0067] The network structure of the 14th layer is a splicing layer. The input of the 14th layer is the output of the 2nd layer, the output of the 13c layer, and the output of the 13b layer. The function of the 14th layer is to merge the outputs of the 2nd layer, the 13c layer, and the 13b layer.

[0068] The network structure of the 15th layer is a convolutional layer. The input of the 15th layer is the output of the 14th layer. The network parameters of the 15th layer are: 98 input channels, 2 output channels, 3 convolutional kernels, and 1 stride.

[0069] The 16th layer is an upsampling layer. Its input is the output of the 15th layer. The network parameters for the 16th layer are: scale parameter of 4, mode of bilinearity. The output of the 16th layer is the output of the entire network, which is the output of the optical flow model, with dimensions (2, h, w).

[0070] For example, taking h=640 and w=832 as an example, the process of obtaining the offset vector using the optical flow model is explained as follows: Figure 3 As shown. Figure 3 This is a schematic diagram illustrating the process of obtaining the offset vector using an optical flow model, as provided in an embodiment of this application.

[0071] exist Figure 3 In the process, the two images with resolutions of (H, W) are first downsampled to obtain two images with resolutions of (640, 832).

[0072] The vectors of two images with a resolution of (640, 832) with a vector dimension of (3, 640, 832) are concatenated to obtain a concatenated vector with a vector dimension of (6, 640, 832). Here, the 3 in the vector dimension (3, 640, 832) represents three channels: the red channel, green channel, and blue channel of the image. The 6 in the vector dimension (6, 640, 832) represents six channels, i.e., the concatenation of two 3-channel images.

[0073] The concatenated vector is input into the optical flow model. The first layer of the optical flow model is a convolutional layer with 32 output channels and a stride of 2. Based on the stride, the convolutional kernel of the first layer is multiplied by the concatenated vector, i.e., convolution processing, to obtain the convolutional vector, which is the output of the first layer. The process is as follows: the number of channels changes from 6 input channels to 32 output channels, h changes from 640 input channels to 640 / stride = 640 / 2 = 320, and w changes from 832 input channels to 832 / stride = 832 / 2 = 416.

[0074] Similarly, in the optical flow model, layers 2-9a are convolutional layers, and the output of layer 1 is used to perform convolution processing on layers 2-9a. In the optical flow model, layers 9b and 9c are deconvolutional layers, and the output of layer 9a is used to perform deconvolution processing on layer 9b, and the output of layer 8 is used to perform deconvolution processing on layer 9c, thus obtaining the output of each layer.

[0075] The 10th network layer of the optical flow model is a splicing layer, which merges the outputs of the 6th, 9c, and 9b network layers to obtain the output of the 10th network layer.

[0076] The 11a layer of the optical flow model is a convolutional layer. The 11th layer is convolved based on the output of the 10th layer. The 11b and 11c layers of the optical flow model are deconvolutional layers. The 11b layer is deconvolved based on the output of the 11a layer, and the 11c layer is deconvolved based on the output of the 10th layer, thus obtaining the output of the 11a-11c layers.

[0077] The 12th network layer of the optical flow model is a splicing layer. Referring to the processing method of the 10th network layer, the outputs of the 4th network layer, the 11c network layer, and the 11b network layer are merged to obtain the output of the 12th network layer.

[0078] The 13a layer of the optical flow model is a convolutional layer, and the 13b-13c layers are deconvolutional layers. The output of the 13a-13c layers is obtained by referring to the processing method of the 11a-11c layers.

[0079] The 14th network layer of the optical flow model is a splicing layer. Referring to the processing method of the 10th network layer, the outputs of the 2nd network layer, the 13c network layer, and the 13b network layer are merged to obtain the output of the 14th network layer.

[0080] The 15th network layer of the optical flow model is a convolutional layer. Based on the output of the 14th network layer, convolutional processing is performed on the 15th network layer to obtain the output of the 15th network layer.

[0081] The 16th network layer of the optical flow model is an upsampling layer with a scale parameter of 4. The output of the 15th network layer is magnified according to the scale parameter to obtain the output of the 16th network layer. Among them, the output of the 16th network layer is the output of the optical flow model.

[0082] The vector output by the 16th network layer is upsampled to obtain an offset vector with a vector dimension of (2, H, W) between two images with a resolution of (H, W).

[0083] It should be noted that when the resolution (H, W) of the obtained image is greater than the height h and width w corresponding to the optical flow model, the obtained image needs to be downsampled to obtain an image with a resolution of (h, w); when the resolution (H, W) of the obtained image is less than the height h and width w corresponding to the optical flow model, the obtained image needs to be upsampled to obtain an image with a resolution of (h, w). The specific processes of image upsampling and downsampling in the embodiments of the present application are not limited, and the specific processes of image upsampling and downsampling in related technologies can be referred to.

[0084] It should also be noted that when the optical flow model outputs an offset vector with a vector dimension of (2, h, w) for two images with a resolution of (h, w), the offset vector with a vector dimension of (2, h, w) needs to be sampled to a vector dimension of (2, H, W) to obtain an offset vector with a vector dimension of (2, H, W) for the two images with a resolution of (H, W) obtained.

[0085] In some possible implementations of the embodiments of the present application, preferably, h < H and w < W. When h < H and w < W, the optical flow model in the embodiments of the present application can be called a sparse optical flow model.

[0086] In some possible implementations of the embodiments of the present application, sample images can be obtained. For any two sample images, the two sample images and their corresponding actual offset vectors can be input into the neural network model to obtain the offset prediction results corresponding to the two sample images. The neural network model is trained according to the actual offset vectors and the offset prediction results to obtain the optical flow model. Among them, when training the neural network model, the actual offset vector is used as the expected output of the offset vector corresponding to the two sample images.

[0087] In some possible implementations of the embodiments of the present application, when training the neural network model, a loop iteration training times threshold can be set. When the number of times of training the neural network model reaches the set loop iteration training times threshold, the training is stopped to obtain the optical flow model.

[0088] In some possible implementations of the embodiments of this application, when training the neural network model, the loss value of the neural network model can also be calculated. When the loss value of the neural network model reaches the minimum, training is stopped and an optical flow model is obtained.

[0089] In some possible implementations of the embodiments of this application, when training the neural network model, the accuracy of the neural network model prediction can also be calculated based on the actual offset vector and the above offset prediction results. When the accuracy of the neural network model prediction is greater than the accuracy threshold, training is stopped and an optical flow model is obtained.

[0090] Specifically, a subset of sample images can be selected as test samples. After the neural network model has been trained for a period of time, the test samples are input into the neural network model to obtain the offset prediction results for the test samples. Based on the offset prediction results and the actual offset vector of the test samples, the accuracy of the neural network model's prediction is calculated. When the accuracy of the neural network model's prediction is not greater than the accuracy threshold, the neural network model is trained again using the sample images. After a period of time, the test samples are input into the neural network model again, and the accuracy of the neural network model's prediction is calculated until the accuracy of the neural network model's prediction is greater than the accuracy threshold. The neural network model at this point is then used as the optical flow model.

[0091] This application does not limit the specific process of training a neural network model to obtain an optical flow model. For the specific process, please refer to the process of training a neural network model in related technologies. This application will not elaborate on it here.

[0092] In some possible implementations of the embodiments of this application, after obtaining the offset vector of the first image relative to the second image with a vector dimension of (2, H, W), the first image and the second image are registered according to the offset vector of the vector dimension of (2, H, W).

[0093] In some possible implementations of the embodiments of this application, in S103, during image registration, the second image is used as a reference image, and the first image is registered with it to obtain the first registered image corresponding to the first image.

[0094] In some possible implementations of the embodiments of this application, after S103, the image processing method provided in the embodiments of this application may further include: generating a target image based on the first registration image and the second image corresponding to the first image.

[0095] In some possible implementations of the embodiments of this application, generating a target image based on a first registration image and a second image corresponding to a first image includes: averaging the pixel values ​​of pixels at the same position in the first registration image and the second image to obtain the target image.

[0096] In some possible implementations of the embodiments of this application, generating a target image based on a first registration image and a second image corresponding to a first image includes: performing a weighted average of the second image and the first registration image according to the weights corresponding to the first registration image to obtain the target image.

[0097] The target image is obtained by weighted averaging the second image and the first registered image as shown in formula (1).

[0098]

[0099] In formula (1) above, Cr represents the target image; C j For the j-th first registration image, W j For the j-th first registration image C j The corresponding weight, C i This is the second image.

[0100] In some possible implementations of the embodiments of this application, when the target image is obtained by weighted averaging of the second image and the first registered image according to the weight corresponding to the first registered image, the pixel value of the pixel with coordinates (M, N) in the first registered image and the pixel value of the pixel with coordinates (M, N) in the second image can be weighted and averaged, and the obtained pixel value can be used as the pixel value of the pixel with coordinates (M, N) in the target image.

[0101] In some possible implementations of the embodiments of this application, before performing a weighted average of the second image and the first registered image according to the weight corresponding to the first registered image to obtain the target image, the image processing method provided in the embodiments of this application may further include: determining the weight corresponding to the first registered image according to the image gradient matrix of the second image and the image distance between the first registered image and the second image.

[0102] Specifically, the weight W corresponding to the j-th first registration image j As shown in formula (2).

[0103]

[0104] In formula (2) above, W j For the j-th first registration image C j The corresponding weights; λ is a hyperparameter used to control the weights of the registered images, with a value range of (0, 1); D i -D j For the j-th first registration image C j With the second image C i Image distance; T i For the second image C iThe image gradient matrix.

[0105] In some possible implementations of the embodiments of this application, when calculating the j-th first registration image C j With the second image C i When calculating the image distance, the j-th first registration image C can be calculated. j A certain pixel in the second image C i The distance between the j-th pixel and the corresponding pixel is calculated, and then the average of the distances of all pixels is used as the first registered image C. j With the second image C i Image distance.

[0106] It should be noted that the j-th first registration image C j The pixel with coordinates (M, N) in the middle image and the second image C i The corresponding pixel with coordinates (M, N) in the middle.

[0107] The j-th first registration image C j The pixel with coordinates (M, N) in the middle image and the second image C i The pixel distance d of the pixel with coordinates (M, N) in the middle (M,N) It can be shown in the following formula (3).

[0108]

[0109] In formula (3) above, C represents the j-th first registration image. j The red, green, and blue components of the pixel value of the pixel with coordinates (M, N); The second image C is shown below. i The pixel value at coordinates (M, N) contains the red, green, and blue components. In other words, the j-th first registered image C... j The pixel value of the pixel with coordinates (M, N) is Second image C i The pixel value of the pixel with coordinates (M, N) is

[0110] In some possible implementations of the embodiments of this application, in S101, one image can be randomly selected from the at least two acquired images as the second image, and each of the other images in the at least two images, excluding the second image, can be used as the first image.

[0111] In some possible implementations of the embodiments of this application, S101 may include: determining a second image and a first image from at least two acquired images based on image sharpness.

[0112] In some possible implementations of the embodiments of this application, the image with the highest image clarity among the at least two acquired images can be used as the second image, and each of the other images among the at least two acquired images besides the second image can be used as the first image.

[0113] In some possible implementations of the embodiments of this application, determining the second image and the first image from at least two acquired images based on image sharpness may include: sorting the at least two images in descending order of sharpness; using the image ranked first as the second image; and using each of the top N images except the second image as a first image, where N is a positive integer greater than or equal to 2.

[0114] For example, suppose we acquire 5 images, namely image A, image B, image C, image D, and image E, where N=3. The 5 images are sorted in descending order of sharpness, resulting in the order: image D, image B, image A, image E, and image C. Image D is then designated as the second image, and both images B and A are designated as the first images.

[0115] The process of generating a target image is explained below using a photo-taking scenario as an example. Figure 4 This is a schematic diagram illustrating the process of synthesizing an image from a photographic scene according to an embodiment of this application. The image synthesis process includes two stages: an offline training stage and a photograph synthesis stage.

[0116] During the offline training phase, sample images and corresponding actual offset vectors are input into the neural network model to train the neural network model and obtain the optical flow model.

[0117] In the image synthesis stage, multiple images are first captured using a camera (for example, five images are captured in a burst, namely image A, image B, image C, image D and image E). The five images are sorted in descending order of sharpness, resulting in image D, image B, image A, image E and image C. Image D is then used as the second image, and both image B and image A are used as the first image.

[0118] Images D, B, and A with resolutions of (H, W) are downsampled to obtain images D0, B0, and A0 with resolutions of (h, w).

[0119] The vectors of image D0 and image B0 with resolution (h, w) are concatenated. The concatenated vector with dimension (6, h, w) is input into the optical flow model to obtain the offset vector of image B0 relative to image D0 with dimension (2, h, w). This offset vector with dimension (2, h, w) is upsampled to dimension (2, H, W) to obtain the offset vector of image B relative to image D with dimension (2, H, W). Using this offset vector with dimension (2, H, W), image B is registered with image D as the reference image to obtain the registered image B1 corresponding to image B.

[0120] Similarly, the vectors of image D0 and image A0 with resolution (h, w) are concatenated. The concatenated vector with dimension (6, h, w) is input into the optical flow model to obtain the offset vector of image A0 relative to image D0 with dimension (2, h, w). This offset vector with dimension (2, h, w) is upsampled to dimension (2, H, W) to obtain the offset vector of image A relative to image D with dimension (2, H, W). Using this offset vector with dimension (2, H, W), image A is registered with image D as the reference image to obtain the registered image A1 corresponding to image A.

[0121] Images D, B1, and A1 are fused to obtain the target image, i.e., the image synthesized from the photographs. When fusing images D, B1, and A1 to obtain the target image, a weighted average is applied to images D, B1, and A1.

[0122] In some possible implementations of the embodiments of this application, the pre-trained optical flow model can be stored in the read-only memory (ROM) of the electronic device. When the user needs to combine multiple photos taken by the electronic device in burst mode, the optical flow model in the ROM of the electronic device is called to synthesize the image.

[0123] It should be noted that the image processing method provided in this application embodiment can be executed by an image processing device. This application embodiment uses an image processing device executing the image processing method as an example to illustrate the image processing device provided in this application embodiment.

[0124] Figure 5 This is a schematic diagram of the structure of the image processing apparatus provided in an embodiment of this application. Figure 5 As shown, the image processing apparatus 500 may include:

[0125] The acquisition module 501 is used to acquire the first image and the second image;

[0126] The splicing module 502 is used to splice the first vector of the first image and the second vector of the second image to obtain the third vector;

[0127] The first determining module 503 is used to input the third vector into the pre-trained optical flow model to obtain the offset vector of the first image relative to the second image, wherein the optical flow model is used to determine the offset vector of the two images.

[0128] The registration module 504 is used to register the first image and the second image according to the offset vector.

[0129] In this embodiment, by concatenating the vectors of two images and then inputting the concatenated vector into a pre-trained optical flow model, the offset vector of one image relative to the other can be obtained. The two images can then be registered based on this offset vector. Compared to related technologies, this method eliminates the need for Fourier transform to obtain the offset vector for image registration, resulting in lower computational complexity and improved image registration efficiency.

[0130] In some possible implementations of the embodiments of this application, the image processing apparatus 500 further includes:

[0131] The generation module is used to generate a target image based on the first registration image and the second image corresponding to the first image.

[0132] In some possible implementations of the embodiments of this application, the generation module is specifically used for:

[0133] Based on the weights corresponding to the first registered image, the second image and the first registered image are weighted and averaged to obtain the target image.

[0134] In some possible implementations of the embodiments of this application, the image processing apparatus 500 further includes:

[0135] The second determining module is used to determine the weights corresponding to the first registered image based on the image gradient matrix of the second image and the image distance between the first registered image and the second image.

[0136] In some possible implementations of the embodiments of this application, the acquisition module is specifically used for:

[0137] Sort the acquired at least two images in descending order of sharpness;

[0138] Use the image ranked first as the second image;

[0139] Take each of the top N images except the second image as a first image, where N is a positive integer greater than or equal to 2.

[0140] In some possible implementations of this application, the optical flow model includes a feature extraction layer and a feature recovery layer. The feature extraction layer includes at least one first convolutional layer, used to extract the image feature vector corresponding to the third vector through different convolutional kernel sizes and strides. The feature recovery layer includes at least one second convolutional layer, at least one deconvolutional layer, at least one feature concatenation layer, and a sampling layer. The second convolutional layer is used to convolve the image feature vector output by the previous layer. The deconvolutional layer is used to deconvolve the image feature vector output by the previous layer. The feature concatenation layer is used to concatenate the image feature vector output by the deconvolutional layer with the image feature vector of the same dimension extracted by at least one first convolutional layer. The sampling layer is used to adjust the image feature vector output by the previous layer to have the same dimension as the third vector to obtain an offset vector.

[0141] In some possible implementations of the embodiments of this application, the activation function of the optical flow model is a non-saturating activation function.

[0142] The image processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0143] The image processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0144] The image processing apparatus provided in this application embodiment can achieve... Figures 1 to 4 The various processes in the image processing method embodiments are not described again here to avoid repetition.

[0145] Optional, such as Figure 6 As shown, this application embodiment also provides an electronic device 600, including a processor 601 and a memory 602. The memory 602 stores a program or instructions that can run on the processor 601. When the program or instructions are executed by the processor 601, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0146] In some possible implementations of the embodiments of this application, processor 601 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0147] In some possible implementations of embodiments of this application, memory 602 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, typically, memory 602 includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the image processing method according to embodiments of this application.

[0148] Figure 7 This is a schematic diagram of the hardware structure of an electronic device that implements the embodiments of this application.

[0149] The electronic device 700 includes, but is not limited to, components such as: radio frequency unit 701, network module 702, audio output unit 703, input unit 704, sensor 705, display unit 706, user input unit 707, interface unit 708, memory 709, and processor 710.

[0150] Those skilled in the art will understand that the electronic device 700 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 710 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 7 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0151] The processor 710 is used to: acquire a first image and a second image; concatenate a first vector of the first image and a second vector of the second image to obtain a third vector; input the third vector into a pre-trained optical flow model to obtain an offset vector of the first image relative to the second image, wherein the optical flow model is used to determine the offset vectors of the two images; and register the first image and the second image according to the offset vector.

[0152] In this embodiment, by concatenating the vectors of two images and then inputting the concatenated vector into a pre-trained optical flow model, the offset vector of one image relative to the other can be obtained. The two images can then be registered based on this offset vector. Compared to related technologies, this method eliminates the need for Fourier transform to obtain the offset vector for image registration, resulting in lower computational complexity and improved image registration efficiency.

[0153] In some possible implementations of embodiments of this application, the processor 710 is further configured to:

[0154] The target image is generated based on the first registration image and the second image corresponding to the first image.

[0155] In some possible implementations of the embodiments of this application, the processor 710 is specifically used for:

[0156] Based on the weights corresponding to the first registered image, the second image and the first registered image are weighted and averaged to obtain the target image.

[0157] In some possible implementations of embodiments of this application, the processor 710 is further configured to:

[0158] The weights corresponding to the first registered image are determined based on the image gradient matrix of the second image and the image distance between the first registered image and the second image.

[0159] In some possible implementations of the embodiments of this application, the processor 710 is specifically used for:

[0160] Sort the acquired at least two images in descending order of sharpness;

[0161] Use the image ranked first as the second image;

[0162] Take each of the top N images except the second image as a first image, where N is a positive integer greater than or equal to 2.

[0163] In some possible implementations of this application, the optical flow model includes a feature extraction layer and a feature recovery layer. The feature extraction layer includes at least one first convolutional layer, used to extract the image feature vector corresponding to the third vector through different convolutional kernel sizes and strides. The feature recovery layer includes at least one second convolutional layer, at least one deconvolutional layer, at least one feature concatenation layer, and a sampling layer. The second convolutional layer is used to convolve the image feature vector output by the previous layer. The deconvolutional layer is used to deconvolve the image feature vector output by the previous layer. The feature concatenation layer is used to concatenate the image feature vector output by the deconvolutional layer with the image feature vector of the same dimension extracted by at least one first convolutional layer. The sampling layer is used to adjust the image feature vector output by the previous layer to have the same dimension as the third vector to obtain an offset vector.

[0164] In some possible implementations of the embodiments of this application, the activation function of the optical flow model is a non-saturating activation function.

[0165] It should be understood that, in this embodiment, the input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042. The GPU 7041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 706 may include a display panel 7061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 707 includes at least one of a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0166] The memory 709 can be used to store software programs and various data. The memory 709 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 709 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 709 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0167] Processor 710 may include one or more processing units; optionally, processor 710 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 710.

[0168] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0169] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer-readable storage medium, and examples of computer-readable storage media include non-transitory computer-readable storage media such as ROM, RAM, magnetic disks, or optical disks.

[0170] This application also provides a chip, including a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described image processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0171] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0172] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described image processing method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0173] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0174] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0175] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method, characterized by, The method comprises: obtaining a first image and a second image; splicing a first vector of the first image and a second vector of the second image to obtain a third vector; inputting the third vector into a pre-trained optical flow model to obtain an offset vector of the first image relative to the second image, wherein the optical flow model is used to determine the offset vector of two images; registering the first image and the second image according to the offset vector.

2. The method of claim 1, wherein, After the registering the first image and the second image according to the offset vector, the method further comprises: generating a target image according to a first registered image corresponding to the first image and the second image.

3. The method of claim 2, wherein, The generating a target image according to a first registered image corresponding to the first image and the second image comprises: performing weighted average on the second image and the first registered image according to a weight corresponding to the first registered image to obtain a target image.

4. The method of claim 3, wherein, Before the performing weighted average on the second image and the first registered image according to a weight corresponding to the first registered image to obtain a target image, the method further comprises: determining the weight corresponding to the first registered image according to an image gradient matrix of the second image and an image distance between the first registered image and the second image.

5. The method of claim 1, wherein, The obtaining a first image and a second image comprises: sorting at least two images obtained in order of clarity from large to small; taking a first-ranked image as the second image; taking each image except the second image in the top N images as a first image, wherein N is a positive integer greater than or equal to 2.

6. The method of claim 1, wherein, The optical flow model comprises a feature extraction layer and a feature recovery layer, wherein the feature extraction layer comprises at least one first convolutional layer for extracting an image feature vector corresponding to the third vector through different convolution kernel sizes and steps, the feature recovery layer comprises at least one second convolutional layer, at least one deconvolutional layer, at least one feature splicing layer and a sampling layer, the second convolutional layer is used to convolve the image feature vector output by the previous layer, the deconvolutional layer is used to deconvolve the image feature vector output by the previous layer, the feature splicing layer is used to splice the image feature vector output by the deconvolutional layer with the image feature vector of the same dimension extracted by the at least one first convolutional layer, and the sampling layer is used to adjust the image feature vector output by the previous layer to the same dimension as the third vector to obtain the offset vector.

7. An image processing apparatus characterized by comprising: The device comprises: an acquisition module configured to acquire a first image and a second image; a splicing module configured to splice a first vector of the first image and a second vector of the second image to obtain a third vector; a first determination module configured to input the third vector into a pre-trained optical flow model to obtain an offset vector of the first image relative to the second image, wherein the optical flow model is used to determine the offset vector of two images; a registration module configured to register the first image and the second image according to the offset vector.

8. The apparatus of claim 7, wherein, The device further comprises: The generating module is configured to generate a target image according to the first image and the second image.

9. The apparatus of claim 8, wherein, The generating module is specifically configured to: perform weighted averaging on the second image and the first registration image according to the weight corresponding to the first registration image to obtain the target image.

10. The apparatus of claim 9, wherein, The device further includes: The second determining module is configured to determine the weight corresponding to the first registration image according to an image gradient matrix of the second image and an image distance between the first registration image and the second image.

11. The apparatus of claim 7, wherein, The acquiring module is specifically configured to: sort the acquired at least two images in order of clarity from large to small; rank the first image as the second image; rank each image other than the second image in the top N images as a first image, where N is a positive integer greater than or equal to 2.

12. The apparatus of claim 7, wherein, The optical flow model includes a feature extraction layer and a feature recovery layer, wherein the feature extraction layer includes at least one first convolutional layer for extracting an image feature vector corresponding to the third vector through different convolution kernel sizes and steps, the feature recovery layer includes at least one second convolutional layer, at least one deconvolutional layer, at least one feature splicing layer, and a sampling layer, the second convolutional layer is configured to convolve the image feature vector output by the previous layer, the deconvolutional layer is configured to deconvolve the image feature vector output by the previous layer, the feature splicing layer is configured to splice the image feature vector output by the deconvolutional layer with the image feature vector of the same dimension extracted by the at least one first convolutional layer, and the sampling layer is configured to adjust the image feature vector output by the previous layer to the same dimension as the third vector to obtain the offset vector.

13. An electronic device, comprising: The electronic device includes a processor and a memory, the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the image processing method according to any one of claims 1 to 6.

14. A readable storage medium, characterized by, The readable storage medium stores programs or instructions, and the programs or instructions are executed by the processor to implement the steps of the image processing method according to any one of claims 1 to 6.

Images