Shooting method and device

Abstract

The present application discloses a shooting method and a device thereof, belonging to the field of photography technology. The shooting method comprises: shooting a second image according to image information of a first image; generating a first blurred image according to the first image, depth information of the first image and depth information of the second image; generating a second blurred image according to a target imaging model, the first image and the second image; fusing the first blurred image and the second blurred image to obtain a target blurred image.

Description

Technical Field

[0001] The present application belongs to the field of photographic technology, and specifically relates to a photographing method and a device thereof. Background Art

[0002] When users take pictures with electronic devices, in order to present a better visual effect, they often highlight the main object of the image by blurring the background of the image. At present, the commonly used image blurring method is: according to the position of the camera device in the electronic device, the object distance from the corresponding point of each pixel point of the captured image in the shooting scene to the camera lens is calculated, that is, the depth map of the captured image is determined, and then the pixels of different depths are blurred to different degrees, so as to simulate the image result of aperture blur.

[0003] However, the above-mentioned method of blurring the pixels in the captured image to different degrees according to the depth map to blur the image, when the blurring degree of the pixels is inappropriate, it is easy to make the background part and the main part of the blurred image not smoothly connected, thereby making the imaging effect unnatural. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a shooting method and a device thereof, which can improve the smoothness of the connection between the background part and the main part of the blurred image, thereby ensuring the naturalness of the imaging effect.

[0005] In a first aspect, an embodiment of the present application provides a shooting method, the method comprising: shooting a second image based on image information of a first image; generating a first blurred image based on the first image, depth information of the first image and depth information of the second image; generating a second blurred image based on a target imaging model, the first image and the second image; and fusing the first blurred image and the second blurred image to obtain a target blurred image.

[0006] In a second aspect, an embodiment of the present application provides a shooting device, which includes: a shooting unit, used to shoot a second image based on image information of a first image; a first processing unit, used to generate a first blurred image based on the first image, depth information of the first image and depth information of the second image; a second processing unit, used to generate a second blurred image based on a target imaging model, the first image and the second image; and a third processing unit, used to fuse the first blurred image and the second blurred image to obtain a target blurred image.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the shooting method of the first aspect are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium having a program or instruction stored thereon, which, when executed by a processor, implements the steps of the shooting method of the first aspect.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of the shooting method of the first aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the shooting method of the first aspect.

[0011] In the shooting method provided in the embodiment of the present application, the second image is obtained by shooting according to the image information of the first image. On this basis, the first blurred image is generated according to the first image, the depth information of the first image and the depth information of the second image, and the second blurred image is generated according to the target imaging model, the first image and the second image. Further, the first blurred image and the second blurred image are fused to obtain the target blurred image. Through the above shooting method, in the process of shooting the blurred image, the first blurred image is generated based on the depth information of the first image and the second image obtained by shooting, and the second virtual image is generated according to the first image, the second image and the target imaging model, and then the obtained first virtual image and the second virtual image are fused to obtain the target blurred image with a blurred effect. Among them, the second image is shot based on the image information of the first image. In this way, the shooting of the second image is guided based on the first image, and then the two blurred images determined by the first image and the second image are fused to obtain the target blurred image with a blurred effect, which improves the smoothness of the connection between the background part and the main part in the target blurred image, thereby ensuring the naturalness of the imaging effect of the target blurred image. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A schematic diagram of a process flow of a photographing method provided in an embodiment of the present application;

[0013] Figure 2 One of the principle diagrams of the shooting method provided in the embodiment of the present application;

[0014] Figure 3 The second schematic diagram of the photographing method provided in the embodiment of the present application;

[0015] Figure 4 A structural block diagram of a photographing device provided in an embodiment of the present application;

[0016] Figure 5A structural block diagram of an electronic device provided in an embodiment of the present application;

[0017] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0019] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0020] The embodiment of the first aspect of the present application proposes a shooting method. The execution subject of the technical solution of the shooting method provided by the embodiment of the present application can be a shooting device, which can be determined according to actual use requirements and is not limited by the embodiment of the present application. In order to more clearly describe the shooting method provided by the embodiment of the present application, the following method embodiment uses the execution subject of the shooting method as a shooting device for exemplary description.

[0021] The following is a detailed description of the shooting method provided in the embodiment of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0022] like Figure 1 As shown, the embodiment of the present application provides a shooting method, which may include the following S102 to S108:

[0023] S102: Capture a second image according to the image information of the first image.

[0024] The shooting method proposed in the embodiment of the present application is applied to an electronic device, which has at least two camera devices, and the at least two camera devices are located on the same side of the electronic device. In actual application, the electronic device can be a smart phone, a tablet computer or other electronic device with at least two rear cameras, and no specific limitation is made here.

[0025] The zoom factor of the first image is a first value, the zoom factor of the second image is a second value, and the first value is smaller than the second value. In this way, the first image and the second image with different zoom factors are obtained by shooting, so as to prepare for the subsequent generation of the first blurred image and the second blurred image based on the first image and the second image, so as to obtain the corresponding blurred image by extracting and processing the information of the first image and the second image with different zoom factors.

[0026] Furthermore, the image information of the first image includes subject object information and focus information such as a focus area of ​​the first image.

[0027] Furthermore, in actual application, the first image may be obtained by photographing a first camera device of the electronic device, and the second image may be obtained by photographing a second camera device of the electronic device.

[0028] Specifically, when shooting a blurred image of a target, a first image with a zoom factor of a first value is obtained by shooting with a first camera device of the electronic device. On this basis, image information such as subject information and focus information of the first image is obtained, and shooting parameters of a second camera device of the electronic device are set based on the obtained image information, such as setting the focus center point of the second camera device, and then based on the set shooting parameters, the second camera device is used to shoot with a zoom factor of a second value.

[0029] In addition, it should be noted that in actual application, when the first image is captured by the first camera device, the shooting parameters of the first camera device set by the user can also be obtained in real time, such as the focus area, the shooting subject, etc. of the first camera device, and then the shooting parameters of the second camera device are set according to the shooting parameters of the first camera device, so as to obtain the first image and the second image with different zoom factors through the first camera device and the second camera device.

[0030] S104: Generate a first blurred image according to the first image, the depth information of the first image, and the depth information of the second image.

[0031] Among them, the depth information of the above-mentioned first image includes the distance between the corresponding point position of each pixel point in the first image in the actual shooting scene and the first camera device, and the depth information of the second image includes the distance between the corresponding point position of each pixel point in the second image in the actual shooting scene and the second camera device.

[0032] Specifically, after the first image and the second image are captured, the depth information of the first image and the depth information of the second image are obtained, and a target depth map is determined according to the obtained depth information. On this basis, based on the determined target depth map, the first image is blurred, specifically, each pixel corresponding to a different depth value in the first image is blurred to a different extent, so as to simulate the imaging effect of aperture blur, thereby generating the above-mentioned first blurred image.

[0033] S106: Generate a second blurred image according to the target imaging model, the first image and the second image.

[0034] The target imaging model is used to process the first image and the second image and output the corresponding blurred image. That is, for the target imaging model, the first image and the second image are its input objects, and the second blurred image is its output object.

[0035] Specifically, in the process of shooting a blurred image of the target, after the first image and the second image are captured, the first image and the second image are input into the target imaging model as input objects, and the first image and the second image are processed through the target imaging model to generate the second blurred image.

[0036] Specifically, the target imaging model processes the first image and the second image based on the image information of the first image and the image information of the second image, and the image information may specifically include the depth semantic information of the corresponding image, and the depth semantic information is used to indicate the image content and depth information of the corresponding image. Specifically, when the target imaging model processes the first image and the second image, the target imaging model compares the depth semantic information of the first image and the second image. On this basis, when the depth semantic information of the first image and the second image matches each other, that is, when the first image and the second image contain the same depth semantic information, the image content corresponding to the depth semantic information of this part is enhanced; and when the depth semantic information of the first image and the second image does not match each other, that is, when only one of the first image and the second image contains a certain depth semantic information, the image content corresponding to the depth semantic information is blurred.

[0037] That is to say, the target imaging model enhances the same image contents in the first image and the second image, and blurs the different image contents in the first image and the second image, so as to obtain the above-mentioned second blurred image.

[0038] S108: Fusing the first blurred image and the second blurred image to obtain a target blurred image.

[0039] The first blurred image is a blurred image obtained based on depth information of the first image and the second image, and the second blurred image is a blurred image obtained based on depth semantic information of the first image and the second image.

[0040] Specifically, after obtaining the first blurred image and the second blurred image, the first blurred image and the second blurred image are fused according to a certain weight ratio to obtain a target blurred image with a blurred effect and a natural connection between the blurred part and the enhanced part of the image.

[0041] Among them, for the weight ratio when fusing the first blurred image and the second blurred image, those skilled in the art can set it according to actual conditions, and no specific limitation is made here.

[0042] Through the above-mentioned shooting method provided by the embodiment of the present application, in the process of shooting a blurred image, a second image is shot according to the image information of the first image obtained by shooting. On this basis, a first blurred image is generated according to the first image, the depth information of the first image and the depth information of the second image, and at the same time, a second blurred image is generated according to the target imaging model, the first image and the second image. Further, the first blurred image and the second blurred image are fused to obtain a target blurred image. That is, through the above-mentioned shooting method, in the process of shooting a blurred image, a first blurred image is generated based on the depth information of the first image and the second image obtained by shooting, and a second virtual image is generated according to the first image, the second image and the target imaging model, and then the obtained first virtual image and the second virtual image are fused to obtain a target blurred image with a blurred effect. Among them, the second image is shot based on the guidance of the image information of the first image. In this way, the shooting of the second image is guided by the first image, and then the two blurred images determined by the first image and the second image are fused to obtain a target blurred image with a blurred effect, thereby improving the smoothness of the connection between the background part and the main part in the target blurred image, thereby ensuring the naturalness of the imaging effect of the target blurred image.

[0043] In the embodiment of the present application, the above S102 may specifically include the following S102a to S102d:

[0044] S102a: Identify the first image and determine the target object in the first image.

[0045] The target object is located in a focus area in the first image.

[0046] Specifically, after the first image is captured, target recognition is performed on the first image, and the main object in the first image, namely the target object, is determined according to the target recognition result, and the focus area when the first image is captured is determined according to the recognition result.

[0047] S102b: Extract target object information in the first image.

[0048] Specifically, after determining the target object in the first image, normalized histogram processing is performed on the first image, and target object information in the first image is extracted according to the processing result.

[0049] S102c: Compare the target object information with the scene information in the shooting preview interface, and determine the target area in the shooting preview interface according to the comparison result.

[0050] The shooting preview interface is a shooting preview interface of a second camera device of the electronic device, and the scene information in the shooting preview interface is used to indicate the scene content captured by the second camera device.

[0051] Specifically, after extracting the target object information in the first image, the target object information is compared with the scene information in the shooting preview interface of the second camera device. When the scene information and the target object information match each other, that is, when a certain part of the scene information is consistent with the target object information, the interface area corresponding to the part of the scene information is determined as the above-mentioned target area.

[0052] In actual application, the electronic device can use the center point of the shooting preview interface of the second camera device as the initial point to gradually compare and search outward to compare the above scene information and target object information pixel by pixel. When the scene information at a certain pixel is consistent with part of the object information in the target object information, the pixel is determined as the target pixel, and the above target area is then formed based on the determined multiple target pixels.

[0053] S102d: Using the target area as the focus area of ​​the shooting preview interface, and shooting to obtain a second image.

[0054] Specifically, after the target area is determined, the target area is determined as the focus area of ​​the shooting preview interface of the second camera device, so as to obtain a second image whose focus object, ie, main object, is the same as the first image.

[0055] In the above-mentioned embodiment provided by the present application, when shooting the second image according to the image information of the first image, specifically, the first image is identified, the target object in the focus area of ​​the first image is determined, and the target object information in the first image is extracted. On this basis, the target object information is compared with the scene information in the shooting preview interface, and then the target area in the shooting preview interface is determined according to the comparison result, and the target area is used as the focus area of ​​the shooting preview interface to shoot the second image. In this way, the shooting of the second image is guided based on the first image obtained, so that the focus object, that is, the main object, of the second image obtained is the same as the above-mentioned first image, thereby ensuring the accuracy of the subsequent determination of the first blurred image and the second blurred image based on the first image and the second image, thereby ensuring the imaging effect of the target blurred image.

[0056] In the embodiment of the present application, S106 may specifically include the following S106a to S106e:

[0057] S106a: Input the first image and the second image into the target imaging model.

[0058] The target imaging model is used to process the first image and the second image and output corresponding blurred images.

[0059] Specifically, after the first image and the second image are captured, the first image and the second image are input into the target imaging model as input objects, so that the target imaging model processes the first image and the second image, thereby outputting corresponding blurred images.

[0060] S106b: Divide the first image into M first image blocks and divide the second image into N second image blocks by using the target imaging model.

[0061] Wherein, the above-mentioned M and N are both positive integers greater than 1.

[0062] Further, the ratio of M to N is related to the zoom factor of the first image and the zoom factor of the second image. Specifically, the ratio of M to N is related to the ratio of the second value to the first value. In actual application, the ratio of M to N can be the square of the ratio of the second value to the first value.

[0063] Specifically, after the first image and the second image are input into the target imaging model, the target imaging model automatically performs segmentation and stacking processing on the input first image and the second image.

[0064] Specifically, the target imaging model divides the input first image into M first image blocks, and divides the input second image into N second image blocks. On this basis, after the target imaging model divides the first image into M first image blocks, and divides the second image into N second image blocks, the M first image blocks are stacked and arranged according to the first target order to obtain a first target image containing M image channels, and the N second image blocks are stacked and arranged according to the second target order to obtain a second target image containing N image channels, in preparation for subsequent processing.

[0065] S106c: extracting image information of M first image blocks according to the first convolutional network through the target imaging model to obtain first image information, and extracting image information of N second image blocks according to the second convolutional network to obtain second image information.

[0066] The first image information may specifically be image semantic information in the first image block, and the second image information may specifically be image semantic information in the second image block.

[0067] Furthermore, the first convolutional network is different from the second convolutional network.

[0068] In the actual application process, the above-mentioned first convolution network may specifically include 3 convolution blocks (ConvBlock), wherein each convolution block has two convolution kernels of size 3×3 and inner margin (padding) of 1. That is, the above-mentioned first convolution network may specifically include 6 convolution kernels. Further, the above-mentioned second convolution network may specifically include 4 convolution blocks and a maximum pooling layer (MaxPool), wherein each convolution block has two convolution kernels of size 3×3 and inner margin of 1, and the size of the maximum pooling layer is 2×2. That is, the above-mentioned second convolution network may specifically include 8 convolution kernels.

[0069] Specifically, the target imaging model is provided with a first convolutional network and a second convolutional network. After the target imaging model performs segmentation and stacking processing on the first image and the second image to obtain the first target image and the second target image, the target imaging model continues to perform semantic information extraction processing on the obtained first target image and the second target image. Specifically, the target imaging model extracts the image semantic information of each image channel in the first target image through the first convolutional network therein to obtain the first image information, and at the same time, the target imaging model extracts the image semantic information of each image channel in the second target image through the second convolutional network therein to obtain the second image information.

[0070] S106d: Fusing the first image information and the second image information through a target imaging model to obtain fused image information.

[0071] Specifically, after the target imaging model extracts the first image information and the second image information, the target imaging model continues to perform semantic information comparison and fusion processing on the first image information and the second image information to obtain fused image information.

[0072] Specifically, when the target imaging model compares and fuses the first image information and the second image information, the target imaging model multiplies the first image information of each image channel in the first target image with the second image information of each image channel in the second target image in sequence to obtain a plurality of sub-fusion information, and then rearranges the plurality of sub-fusion information in a third target order to obtain the fused image information.

[0073] Among them, the above-mentioned third target order is the reverse order of the above-mentioned first target order.

[0074] S106e: Processing the fused image information according to the target convolutional network through the target imaging model to generate a second blurred image.

[0075] Among them, a target convolutional network is also provided in the above target imaging model.

[0076] In actual application, the target convolution network may specifically include 3 convolution blocks, wherein each convolution block has two convolution kernels of size 3×3 and inner margin 1. That is, the target convolution network may specifically include 6 convolution kernels.

[0077] Specifically, after the target imaging model compares and fuses the first image information and the second image information to obtain the above-mentioned fused image information, the target imaging model further processes the obtained fused image information through the target convolutional network therein to obtain the above-mentioned second blurred image, and outputs the second blurred image.

[0078] For example, Figure 2As shown in (a) of FIG. 1 , after a first image 202 with a zoom factor of 1 and a second image 204 with a zoom factor of 2 are input into the target imaging model, the target imaging model divides and stacks the first image 202 and the second image 204. Specifically, the target imaging model divides the first image 202 into 16 first image blocks 206, and stacks and arranges the 16 first image blocks 206 in a first target order to obtain a first target image 208 containing 16 image channels, and the target imaging model divides the second image 204 into 4 second image blocks 210, and stacks and arranges the 4 second image blocks 210 in a second target order to obtain a second target image 212 containing 4 image channels. On this basis, the target imaging model processes the first target image 208 through the first convolutional network 224 therein to extract the image semantic information of each image channel in the first target image 208 to obtain the first image information 214, and the target imaging model processes the second target image 212 through the second convolutional network 226 therein to extract the image semantic information of each image channel in the second target image 212 to obtain the second image information 216. On this basis, the target imaging model performs channel-by-channel comparison and fusion processing on the first image information 214 and the second image information 216, that is, the image information of each image channel in the first target image 208 is multiplied by the image information of each image channel in the second target image 212 in sequence to obtain a plurality of sub-fusion information, and the target imaging model then rearranges the plurality of sub-fusion information in the reverse order of the segmentation stacking to obtain the fused image information 218. Further, the target imaging model processes the fused image information 218 through the target convolutional network 220 therein to obtain the second blurred image 222 and outputs it.

[0079] in, Figure 2 The “→” in (a) indicates the direction of data flow. Indicates channel-by-channel comparison and fusion processing. The value on the left of “|” is the number of input channels, and the value on the right of “|” is the number of output channels. Figure 2 The specific structure of each convolution block ConvBlock in (a) is as follows Figure 2 As shown in (b) in .

[0080] In the above-mentioned embodiment provided by the present application, when generating the second blurred image, the first image and the second image are input into the target imaging model, and the first image and the second image are processed by the target imaging model to output the second blurred image. Specifically, after the first image and the second image are input into the target imaging model, the target imaging model divides the first image into M first blocks and the second image into N second blocks based on the ratio between the zoom factor of the first image and the zoom factor of the second image. On this basis, the target imaging model extracts the image information of the M first blocks through the first convolutional network therein to obtain the first image information, and extracts the image information of the N second blocks through the second convolutional network therein to obtain the second image information. Furthermore, the target imaging model performs a fusion process on the extracted first image information and the second image information to obtain fused image information, and processes the fused image information according to the target convolutional network therein to obtain the second blurred image. In this way, based on the depth semantic information of the first image and the second image, the first image and the second image are sequentially segmented and stacked, semantic information is extracted, and semantic information is compared and fused through the target imaging model, and the above-mentioned second blurred image is output, thereby ensuring the smoothness of the blurring transition of the second blurred image, thereby ensuring the naturalness of the effect of the subsequent target blurred image.

[0081] In the embodiment of the present application, the above S108 may specifically include the following S108a:

[0082] S108a: According to the first weight and the second weight, the first blurred image is merged with the second blurred image to obtain a target blurred image.

[0083] The first weight corresponds to the first virtual image, and the second weight corresponds to the second virtual image.

[0084] Specifically, after obtaining the first blurred image and the second blurred image, a first weight is assigned to the first blurred image, and a second weight is assigned to the second blurred image, and then the first blurred image and the second blurred image are fused according to the corresponding weights, that is, the first blurred image and the second blurred image are fused according to a certain weight ratio, so as to obtain a target blurred image with a blurred effect and a natural connection between the blurred part of the image and the enhanced part of the image.

[0085] In actual application, the first blurred image and the second blurred image can be fused using the following formula (1):

[0086]

[0087] Among them, I f represents the target blurred image, I crepresents the first blurred image, I p represents the second blurred image, b is the first weight, a is the second weight, a and b are both positive real numbers, and a+b=1.

[0088] Specifically, the first weight may be 0.7, and the second weight may be 0.3. In actual application, those skilled in the art may set specific values ​​of the first weight and the second weight according to actual conditions, and no specific limitation is made here.

[0089] In the above-mentioned embodiment provided by the present application, when determining the target blurred image, specifically, a first weight is assigned to the first blurred image, and a second weight is assigned to the second blurred image, and then the first blurred image and the second blurred image are fused according to the corresponding weights to obtain the target blurred image. In this way, according to a certain weight ratio, the first blurred image determined according to the depth information of the first image and the second image, and the second blurred image determined according to the depth semantic information of the first image and the second image are fused to obtain a target blurred image with a blurred effect, which improves the smoothness of the connection between the background part and the main part in the target blurred image, thereby ensuring the naturalness of the imaging effect of the target blurred image.

[0090] In the embodiment of the present application, the above-mentioned shooting method may further specifically include the following S110 to S114:

[0091] S110: Establish an initial imaging model and obtain multiple groups of initial training samples.

[0092] Among them, the above-mentioned initial imaging model is used to process the input image and output the corresponding blurred image. The model parameters of the initial imaging model are different from the model parameters of the above-mentioned target imaging model. By optimizing and adjusting the model parameters of the initial imaging model, the above-mentioned target imaging model can be obtained.

[0093] Furthermore, each group of initial training samples includes a third image, a fourth image and a fifth image, the third image is taken by the first electronic device at a first zoom factor, the fourth image is taken by the first electronic device at a second zoom factor, and the fifth image is taken by the second electronic device at the target aperture size, and the first zoom factor is smaller than the second zoom factor.

[0094] Furthermore, the first electronic device is an electronic device with shooting function, algorithm focusing function and image processing function, such as a smart phone, and the second electronic device is an electronic device with physical focusing function, namely aperture focusing function, such as a SLR camera.

[0095] In the actual application process, when taking three images in each group of initial training samples, such as Figure 3As shown, the bracket can be used to Figure 3 The first electronic device 302 with a first zoom factor, the first electronic device 304 with a second zoom factor, and the second electronic device 306 are placed at the positions shown in the figure to obtain the third image, the fourth image, and the fifth image, respectively. It should be noted that in order to avoid a large position deviation of the subject in the third image, the fourth image, and the fifth image, the three electronic devices should be placed as close as possible.

[0096] S112: Preprocessing the multiple groups of initial training samples to obtain multiple groups of target training samples.

[0097] It is understandable that there are deviations between the third image, the fourth image and the fifth image obtained above in terms of imaging angle, image resolution, image zoom factor, etc. In order to ensure the accuracy of training the initial imaging model, before training the initial imaging model through training samples, the acquired initial training samples need to be preprocessed to obtain target training samples that meet the requirements.

[0098] Specifically, after obtaining multiple groups of initial training samples, the zoom factor, scene perspective and resolution of the three images in each group of initial training samples are adjusted, that is, each group of initial training samples is subjected to zoom alignment, scene perspective alignment and resolution alignment operations to obtain target training samples that meet the training requirements.

[0099] S114: Train the initial imaging model according to the multiple groups of target training samples to obtain the target imaging model.

[0100] Specifically, after determining a plurality of groups of target training samples, the initial imaging model is trained according to the determined target training samples to optimize and adjust the model parameters of the initial imaging model, thereby obtaining the target imaging model.

[0101] In the above-mentioned embodiment provided by the present application, before shooting an image, an initial imaging model is also established, and multiple groups of initial training samples are obtained. Each group of initial training samples includes a third image, a fourth image, and a fifth image, the third image is taken by the first electronic device at a first zoom factor, the fourth image is taken by the first electronic device at a second zoom factor, and the fifth image is taken by the second electronic device at a target aperture size, and the first zoom factor is smaller than the second zoom factor. On this basis, multiple groups of initial training samples are preprocessed to obtain multiple groups of target training samples, and the initial imaging model is trained according to the multiple groups of target training samples to obtain a target imaging model. In this way, based on the two images taken by the first electronic device at different zoom factors, and the image taken by the second electronic device at the target aperture size, the initial imaging model is trained to obtain the above-mentioned target imaging model, which ensures the accuracy of the trained target imaging model, thereby ensuring the naturalness of the above-mentioned target blurred image processed by the target imaging model.

[0102] In the embodiment of the present application, the above S112 may specifically include the following S112a to S112d:

[0103] S112a: Determine a target zoom scale according to the shooting focal lengths of the third image and the fifth image, and downsample the fifth image according to the target zoom scale to obtain a sixth image.

[0104] The heights of the main objects in the sixth image and the third image are consistent.

[0105] Specifically, after acquiring the third image and the fifth image, the target zoom scale is determined according to the shooting focal lengths of the third image and the fifth image, and the fifth image is downsampled according to the target zoom scale to obtain a sixth image in which the height of the main object is consistent with the height of the main object in the third image, so as to realize the zoom alignment operation of the third image and the fifth image.

[0106] S112b: Perform feature point detection on the third image, the fourth image and the sixth image, respectively, to obtain first feature information, second feature information and third feature information, respectively, and adjust the scene perspective of the third image and the fourth image to the target perspective based on the first feature information, the second feature information and the third feature information.

[0107] The target viewing angle is the scene viewing angle of the sixth image.

[0108] Specifically, after obtaining the sixth image, feature point detection is performed on the third image, the fourth image, and the sixth image, respectively, to obtain first feature information of the third image, second feature information of the fourth image, and third feature information of the sixth image. On this basis, the first feature information and the second feature information are adjusted based on the third feature information, so as to adjust the scene perspectives of the third image and the fourth image to the scene perspective of the sixth image, i.e., the target perspective.

[0109] That is, based on the feature point detection results of the third image, the fourth image and the sixth image, the scene perspective of the third image and the fourth image are rotated with the scene perspective of the sixth image as a reference to make the scenes of the third image, the fourth image and the sixth image consistent, thereby achieving the scene perspective alignment operation of the third image, the fourth image and the sixth image.

[0110] In the actual application process, the feature point detection can be performed on the third image, the fourth image and the sixth image based on the SIFT (Scale-Invariant Feature Transform) operator. For the specific algorithm of the above feature point detection, those skilled in the art can select it according to the actual situation, and no specific limitation is made here.

[0111] S112c: The third image and the fourth image after adjusting the scene viewing angle are respectively cropped according to the resolution of the sixth image to obtain a seventh image and an eighth image having the same resolution as the sixth image.

[0112] Specifically, after adjusting the scene perspective of the third image and the fourth image, the third image and the fourth image after the adjusted scene perspective are cropped respectively based on the resolution of the sixth image, so that the third image is cropped into a seventh image having a resolution consistent with that of the sixth image, and the fourth image is cropped into an eighth image having a resolution consistent with that of the sixth image, thereby achieving resolution alignment operations on the third image, the fourth image, and the sixth image.

[0113] S112d: Determine the sixth image, the seventh image, and the eighth image as a group of target training samples.

[0114] Specifically, after scaling alignment, scene perspective alignment, and resolution alignment are performed on the third image, the fourth image, and the fifth image in each group of initial training samples, a sixth image and a seventh image with the same subject but different blurring degrees are obtained, and an eighth image with the same subject but different optical magnification degrees is obtained. On this basis, the obtained sixth image, the seventh image, and the eighth image are determined as a group of target training samples, so as to train the initial imaging model through the determined target training samples.

[0115] In the above-mentioned embodiment provided by the present application, after obtaining each group of initial training samples, the target zoom scale is determined according to the shooting focal lengths of the third image and the fifth image in each group of initial training samples, and the fifth image is downsampled according to the target zoom scale to obtain a sixth image, and the height of the main object in the sixth image is consistent with that in the third image. Further, feature point detection is performed on the third image, the fourth image and the sixth image respectively, and the first feature information, the second feature information and the third feature information are obtained respectively, and the scene perspectives of the third image and the fourth image are adjusted to the target perspective according to the first feature information, the second feature information and the third feature information, and the target perspective is the scene perspective of the sixth image. On this basis, the third image and the fourth image after the scene perspective is adjusted are cropped according to the resolution of the sixth image respectively, and the seventh image and the eighth image whose resolutions are consistent with the sixth image are obtained respectively, and the sixth image, the seventh image and the eighth image are determined as a group of target training samples. In this way, before training the initial imaging model, each group of initial training samples is first subjected to scaling alignment, scene perspective alignment, and resolution alignment operations to obtain target training samples that meet the training requirements, thereby reducing the training pressure of subsequent training of the initial imaging model and ensuring the accuracy of training the initial imaging model.

[0116] In the embodiment of the present application, the above S114 may specifically include the following S114a and S114b:

[0117] S114a: Inputting the seventh image and the eighth image in any group of target training samples into the initial imaging model, processing the seventh image and the eighth image through the initial imaging model, and outputting the corresponding ninth image.

[0118] The initial imaging model is used to process the seventh image and the eighth image, and output a corresponding blurred image, namely the ninth image. That is, for the initial imaging model, the seventh image and the eighth image are its input objects, and the ninth image is its output object.

[0119] Specifically, after determining the above-mentioned target training samples, the seventh image and the eighth image in each group of target training samples are input into the above-mentioned initial imaging model as input objects, so that the initial imaging model performs segmentation and stacking, semantic information extraction, and semantic information comparison and fusion operations on the seventh image and the eighth image in sequence, thereby outputting the corresponding blurred image, namely the above-mentioned ninth image.

[0120] S114b: Compare the image information of the ninth image with that of the sixth image. If the image information of the ninth image is inconsistent with that of the sixth image, adjust the model parameters of the initial imaging model, and continue to process the seventh image and the eighth image in the next group of target training samples using the adjusted initial imaging model until the image information of the sixth image in the target training sample is consistent with the image information of the corresponding output ninth image, and determine the initial imaging model as the target imaging model.

[0121] The loss function of the initial imaging model can be specifically expressed by the following formula (2):

[0122]

[0123] Among them, I 1x ,I 2x Represents the two images input to the initial imaging model, I gt represents the target output image of the initial imaging model, f 1x and f 2x Respectively represent the input image I 1x ,I 2x The result of segmentation and stacking, f p Indicates f 1x and f 2x The result of the alignment and rearrangement, G 1x and G 2x Respectively represent the I after segmentation and stacking 1x ,I 2x Convolutional network for semantic information extraction, G p Indicates f p Convolutional network for fusion regression. θ 1x ,θ 2x ,θ p Represent the convolutional network G respectively 1x , G 2x and G p The parameters that need to be learned in 1x ,θ 2x ,θ p represents the model parameters of the initial imaging model, i represents the i-th group of training samples, and N represents the total number of training samples.

[0124] Specifically, after the seventh image and the eighth image in any group of target training samples are input into the initial imaging model to obtain the corresponding ninth image, the sixth image in the target training sample is used as the standard image to compare the image information of the ninth image with that of the sixth image. On this basis, if the image information of the ninth image is inconsistent with that of the sixth image, the model parameters of the initial imaging model are adjusted, and the seventh image and the eighth image of the next training sample are continuously processed through the adjusted initial imaging model until the image information of the sixth image in the target training sample is consistent with that of the corresponding output ninth image, and the initial imaging model is determined as the target imaging model.

[0125] In the above embodiment provided by the present application, after determining the target training samples, the seventh image and the eighth image in any group of target training samples are input into the initial imaging model, the seventh image and the eighth image are processed by the initial imaging model, and the corresponding ninth image is output. On this basis, the image information of the ninth image is compared with the sixth image. When the image information of the ninth image is inconsistent with that of the sixth image, the model parameters of the initial imaging model are adjusted, and the seventh image and the eighth image in the next group of target training samples are continuously processed by the adjusted initial imaging model until the image information of the sixth image in the target training sample is consistent with that of the corresponding output ninth image, and the initial imaging model is determined as the target imaging model. In this way, based on the two images captured by the algorithm focusing and the one image captured by the aperture focusing, the initial imaging model is trained to obtain the above target imaging model, which ensures the accuracy of the trained target imaging model, thereby ensuring the naturalness of the above target blurred image processed by the target imaging model.

[0126] The shooting method provided in the embodiment of the first aspect of the present application can be executed by a shooting device. In the embodiment of the present application, the shooting device provided in the embodiment of the second aspect of the present application is described by taking the shooting device executing the above shooting method as an example.

[0127] like Figure 4 As shown, an embodiment of the present application provides a photographing device 400, which may include the following photographing unit 402, a first processing unit 404, a second processing unit 406 and a third processing unit 408.

[0128] A photographing unit 402, configured to photograph a second image according to image information of the first image;

[0129] A first processing unit 404, configured to generate a first blurred image according to the first image, the depth information of the first image and the depth information of the second image;

[0130] A second processing unit 406, configured to generate a second blurred image according to the target imaging model, the first image and the second image;

[0131] The third processing unit 408 is configured to fuse the first blurred image and the second blurred image to obtain a target blurred image.

[0132] Through the shooting device provided in the embodiment of the present application, in the process of shooting a blurred image, a second image is shot according to the image information of the first image shot. On this basis, a first blurred image is generated according to the first image, the depth information of the first image and the depth information of the second image, and at the same time, a second blurred image is generated according to the target imaging model, the first image and the second image. Further, the first blurred image and the second blurred image are fused to obtain a target blurred image. That is, through the above-mentioned shooting device, in the process of shooting a blurred image, a first blurred image is generated based on the depth information of the first image and the second image shot, and a second virtual image is generated according to the first image, the second image and the target imaging model, and then the first virtual image and the second virtual image are fused to obtain a target blurred image with a blurred effect. Among them, the second image is shot based on the image information of the first image. In this way, the shooting of the second image is guided by the first image, and then the two blurred images determined by the first image and the second image are fused to obtain a target blurred image with a blurred effect, thereby improving the smoothness of the connection between the background part and the main part in the target blurred image, thereby ensuring the naturalness of the imaging effect of the target blurred image.

[0133] In an embodiment of the present application, the shooting unit 402 is specifically used to: identify a first image, determine a target object in the first image, where the target object is located in a focus area in the first image; extract target object information in the first image; compare the target object information with scene information in a shooting preview interface, and determine a target area in the shooting preview interface based on the comparison result; and use the target area as the focus area of ​​the shooting preview interface to capture a second image.

[0134] In the above-mentioned embodiment provided by the present application, when shooting the second image according to the image information of the first image, specifically, the first image is identified, the target object in the focus area of ​​the first image is determined, and the target object information in the first image is extracted. On this basis, the target object information is compared with the scene information in the shooting preview interface, and then the target area in the shooting preview interface is determined according to the comparison result, and the target area is used as the focus area of ​​the shooting preview interface to shoot the second image. In this way, the shooting of the second image is guided based on the first image obtained, so that the focus object, that is, the main object, of the second image obtained is the same as the above-mentioned first image, thereby ensuring the accuracy of the subsequent determination of the first blurred image and the second blurred image based on the first image and the second image, thereby ensuring the imaging effect of the target blurred image.

[0135] In an embodiment of the present application, the second processing unit 406 is specifically used to: input the first image and the second image into a target imaging model; divide the first image into M first blocks and the second image into N second blocks through the target imaging model, wherein the ratio of M to N is related to the zoom factor of the first image and the zoom factor of the second image; extract the image information of the M first blocks according to the first convolutional network through the target imaging model to obtain the first image information, and extract the image information of the N second blocks according to the second convolutional network to obtain the second image information; fuse the first image information and the second image information through the target imaging model to obtain fused image information; process the fused image information according to the target convolutional network through the target imaging model to generate a second blurred image.

[0136] In the above-mentioned embodiment provided by the present application, when generating the second blurred image, the first image and the second image are input into the target imaging model, and the first image and the second image are processed by the target imaging model to output the second blurred image. Specifically, after the first image and the second image are input into the target imaging model, the first image is divided into M first blocks and the second image is divided into N second blocks based on the ratio between the zoom factor of the first image and the zoom factor of the second image by the target imaging model. On this basis, the target imaging model extracts the image information of the M first blocks through the first convolutional network therein to obtain the first image information, and extracts the image information of the N second blocks through the second convolutional network therein to obtain the second image information. Furthermore, the target imaging model performs a fusion process on the extracted first image information and the second image information to obtain fused image information, and processes the fused image information according to the target convolutional network therein to obtain the second blurred image. In this way, based on the depth semantic information of the first image and the second image, the first image and the second image are sequentially segmented and stacked, semantic information is extracted, and semantic information is compared and fused through the target imaging model, and the above-mentioned second blurred image is output, thereby ensuring the smoothness of the blurring transition of the second blurred image, thereby ensuring the naturalness of the effect of the subsequent target blurred image.

[0137] In an embodiment of the present application, the third processing unit 408 is specifically used to: fuse the first blurred image and the second blurred image according to the first weight and the second weight to obtain a target blurred image; wherein the first weight corresponds to the first blurred image, and the second weight corresponds to the second virtual image.

[0138] In the above-mentioned embodiment provided by the present application, when determining the target blurred image, specifically, a first weight is assigned to the first blurred image, and a second weight is assigned to the second blurred image, and then the first blurred image and the second blurred image are fused according to the corresponding weights to obtain the target blurred image. In this way, according to a certain weight ratio, the first blurred image determined according to the depth information of the first image and the second image, and the second blurred image determined according to the depth semantic information of the first image and the second image are fused to obtain a target blurred image with a blurred effect, which improves the smoothness of the connection between the background part and the main part in the target blurred image, thereby ensuring the naturalness of the imaging effect of the target blurred image.

[0139] In an embodiment of the present application, the shooting device 400 also includes a fourth processing unit 410, which is specifically used to: establish an initial imaging model and obtain multiple groups of initial training samples; preprocess the multiple groups of initial training samples to obtain multiple groups of target training samples; train the initial imaging model according to the multiple groups of target training samples to obtain a target imaging model; wherein each group of initial training samples includes a third image, a fourth image and a fifth image, the third image is taken by the first electronic device at a first zoom factor, the fourth image is taken by the first electronic device at a second zoom factor, and the fifth image is taken by the second electronic device at a target aperture size, and the first zoom factor is less than the second zoom factor.

[0140] In the above-mentioned embodiment provided by the present application, before shooting an image, an initial imaging model is also established, and multiple groups of initial training samples are obtained. Each group of initial training samples includes a third image, a fourth image, and a fifth image, the third image is taken by the first electronic device at a first zoom factor, the fourth image is taken by the first electronic device at a second zoom factor, and the fifth image is taken by the second electronic device at a target aperture size, and the first zoom factor is smaller than the second zoom factor. On this basis, multiple groups of initial training samples are preprocessed to obtain multiple groups of target training samples, and the initial imaging model is trained according to the multiple groups of target training samples to obtain a target imaging model. In this way, based on the two images taken by the first electronic device at different zoom factors, and the image taken by the second electronic device at the target aperture size, the initial imaging model is trained to obtain the above-mentioned target imaging model, which ensures the accuracy of the trained target imaging model, thereby ensuring the naturalness of the above-mentioned target blurred image processed by the target imaging model.

[0141] In an embodiment of the present application, the fourth processing unit 410 is specifically used to: determine a target zoom scale according to the shooting focal lengths of the third image and the fifth image, downsample the fifth image according to the target zoom scale to obtain a sixth image, and the height of the main object in the sixth image is consistent with that in the third image; perform feature point detection on the third image, the fourth image, and the sixth image, respectively, to obtain first feature information, second feature information, and third feature information, respectively, and adjust the scene perspectives of the third image and the fourth image to the target perspective according to the first feature information, the second feature information, and the third feature information, and the target perspective is the scene perspective of the sixth image; crop the third image and the fourth image after adjusting the scene perspective according to the resolution of the sixth image, respectively, to obtain a seventh image and an eighth image whose resolutions are consistent with the sixth image, respectively; and determine the sixth image, the seventh image, and the eighth image as a group of target training samples.

[0142] In the above-mentioned embodiment provided by the present application, after obtaining each group of initial training samples, the target zoom scale is determined according to the shooting focal lengths of the third image and the fifth image in each group of initial training samples, and the fifth image is downsampled according to the target zoom scale to obtain a sixth image, and the height of the main object in the sixth image is consistent with that in the third image. Further, feature point detection is performed on the third image, the fourth image and the sixth image respectively, and the first feature information, the second feature information and the third feature information are obtained respectively, and the scene perspectives of the third image and the fourth image are adjusted to the target perspective according to the first feature information, the second feature information and the third feature information, and the target perspective is the scene perspective of the sixth image. On this basis, the third image and the fourth image after the scene perspective is adjusted are cropped according to the resolution of the sixth image respectively, and the seventh image and the eighth image whose resolutions are consistent with the sixth image are obtained respectively, and the sixth image, the seventh image and the eighth image are determined as a group of target training samples. In this way, before training the initial imaging model, each group of initial training samples is first subjected to scaling alignment, scene perspective alignment, and resolution alignment operations to obtain target training samples that meet the training requirements, thereby reducing the training pressure of subsequent training of the initial imaging model and ensuring the accuracy of training the initial imaging model.

[0143] In an embodiment of the present application, the fourth processing unit 410 is specifically used to: input the seventh image and the eighth image in any group of target training samples into the initial imaging model, process the seventh image and the eighth image through the initial imaging model, and output the corresponding ninth image; compare the image information of the ninth image with that of the sixth image, and when the image information of the ninth image is inconsistent with that of the sixth image, adjust the model parameters of the initial imaging model, and continue to process the seventh image and the eighth image in the next group of target training samples through the adjusted initial imaging model until the image information of the sixth image in the target training sample is consistent with the image information of the corresponding output ninth image, and determine the initial imaging model as the target imaging model.

[0144] In the above embodiment provided by the present application, after determining the target training samples, the seventh image and the eighth image in any group of target training samples are input into the initial imaging model, the seventh image and the eighth image are processed by the initial imaging model, and the corresponding ninth image is output. On this basis, the image information of the ninth image is compared with the sixth image. When the image information of the ninth image is inconsistent with that of the sixth image, the model parameters of the initial imaging model are adjusted, and the seventh image and the eighth image in the next group of target training samples are continuously processed by the adjusted initial imaging model until the image information of the sixth image in the target training sample is consistent with that of the corresponding output ninth image, and the initial imaging model is determined as the target imaging model. In this way, based on the two images captured by the algorithm focusing and the one image captured by the aperture focusing, the initial imaging model is trained to obtain the above target imaging model, which ensures the accuracy of the trained target imaging model, thereby ensuring the naturalness of the above target blurred image processed by the target imaging model.

[0145] The shooting device 400 in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0146] The photographing device 400 in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0147] The shooting device 400 provided in the second aspect of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0148] Alternatively, if Figure 5 As shown, an embodiment of the present application further provides an electronic device 500, including a processor 502 and a memory 504, wherein the memory 504 stores programs or instructions that can be executed on the processor 502, and when the program or instructions are executed by the processor 502, the various steps of the shooting method embodiment of the first aspect mentioned above are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described again here.

[0149] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0150] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0151] The electronic device 600 includes but is not limited to: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, a processor 610 and other components.

[0152] Those skilled in the art will appreciate that the electronic device 600 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 610 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0153] The electronic device 600 of the embodiment of the present application can be used to implement the various steps of the embodiment of the shooting method of the first aspect mentioned above.

[0154] The input unit 604 is used to capture a second image according to the image information of the first image;

[0155] Processor 610, configured to generate a first blurred image according to the first image, the depth information of the first image, and the depth information of the second image;

[0156] The processor 610 is further configured to generate a second blurred image according to the target imaging model, the first image, and the second image;

[0157] The processor 610 is further configured to fuse the first blurred image and the second blurred image to obtain a target blurred image.

[0158] In an embodiment of the present application, in the process of shooting a blurred image, a second image is shot according to the image information of the first image. On this basis, a first blurred image is generated according to the first image, the depth information of the first image and the depth information of the second image, and at the same time, a second blurred image is generated according to the target imaging model, the first image and the second image. Further, the first blurred image and the second blurred image are fused to obtain a target blurred image. That is, through the above-mentioned shooting device, in the process of shooting a blurred image, a first blurred image is generated based on the depth information of the first image and the second image, and a second virtual image is generated according to the first image, the second image and the target imaging model, and then the obtained first virtual image and the second virtual image are fused to obtain a target blurred image with a blurred effect. Among them, the second image is shot based on the image information of the first image. In this way, the shooting of the second image is guided based on the first image, and then the two blurred images determined by the first image and the second image are fused to obtain a target blurred image with a blurred effect, which improves the smoothness of the connection between the background part and the main part in the target blurred image, thereby ensuring the naturalness of the imaging effect of the target blurred image.

[0159] In an embodiment of the present application, the input unit 604 is specifically used to: identify a first image, determine a target object in the first image, where the target object is located in a focus area in the first image; extract target object information in the first image; compare the target object information with scene information in a shooting preview interface, and determine a target area in the shooting preview interface based on the comparison result; and use the target area as the focus area of ​​the shooting preview interface to capture a second image.

[0160] In the above-mentioned embodiment provided by the present application, when shooting the second image according to the image information of the first image, specifically, the first image is identified, the target object in the focus area of ​​the first image is determined, and the target object information in the first image is extracted. On this basis, the target object information is compared with the scene information in the shooting preview interface, and then the target area in the shooting preview interface is determined according to the comparison result, and the target area is used as the focus area of ​​the shooting preview interface to shoot the second image. In this way, the shooting of the second image is guided based on the first image obtained, so that the focus object, that is, the main object, of the second image obtained is the same as the above-mentioned first image, thereby ensuring the accuracy of the subsequent determination of the first blurred image and the second blurred image based on the first image and the second image, thereby ensuring the imaging effect of the target blurred image.

[0161] In an embodiment of the present application, the processor 610 is specifically used to: input the first image and the second image into a target imaging model; divide the first image into M first blocks and the second image into N second blocks through the target imaging model, wherein the ratio of M to N is related to the zoom factor of the first image and the zoom factor of the second image; extract the image information of the M first blocks according to the first convolutional network through the target imaging model to obtain the first image information, and extract the image information of the N second blocks according to the second convolutional network to obtain the second image information; fuse the first image information and the second image information through the target imaging model to obtain fused image information; process the fused image information according to the target convolutional network through the target imaging model, and output a second blurred image.

[0162] In the above-mentioned embodiment provided by the present application, when generating the second blurred image, the first image and the second image are input into the target imaging model, and the first image and the second image are processed by the target imaging model to output the second blurred image. Specifically, after the first image and the second image are input into the target imaging model, the first image is divided into M first blocks and the second image is divided into N second blocks based on the ratio between the zoom factor of the first image and the zoom factor of the second image by the target imaging model. On this basis, the target imaging model extracts the image information of the M first blocks through the first convolutional network therein to obtain the first image information, and extracts the image information of the N second blocks through the second convolutional network therein to obtain the second image information. Furthermore, the target imaging model performs a fusion process on the extracted first image information and the second image information to obtain fused image information, and processes the fused image information according to the target convolutional network therein to obtain the second blurred image. In this way, based on the depth semantic information of the first image and the second image, the first image and the second image are sequentially segmented and stacked, semantic information is extracted, and semantic information is compared and fused through the target imaging model, and the above-mentioned second blurred image is output, thereby ensuring the smoothness of the blurring transition of the second blurred image, thereby ensuring the naturalness of the effect of the subsequent target blurred image.

[0163] In an embodiment of the present application, the processor 610 is specifically used to: fuse the first blurred image and the second blurred image according to the first weight and the second weight to obtain a target blurred image; wherein the first weight corresponds to the first blurred image, and the second weight corresponds to the second virtual image.

[0164] In the above-mentioned embodiment provided by the present application, when determining the target blurred image, specifically, a first weight is assigned to the first blurred image, and a second weight is assigned to the second blurred image, and then the first blurred image and the second blurred image are fused according to the corresponding weights to obtain the target blurred image. In this way, according to a certain weight ratio, the first blurred image determined according to the depth information of the first image and the second image, and the second blurred image determined according to the depth semantic information of the first image and the second image are fused to obtain a target blurred image with a blurred effect, which improves the smoothness of the connection between the background part and the main part in the target blurred image, thereby ensuring the naturalness of the imaging effect of the target blurred image.

[0165] In an embodiment of the present application, the processor 610 is specifically used to: establish an initial imaging model and obtain multiple groups of initial training samples; preprocess the multiple groups of initial training samples to obtain multiple groups of target training samples; train the initial imaging model according to the multiple groups of target training samples to obtain a target imaging model; wherein each group of initial training samples includes a third image, a fourth image and a fifth image, the third image is taken by the first electronic device at a first zoom factor, the fourth image is taken by the first electronic device at a second zoom factor, and the fifth image is taken by the second electronic device at a target aperture size, and the first zoom factor is less than the second zoom factor.

[0166] In the above-mentioned embodiment provided by the present application, before shooting an image, an initial imaging model is also established, and multiple groups of initial training samples are obtained. Each group of initial training samples includes a third image, a fourth image, and a fifth image, the third image is taken by the first electronic device at a first zoom factor, the fourth image is taken by the first electronic device at a second zoom factor, and the fifth image is taken by the second electronic device at a target aperture size, and the first zoom factor is smaller than the second zoom factor. On this basis, multiple groups of initial training samples are preprocessed to obtain multiple groups of target training samples, and the initial imaging model is trained according to the multiple groups of target training samples to obtain a target imaging model. In this way, based on the two images taken by the first electronic device at different zoom factors, and the image taken by the second electronic device at the target aperture size, the initial imaging model is trained to obtain the above-mentioned target imaging model, which ensures the accuracy of the trained target imaging model, thereby ensuring the naturalness of the above-mentioned target blurred image processed by the target imaging model.

[0167] In an embodiment of the present application, the processor 610 is specifically used to: determine a target zoom scale according to the shooting focal lengths of the third image and the fifth image, downsample the fifth image according to the target zoom scale to obtain a sixth image, and the height of the main object in the sixth image is consistent with that in the third image; perform feature point detection on the third image, the fourth image, and the sixth image, respectively, to obtain first feature information, second feature information, and third feature information, respectively, and adjust the scene perspectives of the third image and the fourth image to the target perspective according to the first feature information, the second feature information, and the third feature information, and the target perspective is the scene perspective of the sixth image; crop the third image and the fourth image after adjusting the scene perspective according to the resolution of the sixth image, respectively, to obtain a seventh image and an eighth image whose resolutions are consistent with the sixth image, respectively; and determine the sixth image, the seventh image, and the eighth image as a group of target training samples.

[0168] In the above-mentioned embodiment provided by the present application, after obtaining each group of initial training samples, the target zoom scale is determined according to the shooting focal lengths of the third image and the fifth image in each group of initial training samples, and the fifth image is downsampled according to the target zoom scale to obtain a sixth image, and the height of the main object in the sixth image is consistent with that in the third image. Further, feature point detection is performed on the third image, the fourth image and the sixth image respectively, and the first feature information, the second feature information and the third feature information are obtained respectively, and the scene perspectives of the third image and the fourth image are adjusted to the target perspective according to the first feature information, the second feature information and the third feature information, and the target perspective is the scene perspective of the sixth image. On this basis, the third image and the fourth image after the scene perspective is adjusted are cropped according to the resolution of the sixth image respectively, and the seventh image and the eighth image whose resolutions are consistent with the sixth image are obtained respectively, and the sixth image, the seventh image and the eighth image are determined as a group of target training samples. In this way, before training the initial imaging model, each group of initial training samples is first subjected to scaling alignment, scene perspective alignment, and resolution alignment operations to obtain target training samples that meet the training requirements, thereby reducing the training pressure of subsequent training of the initial imaging model and ensuring the accuracy of training the initial imaging model.

[0169] In an embodiment of the present application, the processor 610 is specifically used to: input the seventh image and the eighth image in any group of target training samples into the initial imaging model, process the seventh image and the eighth image through the initial imaging model, and output the corresponding ninth image; compare the image information of the ninth image with that of the sixth image, and when the image information of the ninth image is inconsistent with that of the sixth image, adjust the model parameters of the initial imaging model, and continue to process the seventh image and the eighth image in the next group of target training samples through the adjusted initial imaging model, until the image information of the sixth image in the target training sample is consistent with the image information of the corresponding output ninth image, and determine the initial imaging model as the target imaging model.

[0170] In the above embodiment provided by the present application, after determining the target training samples, the seventh image and the eighth image in any group of target training samples are input into the initial imaging model, the seventh image and the eighth image are processed by the initial imaging model, and the corresponding ninth image is output. On this basis, the image information of the ninth image is compared with the sixth image. When the image information of the ninth image is inconsistent with that of the sixth image, the model parameters of the initial imaging model are adjusted, and the seventh image and the eighth image in the next group of target training samples are continuously processed by the adjusted initial imaging model until the image information of the sixth image in the target training sample is consistent with that of the corresponding output ninth image, and the initial imaging model is determined as the target imaging model. In this way, based on the two images captured by the algorithm focusing and the one image captured by the aperture focusing, the initial imaging model is trained to obtain the above target imaging model, which ensures the accuracy of the trained target imaging model, thereby ensuring the naturalness of the above target blurred image processed by the target imaging model.

[0171] It should be understood that in the embodiment of the present application, the input unit 604 may include a graphics processor (GPU) 6041 and a microphone 6042, and the graphics processor 6041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 606 may include a display panel 6061, and the display panel 6061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 607 includes a touch panel 6071 and at least one of other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 may include two parts: a touch detection device and a touch controller. Other input devices 6072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0172] The memory 609 can be used to store software programs and various data. The memory 609 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 609 may include a volatile memory or a non-volatile memory, or the memory 609 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 609 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0173] The processor 610 may include one or more processing units; optionally, the processor 610 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 610.

[0174] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned first aspect shooting method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0175] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0176] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned first aspect of the shooting method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0177] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0178] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the shooting method embodiment of the first aspect as described above, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0179] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0180] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0181] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A shooting method, characterized in that: The shooting method comprises: capturing a second image based on image information of the first image; generating a first blurred image according to the first image, the depth information of the first image, and the depth information of the second image; generating a second blurred image according to the target imaging model, the first image and the second image; fusing the first blurred image and the second blurred image to obtain a target blurred image; The step of generating a second blurred image according to the target imaging model, the first image and the second image comprises: inputting the first image and the second image into the target imaging model; Dividing the first image into M first image blocks and dividing the second image into N second image blocks by using the target imaging model, wherein a ratio of M to N is related to a zoom factor of the first image and a zoom factor of the second image; Extracting image information of the M first image blocks according to a first convolutional network through the target imaging model to obtain first image information, and extracting image information of the N second image blocks according to a second convolutional network to obtain second image information; fusing the first image information and the second image information using the target imaging model to obtain fused image information; The second blurred image is generated by processing the fused image information according to the target convolutional network through the target imaging model.

2. The shooting method according to claim 1, characterized in that: The step of capturing a second image according to image information of the first image comprises: Identify a first image and determine a target object in the first image, wherein the target object is located in a focus area in the first image; extracting target object information in the first image; Comparing the target object information with the scene information in the shooting preview interface, and determining the target area in the shooting preview interface according to the comparison result; The target area is used as the focus area of ​​the shooting preview interface to capture the second image.

3. The shooting method according to claim 1 or 2, characterized in that: The shooting method further includes: Establish an initial imaging model and obtain multiple groups of initial training samples; Preprocessing the multiple groups of initial training samples to obtain multiple groups of target training samples; Training the initial imaging model according to the multiple groups of target training samples to obtain the target imaging model; Among them, each group of initial training samples includes a third image, a fourth image and a fifth image, the third image is taken by the first electronic device at a first zoom factor, the fourth image is taken by the first electronic device at a second zoom factor, and the fifth image is taken by the second electronic device at a target aperture size, and the first zoom factor is smaller than the second zoom factor.

4. The shooting method according to claim 3, characterized in that: The preprocessing of the multiple groups of initial training samples to obtain multiple groups of target training samples includes: determining a target zoom scale according to the shooting focal lengths of the third image and the fifth image, and downsampling the fifth image according to the target zoom scale to obtain a sixth image, wherein the height of the main object in the sixth image is consistent with that in the third image; Performing feature point detection on the third image, the fourth image, and the sixth image, respectively, to obtain first feature information, second feature information, and third feature information, respectively, and adjusting the scene perspectives of the third image and the fourth image to target perspectives according to the first feature information, the second feature information, and the third feature information, where the target perspective is the scene perspective of the sixth image; The third image and the fourth image after adjusting the scene perspective are respectively cropped according to the resolution of the sixth image to obtain a seventh image and an eighth image having the same resolution as the sixth image; The sixth image, the seventh image and the eighth image are determined as a group of target training samples.

5. A photographing device, characterized in that: The photographing device comprises: a photographing unit, configured to photograph a second image according to image information of the first image; A first processing unit, configured to generate a first blurred image according to the first image, the depth information of the first image, and the depth information of the second image; A second processing unit, configured to generate a second blurred image according to the target imaging model, the first image and the second image; a third processing unit, configured to fuse the first blurred image and the second blurred image to obtain a target blurred image; The second processing unit is specifically used for: inputting the first image and the second image into the target imaging model; Dividing the first image into M first image blocks and dividing the second image into N second image blocks by using the target imaging model, wherein a ratio of M to N is related to a zoom factor of the first image and a zoom factor of the second image; Extracting image information of the M first image blocks according to a first convolutional network through the target imaging model to obtain first image information, and extracting image information of the N second image blocks according to a second convolutional network to obtain second image information; fusing the first image information and the second image information using the target imaging model to obtain fused image information; The second blurred image is generated by processing the fused image information according to the target convolutional network through the target imaging model.

6. The photographing device according to claim 5, characterized in that: The shooting unit is specifically used for: Identify a first image and determine a target object in the first image, wherein the target object is located in a focus area in the first image; extracting target object information in the first image; Comparing the target object information with the scene information in the shooting preview interface, and determining the target area in the shooting preview interface according to the comparison result; The target area is used as the focus area of ​​the shooting preview interface to capture the second image.

7. The photographing device according to claim 5 or 6, characterized in that: The photographing device further includes a fourth processing unit, and the fourth processing unit is specifically configured to: Establish an initial imaging model and obtain multiple groups of initial training samples; Preprocessing the multiple groups of initial training samples to obtain multiple groups of target training samples; Training the initial imaging model according to the multiple groups of target training samples to obtain the target imaging model; Among them, each group of initial training samples includes a third image, a fourth image and a fifth image, the third image is taken by the first electronic device at a first zoom factor, the fourth image is taken by the first electronic device at a second zoom factor, and the fifth image is taken by the second electronic device at a target aperture size, and the first zoom factor is smaller than the second zoom factor.

8. The photographing device according to claim 7, characterized in that: The fourth processing unit is specifically configured to: determining a target zoom scale according to the shooting focal lengths of the third image and the fifth image, and downsampling the fifth image according to the target zoom scale to obtain a sixth image, wherein the height of the main object in the sixth image is consistent with that in the third image; Performing feature point detection on the third image, the fourth image, and the sixth image, respectively, to obtain first feature information, second feature information, and third feature information, respectively, and adjusting the scene perspectives of the third image and the fourth image to target perspectives according to the first feature information, the second feature information, and the third feature information, where the target perspective is the scene perspective of the sixth image; The third image and the fourth image after adjusting the scene perspective are respectively cropped according to the resolution of the sixth image to obtain a seventh image and an eighth image having the same resolution as the sixth image; The sixth image, the seventh image and the eighth image are determined as a group of target training samples.

Images