Photographing method and device and electronic equipment

Abstract

The invention discloses a photographing method and device and electronic equipment, and belongs to the technical field of electronic equipment. The method comprises the following steps: receiving a first input of a user under the condition of displaying a preview image; in response to the first input, the preview image is imaged to obtain a first image, and the first image comprises a first object and M second objects; video clips within a first time period after the first input is received are collected, N frames of second images are obtained, and both M and N are positive integers; extracting a first pixel position set of M second objects in the first image and background pixels at positions corresponding to the M second objects in the first image in the N frames of second images; and performing pixel filling on the first pixel position set by using background pixels at positions corresponding to the M second objects in the first image in the N frames of second images to obtain a target image.

Description

Technical Field

[0001] The present application belongs to the technical field of electronic equipment, and specifically relates to a photographing method, device and electronic equipment. Background Art

[0002] With the rise of smart photography technology, various cutout solutions have emerged. People can use electronic devices to take pictures in daily life or work, and use the smart photo editing function in the electronic devices to remove interfering objects in the picture with one click, and complete the background of the area where the interfering objects are located in the picture.

[0003] Most of the existing intelligent modification solutions simply analyze the symmetry and sequence of the image background, and then use adjacent pixels or symmetrical pixels to fill in the foreground blocked by the interfering object.

[0004] The above scheme has a good effect on retouching pictures with simple backgrounds, such as on the road, in front of regular buildings, etc. However, when the background is complex, after the above scheme eliminates the interference objects and completes the background of the area where the interference objects are located in the picture, the obtained pictures often show artifacts, blurs, etc., making the overall processing traces of the picture obvious and the effect is not good. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a photographing method, device and electronic device, which can improve the problems of image artifacts, blur and the like caused by existing intelligent photo editing solutions and obtain pictures with better effects.

[0006] In a first aspect, an embodiment of the present application provides a photographing method, the method comprising:

[0007] In case of displaying the preview image, receiving a first input from a user;

[0008] In response to the first input, imaging the preview image to obtain a first image, wherein the first image includes a first object and M second objects;

[0009] Capturing video clips within a first time period after receiving the first input, obtaining N frames of second images, where M and N are both positive integers;

[0010] Extracting a first pixel position set of the M second objects in the first image and background pixels at positions corresponding to the M second objects in the first image in the N frames of second images;

[0011] The first pixel position set is pixel-filled using background pixels at positions corresponding to the M second objects in the first image in the N frames of second images to obtain a target image.

[0012] In a second aspect, an embodiment of the present application provides a photographing device, the device comprising:

[0013] A first receiving module, configured to receive a first input from a user when displaying a preview image;

[0014] A first determining module is used to image the preview image in response to the first input to obtain a first image, wherein the first image includes a first object and M second objects;

[0015] A collection module, used for collecting video clips within a first time period after receiving the first input, to obtain N frames of second images, where M and N are both positive integers;

[0016] An extraction module, configured to extract a first pixel position set of the M second objects in the first image and background pixels at positions corresponding to the M second objects in the first image in the N frames of second images;

[0017] The second determination module is used to perform pixel filling on the first pixel position set using background pixels at positions corresponding to the M second objects in the first image in the N frames of second images to obtain a target image.

[0018] In a third aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the second aspect are implemented.

[0019] In a fourth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the second aspect.

[0020] In a fifth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the second aspect.

[0021] In an embodiment of the present application, when displaying a preview image of a captured scene, the preview image is imaged in response to a first input from a user to obtain a first image, in which a first object and M second objects are included, and then a video clip of the captured scene within a first time period after the first input is received is collected to obtain N frames of second images, and then the first pixel position set is pixel-filled using background pixels at positions corresponding to the M second objects in the first image in the N frames of second images of the captured scene within the first time period after the first input is received. Compared with the prior art method of filling with background pixels at positions symmetrical to the first pixel position set in the first image, the solution of the embodiment of the present application can improve the artifacts and blurring problems of the picture obtained by the prior art solution when facing a complex background, and a better image can be obtained by the solution of the embodiment of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flowchart of a photographing method provided by some embodiments of the present application;

[0023] Figure 2 is a schematic diagram of skeleton feature points of a human body provided by some embodiments of the present application;

[0024] Figure 3 is a schematic diagram of a first pixel position set provided by some embodiments of the present application;

[0025] FIG4( a) is a schematic diagram of a first image provided by some embodiments of the present application;

[0026] FIG4( b ) is a schematic diagram of a first frame of a second image provided by some embodiments of the present application;

[0027] Figure 5 is a schematic diagram of a first frame of a second image provided by some embodiments of the present application;

[0028] Figure 6 is a schematic diagram of candidate target images provided by some embodiments of the present application;

[0029] Figure 7 is a flowchart of a photographing method provided by some embodiments of the present application;

[0030] Figure 8 is a schematic diagram of the structure of a photographing device shown in some embodiments of the present application;

[0031] Fig. 9 is a schematic diagram of the structure of an electronic device shown in some embodiments of the present application;

[0032] Fig.10It is a schematic diagram of the hardware structure of an electronic device shown in some embodiments of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0034] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or N. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0035] Before introducing the technical solution of the embodiment of the present application, the background technology of the embodiment of the present application is first introduced:

[0036] As described in the background technology, in the prior art, after background completion is performed on the area where the interference object is located in the picture, the obtained picture often presents artifacts, blurring and the like. In order to solve the above problems, a photographing method, device and electronic device are provided in the embodiments of the present application. When displaying a preview image of a shooting scene, the preview image is imaged in response to a first input of a user to obtain a first image, in which a first object and M second objects are included. Then, a video clip of the shooting scene within a first time period after receiving the first input is collected to obtain N frames of second images. Then, the first pixel position set is pixel-filled by using the background pixels at positions corresponding to the M second objects in the first image in the N frames of the second images of the shooting scene within the first time period after receiving the first input. In this way, compared with the prior art that uses background pixels at positions symmetrical to the first pixel position set in the first image to fill, the solution of the embodiments of the present application can improve the artifacts and blurring problems of the picture obtained by the prior art solution when facing a complex background, and a better image can be obtained by using the solution of the embodiments of the present application.

[0037] The technical solution of the embodiment of the present application can be applied to the scenario of supplementing the background of the area where the interfering objects in the picture are located. For example, object A took a photo at a tourist attraction. In addition to object A, there are some other interfering objects in the photo. In this case, after object A eliminates other interfering objects from the photo, it is necessary to supplement the background of the area where the interfering objects are located.

[0038] The photographing method provided by the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0039] Figure 1 This is a flow chart of a photographing method provided in an embodiment of the present application. The execution subject of the method may be an electronic device, and the electronic device here may be, but is not limited to, a personal computer (PC), a smart phone, a tablet computer or a personal digital assistant (PDA). It should be noted that the above execution subject does not constitute a limitation on the embodiments of the present application.

[0040] like Figure 1 As shown, the photographing method provided in the embodiment of the present application may include steps 110 to 150.

[0041] Step 110: When the preview image is displayed, receive a first input from the user.

[0042] The preview image may be an image of the shooting scene displayed in the electronic device before the user performs a shooting input when the shooting scene is photographed by the electronic device, and the preview image may include at least one object. The shooting scene here may be a pre-set shooting scene, for example, the shooting scene may be a scene of a tourist attraction.

[0043] The first input is an input to the electronic device, and the first input is used to image the preview image to obtain the first image. The first input can be a first operation. Exemplarily, the first input includes but is not limited to: a touch input of the electronic device by a user through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible inputs, which can be determined according to actual use requirements, and the embodiments of the present invention are not limited. The specific gesture in the embodiment of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiment of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input. For example, the first input can be: a touch input of the user to the camera control in the electronic device.

[0044] Step 120 : In response to the first input, image the preview image to obtain a first image, wherein the first image includes a first object and M second objects.

[0045] The first image may be an image of a captured scene, and the first image may include at least one object.

[0046] The first object may be an object to be retained in the first image, such as a person, for example, a holder of an electronic device, or at least one object in the first image arbitrarily selected by the user according to needs.

[0047] The second object may be an object that interferes with the first object.

[0048] Continuing with the above, object A takes a photo at a tourist attraction. In addition to object A, the photo also includes object B and object C. If object A wants to process the photo and only retain itself in the photo and eliminate object B and object C from the photo, then object B and object C in the photo are equivalent to the second object.

[0049] In some embodiments of the present application, a preview image of the shooting scene may be imaged in response to a first input from a user to obtain a first image.

[0050] In some embodiments of the present application, in order to improve the accuracy of determining the first object and the second object, before step 110, the above method may further include:

[0051] receiving a second input from the user regarding the preview image;

[0052] In response to the second input, determining first position information of the second input;

[0053] After step 120, the above method may further include:

[0054] Determine second position information corresponding to at least one object according to a first skeleton coordinate set corresponding to at least one object;

[0055] A first object and M second objects are determined from the at least one object according to the distance between the first position information and the second position information respectively corresponding to the at least one object.

[0056] Among them, the second input is an input to the preview image, and the above-mentioned second input is used to determine the first position information of the second input, and the second input can be a second operation. Exemplarily, the above-mentioned second input includes but is not limited to: the user's touch input to the preview image through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible inputs, which can be determined according to actual use requirements, and the embodiment of the present invention is not limited. The specific gesture in the embodiment of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiment of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input. For example, the above-mentioned second input can be: the user's touch input to an object in the preview image.

[0057] The first position information may be position information at which the second input is performed.

[0058] For any object of the at least one object in the preview image, the first skeleton coordinate set corresponding to the object may be a set of skeleton coordinates corresponding to the object.

[0059] For any object among the at least one object in the preview image, the second position information corresponding to the object may be position information of the object in the preview image.

[0060] In some embodiments of the present application, when a user clicks on an object in a preview image, the location where the user clicks can be recorded and used as the first location information. Then, when the user performs a first input and images the preview image to obtain the first image, the second location information corresponding to each object in the first image can be determined based on the first skeleton coordinate set corresponding to each object in the first image. Then, based on the distance between the first location information and the second location information corresponding to at least one object, the first object and M second objects can be determined from at least one object.

[0061] In some embodiments of the present application, when taking a photo, generally if the user wants to highlight an object, he will click on the object in the preview image to focus on the object. Therefore, when the user clicks on an object in the preview image, the location where the user clicks can be recorded and used as the first position information, that is, the object at the first position information is generally the first object. However, the first object and the second object determined in this way are not accurate enough, and the first object and the second object need to be determined based on the distance between the subsequent first position information and the second position information. The specific determination method will be introduced in detail in the subsequent embodiments.

[0062] In some embodiments of the present application, the first image can be input into a pre-trained skeleton extraction model, so that a first skeleton coordinate set corresponding to each object in the first image can be obtained. Specifically, 33 skeleton coordinates corresponding to each object can be extracted.

[0063] refer to Figure 2 , Figure 2 This is a schematic diagram of the 33 skeleton coordinates of the human body. Figure 2 The meanings of the 33 skeleton coordinates are as follows: "0" is the skeleton coordinate of the nose, "1" is the internal skeleton coordinate of the left eye, "2" is the central skeleton coordinate of the left eye, "3" is the external skeleton coordinate of the left eye, "4" is the internal skeleton coordinate of the right eye, "5" is the central skeleton coordinate of the right eye, "6" is the external skeleton coordinate of the right eye, "7" is the left ear skeleton coordinate, "8" is the right ear skeleton coordinate, "9" is the left face skeleton coordinate, "10" is the right face skeleton coordinate, "11" is the left shoulder skeleton coordinate, "12" is the right shoulder skeleton coordinate, "13" is the left elbow skeleton coordinate, "14" is the right elbow skeleton coordinate, "15" is the left wrist skeleton coordinate, "16" is the right wrist skeleton coordinate Skeleton coordinates, "17" are the left little finger skeleton coordinates, "18" are the right little finger skeleton coordinates, "19" are the left index finger skeleton coordinates, "20" are the right index finger skeleton coordinates, "21" are the left thumb skeleton coordinates, "22" are the right thumb skeleton coordinates, "23" are the left hip skeleton coordinates, "24" are the right hip skeleton coordinates, "25" are the left knee skeleton coordinates, "26" are the right knee skeleton coordinates, "27" are the left ankle skeleton coordinates, "28" are the right ankle skeleton coordinates, "29" are the left heel skeleton coordinates, "30" are the right heel skeleton coordinates, "31" are the left index finger skeleton coordinates, and "32" are the right index finger skeleton coordinates.

[0064] In some embodiments of the present application, the skeleton coordinates located at the center of gravity of the object in the first skeleton coordinate set corresponding to each object can be selected to determine the second position information corresponding to each object, such as Figure 2 As shown, the midpoint of the line between the "23" skeleton coordinates and the "24" skeleton coordinates of the human body can be selected as the second position information of the object.

[0065] In an embodiment of the present application, by responding to the user's second input to the preview image, the first position information of the second input is determined, and then the second position information corresponding to at least one object is determined according to the first skeleton coordinate set corresponding to the at least one object. Therefore, the first object and M second objects can be determined from at least one object according to the distance between the first position information and the second position information corresponding to the at least one object. In this way, the first object and the second object can be determined according to the skeleton coordinate set of each object in the preview image, thereby improving the accuracy of determining the first object and the second object.

[0066] In some embodiments of the present application, in order to further improve the accuracy of determining the first object and the second object, determining the first object and M second objects from at least one object according to the distance between the first position information and the second position information respectively corresponding to the at least one object may specifically include:

[0067] Calculating the distance between the first position information and the second position information respectively corresponding to the at least one object to obtain the first distance respectively corresponding to the at least one object;

[0068] Determine the object corresponding to the minimum value of the first distances respectively corresponding to the at least one object as the first object;

[0069] The other objects except the first object in the at least one object are determined as M second objects.

[0070] For any object among the at least one object, the corresponding first distance may be the distance between the first position information and the second position information corresponding to the object.

[0071] In some embodiments of the present application, the distance between the first position information and the second position information corresponding to each object in the first image can be calculated to obtain the first distance corresponding to each object, and then the object corresponding to the minimum value of the first distance corresponding to each object is determined as the first object, and other objects in the first image except the first object are determined as second objects.

[0072] It should be noted that before the user executes the first input, there is a situation where the user clicks on the preview image multiple times, that is, the user selects the first object multiple times. In this case, before the user executes the first input, the user executes the second input once, and the first object and the second object are determined in the above-mentioned manner of determining the first object and the second object in the first image, that is, after the user executes the second input, the position information of the second input is obtained, and then the frame preview image is input into the pre-trained skeleton extraction model to obtain the skeleton coordinate set of all objects in the preview image, and then the position information of each object is determined based on the skeleton coordinate set of each object in the preview image, and then the distance between the position information of the second input and the position information of each object in the preview image is calculated, and the object corresponding to the minimum distance is taken as the first object, and the other objects in the preview image except the first object are taken as the second object. This is done until the user executes the first input.

[0073] In an example, taking the case where there are m+1 objects in a preview image and each object has 33 skeleton coordinates, after the user clicks on an object in the preview image, the click position information K is recorded, and then the frame of preview image is input into a pre-trained skeleton extraction model to obtain the skeleton coordinate sets U1(pt1, pt2, ..., pt33), U2(pt1, pt2, ..., pt33), ..., Um+1(pt1, pt2, ..., pt33) corresponding to the m+1 objects in the preview image, and taking the midpoint of the line connecting the 23rd skeleton coordinate point and the 24th skeleton coordinate point of each object as the position of the object to obtain the position information K1, K2, ..., Km+1 of each object in the preview image, and then calculating the distances D1, D2, ..., Dm+1 between the position information K and the position information K1, K2, ..., Km+1 of each object, and selecting the object corresponding to the minimum distance as the first object, and selecting other persons as the second object.

[0074] If the user has not executed the first input, the next frame of preview image will continue to be sent to the pre-trained skeleton model to obtain the skeleton coordinate sets U1(pt1, pt2, ..., pt33), U2(pt1, pt2, ..., pt33), ..., Um+1(pt1, pt2, ..., pt33) corresponding to the m+1 objects in the preview image respectively. Repeat the above steps to continuously update the skeleton coordinate set of each object in the preview image. If the user has not executed the first input, the above process will be looped until the user executes the first input.

[0075] After the user executes the first input, the first image is also input into the pre-trained skeleton extraction model to obtain a set of skeleton coordinates of all objects in the preview image, and then the position information of the last time the user clicked the preview image before executing the first input is obtained and used as the first position information. Then, the first object and the second object in the first image are determined in the same manner as described above.

[0076] In an embodiment of the present application, by calculating the distance between the first position information and the second position information corresponding to at least one object, the first distance corresponding to the at least one object is obtained, and then the object corresponding to the minimum value of the first distance corresponding to the at least one object is determined as the first object, and the other objects except the first object in the at least one object are determined as M second objects. In this way, the first object and the second object in the first image can be accurately determined, further improving the accuracy of determining the first object and the second object.

[0077] Step 130: Capture video clips within a first time period after receiving the first input to obtain N frames of second images.

[0078] Among them, the first time period can be a pre-set time period, such as 1 minute. The specific first time period can be set according to user needs and is not limited in the embodiments of the present application.

[0079] The above-mentioned M and N may both be positive integers.

[0080] The second image may be an image of a photographed scene photographed within a first period of time after the first input is received.

[0081] In some embodiments of the present application, after executing the first input and obtaining the first image, the time of obtaining the first image can be recorded, and then the camera device can be kept still and continue shooting to obtain a video stream of the shooting scene in the first time period, and then multiple video frames can be captured from the video stream to obtain N frames of the second image.

[0082] Step 140: extract a first pixel position set of the M second objects in the first image and background pixels at positions corresponding to the M second objects in the first image in N frames of second images.

[0083] The first pixel position set may be a set of pixel positions corresponding to the M second objects in the first image.

[0084] In some embodiments of the present application, after acquiring a first image and N frames of second images, a first pixel position set of M second objects in the first image and background pixels at positions corresponding to the M second objects in the N frames of second images can be extracted.

[0085] In some embodiments of the present application, in order to accurately obtain the first pixel position set, extracting the first pixel position set of M second objects in the first image may specifically include:

[0086] Convert the first image into a grayscale image to obtain a first grayscale image;

[0087] Segmenting the foreground of the first grayscale image to obtain a first foreground image;

[0088] Based on the first foreground image and the first skeleton coordinate sets respectively corresponding to the M second objects, a first pixel position set of the M second objects in the first image is determined.

[0089] The first grayscale image may be an image obtained by converting the first image into a grayscale image.

[0090] The first foreground image may be an image obtained by extracting the foreground of the first grayscale image.

[0091] In some embodiments of the present application, a first grayscale image is obtained by converting a first image into a grayscale image, and then the foreground in the first grayscale image is extracted to obtain a first foreground image. Specifically, a k-nearest neighbor classification algorithm (K-NearestNeighbor, KNN) can be used to perform foreground extraction on the first grayscale image to obtain the first foreground image, and then a first pixel position set of the M second objects in the first image can be obtained based on the first skeleton coordinate sets corresponding to the first foreground image and the M second objects.

[0092] In an embodiment of the present application, a first grayscale image is obtained by converting a first image into a grayscale image, and then the foreground in the first grayscale image is extracted to obtain a first foreground image. Then, based on the first skeleton coordinate sets corresponding to the first foreground image and the M second objects respectively, the first pixel position set of the M second objects in the first image can be accurately obtained.

[0093] In some embodiments of the present application, determining a first pixel position set of the M second objects in the first image based on the first skeleton coordinate sets respectively corresponding to the first foreground image and the M second objects may specifically include:

[0094] Determine foreground areas corresponding to the M second objects based on the first foreground image and the first skeleton coordinate sets respectively corresponding to the M second objects;

[0095] Masks are added to pixels in other foreground areas of the first foreground image except for the foreground areas corresponding to the M second objects, to obtain a first pixel position set of the M second objects in the first image.

[0096] In some embodiments of the present application, the first foreground image and the first skeleton coordinate sets corresponding to the M second objects can be compared to determine the foreground areas corresponding to the M second objects, and then masks are added to the pixels of the foreground areas other than the foreground areas corresponding to the M second objects in the first foreground image, so that the first pixel position set of the M second objects in the first image can be obtained.

[0097] Continuing with the above example, the first image mat_k is converted into a grayscale image to obtain a first grayscale image gray_k, and then the foreground area in the first grayscale image gray_k is extracted to obtain a first foreground area, and then the first foreground area and the first skeleton coordinate sets corresponding to the M second objects are compared to determine the foreground areas corresponding to the M second objects, and the pixels of the foreground areas other than the foreground areas corresponding to the M second objects in the first foreground image are masked. Specifically, the pixel values ​​of the pixel points of the foreground areas other than the foreground areas corresponding to the M second objects in the first foreground image are updated to 0, so as to obtain the following: Figure 3The first pixel position set 31 of the M second objects in the first image selected by the frame is obtained from Figure 3 It can be seen that the pixel values ​​of the pixels of the first pixel position set 31 selected by the frame are 255, and the pixel values ​​of the other pixels are 0, so the first pixel position set 31 can be extracted.

[0098] In an embodiment of the present application, by comparing the first foreground image and the first skeleton coordinate sets corresponding to the M second objects respectively, the foreground areas corresponding to the M second objects can be determined, and then masks are added to the pixels of the foreground areas other than the foreground areas corresponding to the M second objects in the first foreground image. In this way, the first pixel position set of the M second objects in the first image can be accurately obtained, further improving the accuracy of determining the first pixel position set.

[0099] In some embodiments of the present application, in order to accurately extract background pixels at positions corresponding to the M second objects in the first image in the N frames of second images, extracting background pixels at positions corresponding to the M second objects in the first image in the N frames of second images may specifically include:

[0100] Matching the i-th second image in the N-frame second image with the first image, and obtaining a second pixel position set in the i-th second image that matches the first pixel position set;

[0101] The background pixels at the second pixel position set in the i-th frame second image are determined as the background pixels at the positions in the i-th frame second image corresponding to the M second objects in the first image.

[0102] The i-th second image frame may be any second image frame among N second images, where i may be a positive integer, and 1≤i≤N.

[0103] The second pixel position set may be a set of pixel positions in the i-th frame of the second image that matches the first pixel position set.

[0104] In some embodiments of the present application, the i-th second image in N frames of second images can be matched with the first image to obtain a second pixel position set in the i-th second image that matches the first pixel position set, and then the background pixels at the second pixel position set are determined as the background pixels at the positions in the i-th second image corresponding to the M second objects in the first image.

[0105] Continuing with the above example, object A takes a photo Q at a certain tourist attraction. In addition to object A, the photo Q also includes object B and object C. Taking object A as the first object and object B and object C as the second object, after taking photo Q, the camera is kept still and a 1-minute video clip is taken. There are 10 frames of second images in the video clip. During the process of shooting these 10 frames of second images, object B leaves and then object D joins the shooting scene. Therefore, in addition to object A, there are also object C and object D in one or several frames of the 10 frames of second images. For example, in addition to object A, there are also object C and object D in the first frame of the 10 frames of second images.

[0106] Referring to Figures 4(a) and 4(b), Figure 4(a) is a schematic diagram of a first image, in which there are objects A, B and C, and the first pixel position set is an area including objects B and C, i.e., area 41 in Figure 4(a). Figure 4(b) is a schematic diagram of the i-th second image in N frames of second images. Taking the first frame of the second image in 10 frames of second images as an example, in addition to object A, the first frame of the second image also includes objects C and D. In the first frame of the second image, the first pixel position set (i.e., area 41) in the first image is matched with area 42 in Figure 4(b), that is, the background pixels of area 42 are the background pixels in the first frame of the second image corresponding to area 41 in the first image.

[0107] It should be noted that, for each of the N frames of second images, the background pixels at the positions corresponding to the M second objects in the first image in each frame of the second image can be determined according to the method of determining the background pixels at the positions corresponding to the M second objects in the first image in the i-th frame of the second image.

[0108] In an embodiment of the present application, by matching the i-th second image in N frames of second images with the first image, a second pixel position set in the i-th second image that matches the first pixel position set is obtained, so that the background pixels at the second pixel position set can be directly determined as the background pixels at positions in the i-th second image corresponding to the M second objects in the first image. In this way, the background pixels in the i-th second image at positions corresponding to the M second objects in the first image are accurately determined, thereby improving the accuracy of determining the background pixels in the i-th second image at positions corresponding to the M second objects in the first image.

[0109] Step 150 , fill the first pixel position set with pixels using background pixels at positions corresponding to the M second objects in the first image in the N frames of second images to obtain a target image.

[0110] The target image may be an image obtained by pixel-filling the first pixel position set with background pixels at positions corresponding to the M second objects in the first image in the N frames of second images.

[0111] In some embodiments of the present application, in order to improve the artifact and blur problem of the image obtained by the prior art solution, step 150 may specifically include:

[0112] Extracting a set of third pixel positions of P third objects in the i-th frame of the second image;

[0113] Calculate the intersection of the first pixel position set and the third pixel position set to obtain a fourth pixel position set;

[0114] The background pixels of the first target pixel position set in the third pixel position set are filled into the second target pixel position set in the first pixel position set to obtain a target image.

[0115] The third object may be other objects in the second image of the i-th frame except the first object. The number of the third objects may be P, where P is a positive integer.

[0116] The P third objects and the M second objects may not be completely the same. This is because during the photo shooting process, some objects may leave and other objects may enter the shooting scene, so the P third objects and the M second objects may not be completely the same.

[0117] The third pixel position set may be a set of pixel positions corresponding to P third objects in the i-th frame image.

[0118] The fourth pixel position set may be a pixel position set corresponding to the intersection object in the third pixel position set obtained according to the intersection object between the P third objects and the M second objects.

[0119] In some embodiments of the present application, the intersection object between the P third objects and the M second objects may be calculated first, and then the pixel position set corresponding to the intersection object in the third pixel position set is used as the fourth pixel position set.

[0120] Continuing with the above example, refer to Figure 5 , the third pixel position set corresponding to the object C and the object D in the second image of the first frame is area 51, and then calculate the area 41 and Figure 5 The intersection between region 51 in Figure 4(a) and region 41 Figure 5 The intersection of the regions 51 is the pixel position set corresponding to the object C, that is, Figure 5 Area 52 in.

[0121] The first target pixel position set may be a pixel position set in the third pixel position set except the fourth pixel position set.

[0122] Continue to refer Figure 5 , the pixel position set in area 51 except area 52 is determined as the first target pixel position set, that is, Figure 5 The pixel position set corresponding to the object D in is the first target pixel position set, that is, Figure 5 The area 53 in is the first target pixel position set.

[0123] The second target pixel position set may be a pixel position set in the first pixel position set that matches the first target pixel position set.

[0124] Continuing to refer to Figure 4(a) and Figure 5 , in Figure 4(a), the location of object B is Figure 5 Therefore, the position of object B in FIG4(a) is the second target pixel position set, that is, area 43 in FIG4(a).

[0125] In some embodiments of the present application, the target image can be obtained by extracting the third pixel position set of P third objects in the second image of the i-th frame, calculating the intersection of the first pixel position set and the third pixel position set, obtaining the fourth pixel position set, and then filling the background pixels of the first target pixel position set in the third pixel position set into the second target pixel position set in the first pixel position set.

[0126] Continuing to refer to Figure 4(a) and Figure 5 , you can Figure 5 The background pixels in the middle area 53 are filled into the area 43 of FIG. 4( a ), so that the background of the area where the object B in FIG. 4( a ) is located can be filled.

[0127] In an embodiment of the present application, compared with the prior art method of filling with background pixels at positions symmetrical to the first pixel position set in the first image, the embodiment of the present application extracts the third pixel position set of P third objects in the second image of the i-th frame, and then calculates the intersection of the first pixel position set and the third pixel position set to obtain a fourth pixel position set, and fills the background pixels of the first target pixel position set in the third pixel position set into the second target pixel position set in the first pixel position set to obtain a target image. This can improve the artifacts and blurring problems of the picture obtained by the prior art solution and obtain an image with better effect.

[0128] In some embodiments of the present application, after the second target pixel position set in the first image is filled with the background using the second image of the i-th frame, the obtained image may still have other backgrounds of the second object that have not been filled, so the obtained image is not the final image, and therefore the obtained image needs to be further processed. Specifically, the processing can be performed in the following manner:

[0129] In some embodiments of the present application, in order to further improve the artifact and blur problem of the picture obtained by the prior art solution, the background pixels of the first target pixel position set in the third pixel position set are filled into the second target pixel position set in the first pixel position set to obtain the target image, which may specifically include:

[0130] Filling the background pixels of the first target pixel position set in the third pixel position set into the second target pixel position set in the first pixel position set to obtain a candidate target image;

[0131] Updating the first pixel position set to a pixel position set in the first pixel position set excluding the second target pixel position set;

[0132] When it is determined that there is a second image among the N frames of second images that does not match the first image, and the updated first pixel position set is not empty, the j-th second image is updated to the ith second image, the candidate target image is updated to the first image, and the step of filling the first pixel position set with pixels using the N frames of second images to obtain the target image is returned to the step, until all the N frames of second images are matched with the first image or the updated first pixel position set is empty.

[0133] The candidate target image may be an image obtained by filling the background pixels of the first target pixel position set in the third pixel position set into the second target pixel position set in the first pixel position set.

[0134] The jth second image may be any second image in the N second images, where j is a positive integer, and 0≤j≤N, j≠i, that is, the jth second image may be any second image in the N second images except the i-th second image.

[0135] In some embodiments of the present application, after filling the background pixels of the first target pixel position set in the third pixel position set into the second target pixel position set in the first pixel position set to obtain a candidate target image, the first pixel position set can be updated to be a pixel position set in the first pixel position set excluding the second target pixel position set, and then, when it is determined that there is a second image that does not match the first image in the N frames of second images, and the updated first pixel position set is not empty, that is, there is a second image in the N frames of second images that has not been matched with the first image, and the area where the second object in the first image is located has not been completely filled with the background, the jth second image can be updated to the ith second image, the candidate target image can be updated to the first image, and the step of filling the first pixel position set with pixels using the N frames of second images to obtain the target image is returned to be executed, until all N frames of second images are matched with the first image or the updated first pixel position set is empty.

[0136] Continuing with the above example, when using Figure 5 After the background pixels of area 53 in FIG. 4( a) are used to fill the background of area 43, area 41 in FIG. 4( a) is updated to the set of other pixel positions in area 41 except area 43, that is, the set of pixel positions corresponding to object C in FIG. 4( a).

[0137] At present, only the area where object C is located in the first image has not been filled with the background, and the second image of the second frame to the second image of the tenth frame are still used to fill the background of the area where the second object is located in the first image, then the second frame of the second image can be obtained, and then the area where object C is located in the first image can be filled with the background in the manner of filling the background of the area where object B is located in the first image with the first frame of the second image, and then it is determined whether there are any second images in the ten frames of the second image that have not been matched with the first image to fill the background of the first pixel position set in the first image, and whether the area where object C is located in the first image is filled Completed. If it is determined that there are second images in the 10 frames of second images that have not been used to match the first image to perform background filling on the first pixel position set in the first image, and the area where the object C in the first image is located has not been filled, then continue to obtain the third frame of the second image, and then fill the background of the area where the object C is located in accordance with the method of performing background filling on the area where the object B in the first image is located by the first frame of the second image. This cycle is repeated until all 10 frames of second images are used to match the first image to perform background filling on the first pixel position set in the first image, or the background filling of the area where the object C in the first image is located is completed.

[0138] In an embodiment of the present application, a candidate target image is obtained by filling the background pixels of the first target pixel position set in the third pixel position set into the second target pixel position set in the first pixel position set, and then updating the first pixel position set to the fourth pixel position set. When it is determined that there is a second image that does not match the first image in the N frames of second images, and the updated first pixel position set is not empty, the jth second image is updated to the ith second image, the candidate target image is updated to the first image, and the step of filling the first pixel position set with pixels using the N frames of second images to obtain the target image is returned to the execution until all the N frames of second images are matched with the first image or the updated first pixel position set is empty. Compared with the prior art method of filling with background pixels at positions symmetrical to the first pixel position set in the first image, the solution of the embodiment of the present application can improve the artifacts and blurring problems of the picture obtained by the prior art solution, and obtain an image with better effect.

[0139] In some embodiments of the present application, in order to obtain a target image that meets user requirements, after updating the first pixel position set to a fourth pixel position set, the above method may further include:

[0140] When it is determined that all N frames of second images have been matched with the first image and the updated first pixel position set is not empty, determining a fifth pixel position set in the N frames of second images that matches the updated first pixel position set;

[0141] The background pixels of the sixth pixel position set symmetrical to the fifth pixel position set in the N-frame second image are filled into the updated first pixel position set to obtain a target image.

[0142] The fifth pixel position set is a set of pixel positions in the N frames of the second image that matches the updated first pixel position set.

[0143] The sixth pixel position set is a set of pixel positions in the N-frame second image that is symmetrical to the fifth pixel position set.

[0144] In some embodiments of the present application, when it is determined that all N frames of second images are matched with the first image and the updated first pixel position set is not empty, that is, all N frames of second images are used to match with the first image to perform background filling on the first pixel position set in the first image, and there is still an area in the first image where the second object is located that has not been completely filled, the fifth pixel position set in the N frames of second images that matches the updated first pixel position set can be determined, and then the background pixels of the sixth pixel position set in the N frames of second images that is symmetrical to the fifth pixel position set are filled into the updated first pixel position set to obtain the target image.

[0145] It should be noted that when determining the fifth pixel position set in N frames of second images that matches the updated first pixel position set, it is possible to select any one or several frames of the N frames of second images and determine the fifth pixel position set that matches the updated first pixel position set from the selected any one or several frames of the second images.

[0146] When selecting any one or several second images from N frames of second images, the second image with a complete background in the sixth pixel position set may be selected. In this way, when the updated first pixel position set is filled with the background pixels in the sixth pixel position set of the second image, the updated first pixel position set can be filled well.

[0147] Continuing to refer to the above example, a certain frame of the second image is selected from the N frames of the second image, for example, Figure 5 Taking the second image of the first frame in FIG. 4( a) as an example, after updating the area 41 in FIG. 4( a) to the pixel position set corresponding to the object C, refer to Figure 6 , Figure 6 is a schematic diagram of a candidate target image, in which only the area 61 where the object C is located is left unfilled. Figure 5 Zhongyu Figure 6 The area 61 in matches the area 52, that is, the area 52 is the fifth pixel position set.

[0148] Continue to refer Figure 5 ,exist Figure 5 In the figure, region 54 which is symmetrical to region 52 is taken as the sixth pixel position set.

[0149] In an embodiment of the present application, when it is determined that all N frames of second images are matched with the first image and the updated first pixel position set is not empty, the fifth pixel position set in the N frames of second images that matches the updated first pixel position set can be determined, and then the background pixels of the sixth pixel position set in the N frames of second images that is symmetrical to the fifth pixel position set are filled into the updated first pixel position set to obtain the target image. In this way, it can be ensured that all the first pixel position sets corresponding to the M second objects in the first image are filled, so as to ensure that the obtained target image is an image with the background filling completed to meet the needs of users.

[0150] In some embodiments of the present application, in order to improve the accuracy of background filling for the first pixel position set in the first image, before matching the i-th second image in the N frames of second images with the first image and obtaining the second pixel position set in the i-th second image that matches the first pixel position information set, the above method may further include:

[0151] Convert the second image of the i-th frame into a grayscale image to obtain a second grayscale image;

[0152] Matching the second grayscale image with the first grayscale image to obtain a transformation matrix between the second grayscale image and the first grayscale image;

[0153] Based on the transformation matrix, the second grayscale image is converted into a coordinate system corresponding to the first grayscale image to obtain a third grayscale image;

[0154] The matching of the i-th second image in the N-frame second image with the first image to obtain a second pixel position set in the i-th second image that matches the first pixel position information set may specifically include:

[0155] The third grayscale image is matched with the first image to obtain a second pixel position set in the third grayscale image that matches the first pixel position information set.

[0156] The second grayscale image may be a grayscale image obtained by converting the second image of the i-th frame into a grayscale image.

[0157] The third grayscale image may be a grayscale image obtained by converting the second grayscale image into a coordinate system corresponding to the first grayscale image based on a transformation matrix between the first grayscale image and the second grayscale image.

[0158] In some embodiments of the present application, after executing the first input, the user needs to keep the camera still and continue to stay in the first time period. However, a handheld camera cannot guarantee the absolute stillness of the camera, and there may be uncertain factors such as jitter. This makes the N-frame second image in the video clip inconsistent with the picture range, coordinate system, etc. framed by the first image. Therefore, it is necessary to first convert the i-th frame second image into a grayscale image to obtain a second grayscale image, and then match the second grayscale image with the first grayscale image to obtain a transformation matrix between the second grayscale image and the first grayscale image. Based on the transformation matrix, the second grayscale image is converted to the coordinate system corresponding to the first grayscale image to obtain a third grayscale image. In this way, the third grayscale image can be matched with the first image to obtain a second pixel position set in the third grayscale image that matches the first pixel position information set.

[0159] In an embodiment of the present application, the second grayscale image is obtained by converting the i-th frame second image into a grayscale image, the second grayscale image is matched with the first grayscale image, and a transformation matrix between the second grayscale image and the first grayscale image is obtained. Then, based on the transformation matrix, the second grayscale image is converted to a coordinate system corresponding to the first grayscale image to obtain a third grayscale image. Subsequently, the third grayscale image can be matched with the first image, thereby ensuring the consistency of the picture range and coordinate system framed by the N-frame second image and the first image, thereby improving the accuracy of background filling for the first pixel position set in the first image.

[0160] In some embodiments of the present application, in order to accurately obtain the transformation matrix between the second grayscale image and the first grayscale image, matching the second grayscale image with the first grayscale image to obtain the transformation matrix between the second grayscale image and the first grayscale image may specifically include:

[0161] Matching the features in the second grayscale image with the features in the first grayscale image to obtain Q matching point pairs;

[0162] A transformation matrix between the second grayscale image and the first grayscale image is determined based on the Q matching point pairs.

[0163] The matching point pair may be a feature pair in which a feature in the second grayscale image matches a feature in the first grayscale image.

[0164] In some embodiments of the present application, a feature detection and matching algorithm can be used to detect and match features in the first grayscale image and features in the second grayscale image to obtain Q matching point pairs, and then based on the Q matching point pairs, a transformation matrix between the second grayscale image and the first grayscale image can be obtained.

[0165] In some embodiments of the present application, the above-mentioned feature detection and matching algorithms may include but are not limited to: scale-invariant feature transform (SIFT) algorithm, fast feature point extraction description (ORB) algorithm or speeded up robust feature extraction description (SURF) algorithm, etc.

[0166] In some embodiments of the present application, taking the SIFT algorithm as an example, the SIFT algorithm is used to find multiple feature points from the first grayscale image and the second grayscale image respectively, and the descriptors of each feature point in the first grayscale image and the second grayscale image are obtained. Then, the distances between the descriptors of each feature point in the first grayscale image and the descriptors of each feature point in the second grayscale image are calculated, and two feature points whose distance is less than a certain threshold are determined as a matching point pair, so that Q matching point pairs are obtained. Then, the cv2.getPerspectiveTransform() function can be used to process the Q matching point pairs to obtain the transformation matrix between the second grayscale image and the first grayscale image.

[0167] In an embodiment of the present application, by matching features in the second grayscale image with features in the first grayscale image, Q matching point pairs are obtained, and based on the Q matching point pairs, the transformation matrix between the second grayscale image and the first grayscale image is determined, so that the transformation matrix between the second grayscale image and the first grayscale image can be accurately obtained.

[0168] In order to more clearly understand the solution of the embodiment of the present application, the photographing method provided by the embodiment of the present application is introduced below with a specific scenario. Figure 7 is a flow chart of a photographing method provided in an embodiment of the present application, such as Figure 7 As shown, the photographing method provided in the embodiment of the present application may include steps 1 to 14.

[0169] Step 1: Display a preview image of the shooting scene.

[0170] Step 2: In response to a second input of the user to the preview image, a skeleton coordinate set of the first object is recorded.

[0171] In step 2, the skeleton coordinate set of the first object may be determined in the manner described in the above embodiment, which will not be described in detail herein.

[0172] Step 3: Whether to execute the first input, if yes, execute step 4, if not, return to execute step 2.

[0173] In step 3, determine whether the user has executed the first input. If so, execute step 4. If not, return to step 2, redisplay the preview image of the shooting scene, then track the first object selected by the user, and update the skeleton coordinate set of the first object.

[0174] Step 4: Save the first image, the first object, and the first skeleton coordinate set of the M second objects, and record the time of the first input.

[0175] In step 4, after the user performs the first input, a first image can be obtained, the first image is saved, and the skeleton coordinate set of the first object and the skeleton coordinate sets of M second objects in the first image are saved, and then the time of the first input is recorded.

[0176] Step 5: Take a first grayscale image of the first image, and calculate a first pixel position set of M second objects in the first grayscale image.

[0177] In step 5, a first grayscale image gray_k of the first image mat_k is obtained, and then a first pixel position set Zc where the M second objects are located in the first grayscale image gray_k is calculated.

[0178] Step 6: Obtain the second image of the i-th frame, and calculate the second grayscale image of the second image of the i-th frame.

[0179] In step 6, according to the time of the first input, a video clip within the first time period after the first input can be collected to obtain N frames of second images, and then the i-th frame second image mat_video is obtained from the N frames of second images, and then the second grayscale image gray_video of the i-th frame second image is obtained.

[0180] Step 7: Calculate the transformation matrix between the first grayscale image and the second grayscale image.

[0181] In step 7, the transformation matrix between the first grayscale image and the second grayscale image may be calculated according to the calculation method of the transformation matrix in the above embodiment, which will not be described in detail here.

[0182] Step 8: Convert the second grayscale image to the coordinate system of the first grayscale image to obtain a third grayscale image.

[0183] In step 8, mat_video may be subjected to perspective transformation to obtain a transformed image mat_video_transform, which is then subjected to grayscale conversion to obtain a corresponding third grayscale image gray_video_transform.

[0184] Step 9: Perform foreground segmentation on the third grayscale image to obtain a third pixel position set of P third objects in the third grayscale image.

[0185] In step 9, the third grayscale image is foreground segmented to obtain a third pixel position set Zm of P third objects in the third grayscale image.

[0186] Step 10: Fill the background pixels of the first target pixel position set in the third grayscale image into the second target pixel position set in the first pixel position set, and update the first pixel position set.

[0187] In step 10, Zc-(Zc∩Zm) in gray_video_transform, that is, the background pixels in the first target pixel position set, can be filled into mat_k, specifically into the second target pixel position set in mat_k, and Zc=(Zc∩Zm) in the first image is updated.

[0188] Step 11, determine whether all N frames of the second image are used and whether the first pixel position set is empty, if not, return to step 6, if yes, execute step 12.

[0189] In step 11, if all N frames of second images are not used and the first pixel position set is not empty, return to step 6 to reacquire the j-th frame of second image, and re-execute the above steps 6-11 until all N frames of second images are used and the first pixel position set is empty.

[0190] Step 12: determine whether the first pixel position set is empty, if so, execute step 13, if not, execute step 14.

[0191] Step 13: Return the target image.

[0192] Step 14: Perform symmetry and sequence analysis on the remaining areas and fill them.

[0193] In step 14, the remaining area is analyzed and filled for symmetry and sequence, which is the process in the above embodiment, that is, when it is determined that all N frames of second images are matched with the first image and the updated first pixel position set is not empty, a fifth pixel position set in the N frames of second images that matches the updated first pixel position set is determined, and background pixels of the sixth pixel position set in the N frames of second images that is symmetrical to the fifth pixel position set are filled into the updated first pixel position set to obtain the target image.

[0194] The photographing method provided in the embodiment of the present application can be executed by a photographing device. In the embodiment of the present application, the photographing device provided in the embodiment of the present application is described by taking the photographing method executed by the photographing device as an example.

[0195] Figure 8 FIG. 1 is a schematic diagram showing the structure of an information processing device according to an exemplary embodiment. Figure 8 As shown, the photographing device 800 may include:

[0196] A first receiving module 810, configured to receive a first input from a user when displaying a preview image;

[0197] A first determining module 820 is configured to image the preview image in response to the first input to obtain a first image, wherein the first image includes a first object and M second objects;

[0198] An acquisition module 830 is configured to acquire video clips within a first time period after the first input is received, to obtain N frames of second images, where M and N are both positive integers;

[0199] An extraction module 840 is used to extract a first pixel position set of the M second objects in the first image and background pixels at positions corresponding to the M second objects in the first image in the N frames of second images;

[0200] The second determination module 850 is configured to perform pixel filling on the first pixel position set using background pixels at positions corresponding to the M second objects in the first image in the N frames of second images to obtain a target image.

[0201] In an embodiment of the present application, when displaying a preview image of a captured scene, the preview image is imaged in response to a first input from a user to obtain a first image, in which a first object and M second objects are included, and then a video clip of the captured scene within a first time period after the first input is received is collected to obtain N frames of second images, and then the first pixel position set is pixel-filled using background pixels at positions corresponding to the M second objects in the first image in the N frames of second images of the captured scene within the first time period after the first input is received. Compared with the prior art method of filling with background pixels at positions symmetrical to the first pixel position set in the first image, the solution of the embodiment of the present application can improve the artifacts and blurring problems of the picture obtained by the prior art solution when facing a complex background, and a better image can be obtained by the solution of the embodiment of the present application.

[0202] In some embodiments of the present application, the preview image includes at least one object, and the above-mentioned apparatus may further include:

[0203] A second receiving module, configured to receive a second input of the user on the preview image before receiving the first input of the user;

[0204] a third determining module, configured to determine first position information of the second input in response to the second input;

[0205] a fourth determination module, configured to determine, after obtaining the first image, second position information corresponding to the at least one object according to the first skeleton coordinate set corresponding to the at least one object;

[0206] A fifth determining module is used to determine the first object and M second objects from the at least one object according to the distance between the first position information and the second position information respectively corresponding to the at least one object.

[0207] In some embodiments of the present application, the fifth determination module is specifically used to:

[0208] Calculating the distance between the first position information and the second position information respectively corresponding to the at least one object to obtain the first distance respectively corresponding to the at least one object;

[0209] Determine the object corresponding to the minimum value of the first distances respectively corresponding to the at least one object as the first object;

[0210] The other objects among the at least one object except the first object are determined as the M second objects.

[0211] In some embodiments of the present application, the extraction module 840 may include:

[0212] A first determining unit, configured to convert the first image into a grayscale image to obtain a first grayscale image;

[0213] A first segmentation unit, configured to segment the foreground of the first grayscale image to obtain a first foreground image;

[0214] The second determining unit is used to determine a first pixel position set of the M second objects in the first image based on the first skeleton coordinate sets respectively corresponding to the first foreground image and the M second objects.

[0215] In some embodiments of the present application, the second determining unit is specifically configured to:

[0216] Determine a foreground area corresponding to the M second objects based on the first foreground image and the first skeleton coordinate sets respectively corresponding to the M second objects;

[0217] Masks are added to pixels in other foreground areas of the first foreground image except for the foreground areas corresponding to the M second objects to obtain a first pixel position set of the M second objects in the first image.

[0218] In some embodiments of the present application, the extraction module 840 may further include:

[0219] an acquisition unit, configured to match an i-th second image in the N frames of second images with the first image, and acquire a second pixel position set in the i-th second image that matches the first pixel position set, wherein i is a positive integer and 1≤i≤N;

[0220] The third determining unit is used to determine the background pixels at the second pixel position set in the i-th frame of the second image as the background pixels at positions in the i-th frame of the second image corresponding to the M second objects in the first image.

[0221] In some embodiments of the present application, the second determination module 850 may include:

[0222] an extraction unit, configured to extract a third pixel position set of P third objects in the i-th frame of the second image, wherein the P third objects are not completely the same as the M second objects, and P is a positive integer;

[0223] a fourth determining unit, configured to calculate an intersection of the first pixel position set and the third pixel position set to obtain a fourth pixel position set;

[0224] A fifth determination unit is used to fill the background pixels of the first target pixel position set in the third pixel position set into the second target pixel position set in the first pixel position set to obtain a target image, wherein the first target pixel position set is the pixel position set in the second pixel position set excluding the fourth pixel position set, and the second target pixel position set is the pixel position set in the first pixel position set that matches the first target pixel position set.

[0225] In some embodiments of the present application, the fifth determining unit is specifically configured to:

[0226] Filling the background pixels of the first target pixel position set in the third pixel position set into the second target pixel position set in the first pixel position set to obtain a candidate target image;

[0227] Updating the first pixel position set to the fourth pixel position set;

[0228] When it is determined that there is a second image that does not match the first image among the N frames of second images, and the updated first pixel position set is not empty, the j-th second image is updated to the ith second image, the candidate target image is updated to the first image, and the step of filling the first pixel position set with pixels using the N frames of second images to obtain the target image is returned to the step, until all the N frames of second images are matched with the first image or the updated first pixel position set is empty, j is a positive integer, and 1≤j≤N, j≠i.

[0229] In some embodiments of the present application, after updating the first pixel position set to the fourth pixel position set, the fifth determining unit is further specifically configured to:

[0230] When it is determined that the N frames of second images are all matched with the first image and the updated first pixel position set is not empty, determining a fifth pixel position set in the N frames of second images that matches the updated first pixel position set;

[0231] The background pixels of a sixth pixel position set symmetrical to the fifth pixel position set in the N-frame second image are filled into the updated first pixel position set to obtain a target image.

[0232] In some embodiments of the present application, the above-mentioned device may further include:

[0233] a sixth determining module, configured to convert the i-th second image in the N frames of second images into a grayscale image to obtain a second grayscale image before matching the i-th second image in the N frames of second images with the first image to obtain a second pixel position set in the i-th second image that matches the first pixel position information set;

[0234] a seventh determination module, configured to match the second grayscale image with the first grayscale image to obtain a transformation matrix between the second grayscale image and the first grayscale image;

[0235] an eighth determination module, configured to transform the second grayscale image into a coordinate system corresponding to the first grayscale image based on the transformation matrix to obtain a third grayscale image;

[0236] The acquisition unit is specifically used for:

[0237] The third grayscale image is matched with the first image to obtain a second pixel position set in the third grayscale image that matches the first pixel position information set.

[0238] In some embodiments of the present application, the seventh determination module is specifically used to:

[0239] Matching features in the second grayscale image with features in the first grayscale image to obtain Q matching point pairs;

[0240] A transformation matrix between the second grayscale image and the first grayscale image is determined according to the Q matching point pairs.

[0241] The photographing device in the embodiment of the present application may be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal or other devices other than a terminal. Exemplarily, the electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmentedreality, AR) / virtual reality (virtualreality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It may also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0242] The photographing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0243] The photographing device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0244] Alternatively, if Fig. 9 As shown, an embodiment of the present application further provides an electronic device 900, including a processor 901 and a memory 902, wherein the memory 902 stores a program or instruction that can be executed on the processor 901, and when the program or instruction is executed by the processor 901, each step of the above-mentioned photographing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0245] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0246] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0247] Fig.10 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0248] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.

[0249] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.10 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0250] The user input unit 1007 is used to receive a first input from a user when displaying a preview image;

[0251] Processor 1010 is used to image the preview image in response to the first input to obtain a first image, wherein the first image includes a first object and M second objects; collect video clips of the shooting scene within a first time period after receiving the first input to obtain N frames of second images, where M and N are both positive integers; extract a first pixel position set of the M second objects in the first image and background pixels at positions corresponding to the M second objects in the N frames of the second image; and perform pixel filling on the first pixel position set using background pixels at positions corresponding to the M second objects in the N frames of the second image to obtain a target image.

[0252] In this way, when displaying a preview image of a captured scene, the preview image is imaged in response to a first input from a user to obtain a first image, in which a first object and M second objects are included, and then a video clip of the captured scene within a first time period after the first input is received is collected to obtain N frames of second images, and then the first pixel position set is filled with pixels using background pixels at positions corresponding to the M second objects in the first image in the N frames of second images of the captured scene within the first time period after the first input is received. Compared with the prior art method of filling with background pixels at positions symmetrical to the first pixel position set in the first image, the solution of the embodiment of the present application can improve the artifacts and blurring problems of the picture obtained by the prior art solution when facing a complex background, and a better image can be obtained by the solution of the embodiment of the present application.

[0253] Optionally, the preview image includes at least one object, and the user input unit 1007 is further configured to receive a second input from a user on the preview image before receiving the first input from the user;

[0254] Processor 1010 is also used to determine the first position information of the second input in response to the second input; and after obtaining the first image, determine the second position information corresponding to the at least one object according to the first skeleton coordinate set corresponding to the at least one object; and determine the first object and M second objects from the at least one object according to the distance between the first position information and the second position information corresponding to the at least one object.

[0255] In this way, by responding to the user's second input to the preview image, the first position information of the second input is determined, and then the second position information corresponding to at least one object is determined according to the first skeleton coordinate set corresponding to the at least one object. The first object and M second objects can be determined from at least one object based on the distance between the first position information and the second position information corresponding to the at least one object. In this way, the first object and the second object can be determined according to the skeleton coordinate set of each object in the preview image, thereby improving the accuracy of determining the first object and the second object.

[0256] Optionally, the processor 1010 is further used to calculate the distance between the first position information and the second position information corresponding to the at least one object, to obtain the first distance corresponding to the at least one object; determine the object corresponding to the minimum value of the first distances corresponding to the at least one object as the first object; and determine the other objects among the at least one object except the first object as the M second objects.

[0257] In this way, by calculating the distance between the first position information and the second position information corresponding to at least one object, the first distance corresponding to at least one object is obtained, and then the object corresponding to the minimum value of the first distance corresponding to at least one object is determined as the first object, and the other objects except the first object in at least one object are determined as M second objects. In this way, the first object and the second object in the first image can be accurately determined, further improving the accuracy of determining the first object and the second object.

[0258] Optionally, the processor 1010 is further used to convert the first image into a grayscale image to obtain a first grayscale image; segment the foreground of the first grayscale image to obtain a first foreground image; and determine a first pixel position set of the M second objects in the first image based on the first skeleton coordinate sets corresponding to the first foreground image and the M second objects respectively.

[0259] In this way, by converting the first image into a grayscale image, a first grayscale image is obtained, and then the foreground in the first grayscale image is extracted to obtain a first foreground image, and then according to the first skeleton coordinate sets corresponding to the first foreground image and the M second objects respectively, the first pixel position set of the M second objects in the first image can be accurately obtained.

[0260] Optionally, the processor 1010 is further used to determine the foreground area corresponding to the M second objects based on the first foreground image and the first skeleton coordinate sets respectively corresponding to the M second objects; add a mask to the pixels of the foreground area other than the foreground area corresponding to the M second objects in the first foreground image, to obtain a first pixel position set of the M second objects in the first image.

[0261] In this way, by comparing the first foreground image and the first skeleton coordinate sets corresponding to the M second objects respectively, the foreground areas corresponding to the M second objects can be determined, and then masks are added to the pixels of the foreground areas other than the foreground areas corresponding to the M second objects in the first foreground image. In this way, the first pixel position set of the M second objects in the first image can be accurately obtained, further improving the accuracy of determining the first pixel position set.

[0262] Optionally, the processor 1010 is further used to match the i-th second image in the N frames of second images with the first image, obtain a second pixel position set in the i-th second image that matches the first pixel position set, where i is a positive integer and 1≤i≤N; and determine the background pixels at the second pixel position set in the i-th second image as the background pixels at positions in the i-th second image corresponding to the M second objects in the first image.

[0263] In this way, by matching the i-th second image in N frames of second images with the first image, a second pixel position set in the i-th second image that matches the first pixel position set is obtained, so that the background pixels at the second pixel position set can be directly determined as the background pixels at the positions in the i-th second image corresponding to the M second objects in the first image. In this way, the background pixels in the i-th second image at the positions corresponding to the M second objects in the first image are accurately determined, thereby improving the accuracy of determining the background pixels in the i-th second image at the positions corresponding to the M second objects in the first image.

[0264] Optionally, the processor 1010 is further used to extract a third pixel position set of P third objects in the second image of the i-th frame, wherein the P third objects are not completely the same as the M second objects, and P is a positive integer; calculate the intersection of the first pixel position set and the third pixel position set to obtain a fourth pixel position set; fill the background pixels of the first target pixel position set in the third pixel position set into the second target pixel position set in the first pixel position set to obtain a target image, wherein the first target pixel position set is a pixel position set in the third pixel position set excluding the fourth pixel position set, and the second target pixel position set is a pixel position set in the first pixel position set that matches the first target pixel position set.

[0265] In this way, compared with the prior art that uses background pixels at positions symmetrical to the first pixel position set in the first image for filling, the embodiment of the present application extracts the third pixel position set of P third objects in the second image of the i-th frame, and then calculates the intersection of the first pixel position set and the third pixel position set to obtain a fourth pixel position set, and fills the background pixels of the first target pixel position set in the third pixel position set into the second target pixel position set in the first pixel position set to obtain a target image. This can improve the artifacts and blur problems of the picture obtained by the prior art solution and obtain an image with better effect.

[0266] Optionally, the processor 1010 is further used to fill the background pixels of the first target pixel position set in the third pixel position set into the second target pixel position set in the first pixel position set to obtain a candidate target image; update the first pixel position set to be a pixel position set in the first pixel position set excluding the second target pixel position set; when it is determined that there is a second image that does not match the first image among the N frames of second images, and the updated first pixel position set is not empty, update the jth second image to the ith second image, update the candidate target image to the first image, and return to execute the step of filling the first pixel position set with pixels using the N frames of second images to obtain a target image, until the N frames of second images are all matched with the first image or the updated first pixel position set is empty, j is a positive integer, and 1≤j≤N, j≠i.

[0267] In this way, a candidate target image is obtained by filling the background pixels of the first target pixel position set in the third pixel position set into the second target pixel position set in the first pixel position set, and then the first pixel position set is updated to the fourth pixel position set. When it is determined that there is a second image that does not match the first image in the N frames of second images, and the updated first pixel position set is not empty, the jth second image is updated to the ith second image, the candidate target image is updated to the first image, and the step of filling the first pixel position set with pixels using the N frames of second images to obtain the target image is returned to the step until all the N frames of second images are matched with the first image or the updated first pixel position set is empty. Compared with the prior art method of filling with background pixels at positions symmetrical to the first pixel position set in the first image, the solution of the embodiment of the present application can improve the artifacts and blurring problems of the picture obtained by the prior art solution and obtain an image with better effect.

[0268] Optionally, the processor 1010 is further used to, after updating the first pixel position set to the fourth pixel position set, determine a fifth pixel position set in the N frames of second images that matches the updated first pixel position set, when it is determined that the N frames of second images are all matched with the first image and the updated first pixel position set is not empty; and fill the background pixels of a sixth pixel position set in the N frames of second images that is symmetrical to the fifth pixel position set into the updated first pixel position set to obtain a target image.

[0269] In this way, when it is determined that all N frames of second images have matched the first image and the updated first pixel position set is not empty, the fifth pixel position set in the N frames of second images that matches the updated first pixel position set can be determined, and then the background pixels of the sixth pixel position set in the N frames of second images that is symmetrical to the fifth pixel position set are filled into the updated first pixel position set to obtain the target image. In this way, it can be ensured that all the first pixel position sets corresponding to the M second objects in the first image are filled, so as to ensure that the obtained target image is an image with the background filling completed to meet the needs of users.

[0270] Optionally, the processor 1010 is further used to convert the i-th frame second image in the N frames of second images into a grayscale image before matching the i-th frame second image in the N frames of second images with the first image to obtain a second pixel position set in the i-th frame second image that matches the first pixel position information set, to obtain a second grayscale image; match the second grayscale image with the first grayscale image to obtain a transformation matrix between the second grayscale image and the first grayscale image; based on the transformation matrix, convert the second grayscale image to a coordinate system corresponding to the first grayscale image to obtain a third grayscale image; match the third grayscale image with the first image to obtain a second pixel position set in the third grayscale image that matches the first pixel position information set.

[0271] In this way, the second image of the i-th frame is converted into a grayscale image to obtain a second grayscale image, the second grayscale image is matched with the first grayscale image to obtain a transformation matrix between the second grayscale image and the first grayscale image, and then based on the transformation matrix, the second grayscale image is converted to the coordinate system corresponding to the first grayscale image to obtain a third grayscale image, which can be subsequently matched with the first image based on the third grayscale image, thereby ensuring the consistency of the picture range and coordinate system framed by the N-frame second image and the first image, thereby improving the accuracy of background filling for the first pixel position set in the first image.

[0272] Optionally, the processor 1010 is further used to match features in the second grayscale image with features in the first grayscale image to obtain Q matching point pairs; and determine a transformation matrix between the second grayscale image and the first grayscale image based on the Q matching point pairs.

[0273] In this way, by matching the features in the second grayscale image with the features in the first grayscale image, Q matching point pairs are obtained. Based on the Q matching point pairs, the transformation matrix between the second grayscale image and the first grayscale image is determined. In this way, the transformation matrix between the second grayscale image and the first grayscale image can be accurately obtained.

[0274] It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of the static picture or video obtained by the image capture device (such as a color camera) in the video capture mode or the image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0275] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0276] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.

[0277] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned photographing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0278] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0279] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned photographing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0280] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0281] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned photographing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0282] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0283] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0284] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A photographing method, characterized in that: The method comprises: In case of displaying the preview image, receiving a first input from a user; In response to the first input, imaging the preview image to obtain a first image, wherein the first image includes a first object and M second objects; Capturing video clips within a first time period after receiving the first input, obtaining N frames of second images, where M and N are both positive integers; Extracting a first pixel position set of the M second objects in the first image and background pixels at positions corresponding to the M second objects in the first image in the N frames of second images; The first pixel position set is pixel-filled using background pixels at positions corresponding to the M second objects in the first image in the N frames of second images to obtain a target image.

2. The method according to claim 1, characterized in that The preview image includes at least one object. Before receiving the first input from the user, the method further includes: receiving a second input from a user regarding the preview image; In response to the second input, determining first position information of the second input; After obtaining the first image, the method further includes: Determining second position information corresponding to the at least one object respectively according to the first skeleton coordinate sets respectively corresponding to the at least one object; The first object and M second objects are determined from the at least one object according to the distance between the first position information and the second position information respectively corresponding to the at least one object.

3. The method according to claim 2, characterized in that The determining the first object and M second objects from the at least one object according to the first position information and the distance between the second position information respectively corresponding to the at least one object comprises: Calculating the distance between the first position information and the second position information respectively corresponding to the at least one object to obtain the first distance respectively corresponding to the at least one object; Determine the object corresponding to the minimum value of the first distances respectively corresponding to the at least one object as the first object; The other objects among the at least one object except the first object are determined as the M second objects.

4. The method according to claim 2, characterized in that: The extracting the first pixel position set of the M second objects in the first image includes: Converting the first image into a grayscale image to obtain a first grayscale image; Segmenting the foreground of the first grayscale image to obtain a first foreground image; Based on the first foreground image and the first skeleton coordinate sets respectively corresponding to the M second objects, a first pixel position set of the M second objects in the first image is determined.

5. The method according to claim 4, characterized in that The determining, based on the first foreground image and the first skeleton coordinate sets respectively corresponding to the M second objects, a first pixel position set of the M second objects in the first image comprises: Determine a foreground area corresponding to the M second objects based on the first foreground image and the first skeleton coordinate sets respectively corresponding to the M second objects; Masks are added to pixels in other foreground areas of the first foreground image except for the foreground areas corresponding to the M second objects to obtain a first pixel position set of the M second objects in the first image.

6. The method according to claim 1, characterized in that Extracting background pixels at positions corresponding to the M second objects in the first image in the N frames of second images includes: Matching the i-th second image in the N frames of second images with the first image to obtain a second pixel position set in the i-th second image that matches the first pixel position set, where i is a positive integer and 1≤i≤N; The background pixels at the second pixel position set in the i-th frame of the second image are determined as the background pixels at positions in the i-th frame of the second image corresponding to the M second objects in the first image.

7. The method according to claim 6, characterized in that The step of performing pixel filling on the first pixel position set by using background pixels at positions corresponding to the M second objects in the first image in the N frames of second images to obtain a target image includes: Extracting a third pixel position set of P third objects in the i-th frame of the second image, wherein the P third objects are not completely the same as the M second objects, and P is a positive integer; Calculating the intersection of the first pixel position set and the third pixel position set to obtain a fourth pixel position set; The background pixels of the first target pixel position set in the third pixel position set are filled into the second target pixel position set in the first pixel position set to obtain a target image, wherein the first target pixel position set is the pixel position set in the third pixel position set excluding the fourth pixel position set, and the second target pixel position set is the pixel position set in the first pixel position set that matches the first target pixel position set.

8. The method according to claim 7, characterized in that The step of filling the background pixels of the first target pixel position set in the third pixel position set into the second target pixel position set in the first pixel position set to obtain a target image comprises: Filling the background pixels of the first target pixel position set in the third pixel position set into the second target pixel position set in the first pixel position set to obtain a candidate target image; Updating the first pixel position set to a pixel position set in the first pixel position set excluding the second target pixel position set; When it is determined that there is a second image that does not match the first image among the N frames of second images, and the updated first pixel position set is not empty, the j-th second image is updated to the ith second image, the candidate target image is updated to the first image, and the step of filling the first pixel position set with pixels using the N frames of second images to obtain the target image is returned to the step, until all the N frames of second images are matched with the first image or the updated first pixel position set is empty, j is a positive integer, and 1≤j≤N, j≠i.

9. The method according to claim 8, characterized in that After updating the first pixel position set to the fourth pixel position set, the method further includes: When it is determined that the N frames of second images are all matched with the first image and the updated first pixel position set is not empty, determining a fifth pixel position set in the N frames of second images that matches the updated first pixel position set; The background pixels of a sixth pixel position set symmetrical to the fifth pixel position set in the N-frame second image are filled into the updated first pixel position set to obtain a target image.

10. The method according to claim 6, characterized in that Before matching the i-th second image in the N frames of second images with the first image and obtaining a second pixel position set in the i-th second image that matches the first pixel position information set, the method further includes: Convert the second image of the i-th frame into a grayscale image to obtain a second grayscale image; Matching the second grayscale image with the first grayscale image to obtain a transformation matrix between the second grayscale image and the first grayscale image; Based on the transformation matrix, convert the second grayscale image to a coordinate system corresponding to the first grayscale image to obtain a third grayscale image; The matching of the i-th second image in the N-frame second image with the first image to obtain a second pixel position set in the i-th second image that matches the first pixel position information set includes: The third grayscale image is matched with the first image to obtain a second pixel position set in the third grayscale image that matches the first pixel position information set.

11. The method according to claim 10, characterized in that The step of matching the second grayscale image with the first grayscale image to obtain a transformation matrix between the second grayscale image and the first grayscale image includes: Matching features in the second grayscale image with features in the first grayscale image to obtain Q matching point pairs; A transformation matrix between the second grayscale image and the first grayscale image is determined according to the Q matching point pairs.

12. A photographing device, characterized in that: The device comprises: A first receiving module, used for receiving a first input from a user when displaying a preview image; A first determining module is configured to image the preview image in response to the first input to obtain a first image, wherein the first image includes a first object and M second objects; A collection module, used for collecting video clips within a first time period after receiving the first input, to obtain N frames of second images, where M and N are both positive integers; An extraction module, configured to extract a first pixel position set of the M second objects in the first image and background pixels at positions corresponding to the M second objects in the first image in the N frames of second images; The second determination module is used to perform pixel filling on the first pixel position set using background pixels at positions corresponding to the M second objects in the first image in the N frames of second images to obtain a target image.

13. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the photographing method according to any one of claims 1 to 11 are implemented.

Images