Image Processing Method, Apparatus, Electronic Device, and Storage Medium

Abstract

The present application provides an image processing method, apparatus, electronic device and storage medium. The method includes: detecting corner points of a target image in a first image sequence to determine a preset number of first corner points, where the first image sequence includes multiple first images obtained by photographing a target object in different states in a target space; performing matte processing on each first image to obtain multiple second images with the target object removed and multiple matte images including only the target object; determining a preset number of second corner points corresponding to a target background image based on the preset number of first corner points, and determining a set of homography matrices according to the first corner points and the second corner points; transforming the target background image to the image perspective of the second image based on the set of homography matrices to obtain an image set including one or more third images; and obtaining a target image sequence according to the third images in the image set and the multiple matte images. The present application can achieve background replacement of images and has strong feasibility and practicality.

Description

Technical Field

[0001] The present application relates to the field of image processing technologies, and in particular, to an image processing method, apparatus, electronic device, and storage medium. Background Art

[0002] In a panoramic shooting scenario, it is necessary to shoot an object to be photographed from multiple angles to generate a display video for displaying the object to be photographed, so as to display the object to be photographed from different angles through the display video.

[0003] Currently, for the need to change the display background and obtain specific images in a panoramic shooting scenario, it is usually to replace the actual background and reshoot after replacing the actual background to obtain the required images. However, this processing method has high shooting costs, a cumbersome process, a long shooting cycle, and low overall efficiency. Summary of the Invention

[0004] In view of the above problems, embodiments of the present application provide an image processing method, apparatus, electronic device, and storage medium that overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, including:

[0006] Performing corner point detection on a target image to determine a preset number of first corner points corresponding to the background part of the target image, where the target image is an image in a first image sequence, and the first image sequence includes multiple first images obtained by photographing a target object in different states in a target space;

[0007] Performing matte processing on each frame of the first image sequence to obtain multiple second images with the target object removed and multiple matte images including only the target object;

[0008] Determining a preset number of second corner points corresponding to a target background image based on the preset number of first corner points, and determining a homography matrix set according to the preset number of first corner points and the preset number of second corner points;

[0009] Transforming the target background image to the image perspective of the second image based on the homography matrix set to obtain an image set including one or more third images;

[0010] Stitching the third images in the image set with the multiple matte images to obtain a target image sequence.

[0011] In a second aspect, an embodiment of the present application provides an image processing apparatus, including:

[0012] A detection and determination module, configured to perform corner point detection on a target image to determine a preset number of first corner points corresponding to the background part of the target image. The target image is an image in a first image sequence, and the first image sequence includes multiple first images obtained by photographing a target object in different states in a target space.

[0013] A processing and acquisition module, configured to perform matte processing on each first image in the first image sequence to obtain multiple second images with the target object removed and multiple matte images including only the target object.

[0014] A determination module, configured to determine a preset number of second corner points corresponding to a target background image based on the preset number of first corner points, and determine a set of homography matrices according to the preset number of first corner points and the preset number of second corner points.

[0015] A transformation and acquisition module, configured to transform the target background image to the image perspective of the second image based on the set of homography matrices, and obtain an image set including one or more third images.

[0016] A stitching and acquisition module, configured to stitch the third images in the image set with the multiple matte images to obtain a target image sequence.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, including a processor, a memory, and a computer program stored on the memory and executable on the processor. When the computer program is executed by the processor, the steps of the image processing method described in the first aspect above are implemented.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image processing method described in the first aspect above are implemented.

[0019] In the technical solution of the embodiment of the present application, by detecting the corner points of the target image, a preset number of first corner points corresponding to the target image are determined. By performing matte processing on the images in the first image sequence, multiple frames of second images with the target object removed and multiple frames of matte images including only the target object are obtained. In the case of determining the preset number of first corner points, a preset number of second corner points are determined based on the first corner points, and the second corner points can be calibrated quickly and accurately based on the pre-determined first corner points; after determining the first corner points and the second corner points, a set of homography matrices is determined according to the first corner points and the second corner points. Based on the set of homography matrices, the target background image is transformed to the image perspective of one frame or multiple frames of second images, and an image set including one frame or multiple frames of third images is obtained. The images in the image set and the multiple frames of matte images are stitched together to obtain the target image sequence. The background of the image can be replaced through image processing. Compared with the scheme of replacing the real background, it has the advantages of convenience, low cost, short shooting cycle, high efficiency, and strong feasibility and practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram showing the image processing method provided by the embodiment of the present application;

[0021] Figure 2 A schematic diagram showing a matte image provided by the embodiment of the present application;

[0022] Figure 3 A schematic diagram showing the target image provided by the embodiment of the present application;

[0023] Figure 4 A schematic diagram showing the target background image provided by the embodiment of the present application;

[0024] Figure 5 A schematic diagram showing the fourth image provided by the embodiment of the present application;

[0025] Figure 6 A schematic diagram showing the image processing apparatus provided by the embodiment of the present application;

[0026] Figure 7 A schematic diagram showing the structure of the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0028] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the appearances of "in one embodiment" or "in an embodiment" throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner.

[0029] In various embodiments of the present application, it should be understood that the magnitudes of the serial numbers of the following processes do not mean the sequence of execution, and the execution sequence of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0030] The image processing method provided by the embodiments of the present application will be introduced below. See Figure 1 As shown, the image processing method includes the following steps:

[0031] Step 101: Detect the corner points of the target image to determine a preset number of first corner points corresponding to the background part of the target image. The target image is an image in a first image sequence, and the first image sequence includes multiple first images obtained by photographing a target object in different states in a target space.

[0032] For the image processing method provided by the present application, after obtaining a first image sequence including multiple first images, a first image is selected from the first image sequence and used as the target image, and the selected target image is subjected to corner point detection to determine a preset number of first corner points corresponding to the background part of the target image.

[0033] For the first image sequence, it includes multiple first images obtained by photographing a target object in different states in a target space. The part corresponding to the target space in the first image is the background part, and the part corresponding to the target object in the first image is the main part (i.e., the foreground part). The multiple first images included in the first image sequence correspond to the same background, and the states of the target objects corresponding to different first images are different.

[0034] Step 102: Perform matte processing on each first image in the first image sequence to obtain multiple second images with the target object removed and multiple matte images including only the target object.

[0035] For each first image in the first image sequence, perform matte processing on the first image. The matte object is the target object in the first image, that is, perform matte processing on the target object in the first image. By performing matte processing, a second image with the target object removed and a matte image including only the target object are obtained.

[0036] By performing matte extraction on each frame of the first image in the first image sequence, multiple frames of second images and multiple frames of matte images can be obtained. For the multiple frames of matte images, the states of the target objects corresponding to different matte images are different.

[0037] See Figure 2 As shown, it is a specific example of obtaining a matte image including the target object (car) after performing matte extraction on the first image including the target space (shooting studio) and the target object (car).

[0038] Step 103: Determine a preset number of second corner points corresponding to the target background image based on the preset number of first corner points, and determine a set of homography matrices according to the preset number of first corner points and the preset number of second corner points.

[0039] After determining the preset number of first corner points, a preset number of second corner points corresponding to the target background image can be determined based on the preset number of first corner points corresponding to the background part of the target image. The target background image is the background image to be replaced, which can be understood as the provided standard background image. By determining the preset number of second corner points based on the preset number of first corner points, the second corner points can be directly calibrated based on the pre-determined first corner points.

[0040] After determining the preset number of first corner points and the preset number of second corner points, a set of homography matrices is determined based on the preset number of first corner points and the preset number of second corner points. The set of homography matrices can include multiple homography matrices determined based on the first corner points and the second corner points, so as to facilitate subsequent homography transformation based on the homography matrices.

[0041] Step 104: Transform the target background image to the image perspective of the second image based on the set of homography matrices, and obtain an image set including one or more frames of third images.

[0042] After determining the set of homography matrices, perform homography transformation on the target background image based on the set of homography matrices to obtain one or more frames of third images. When transforming the target background image to the image perspective of a second image based on the set of homography matrices, one frame of third image can be obtained. When, for each frame of the second image, transforming the target background image to the corresponding image perspective of the second image based on the set of homography matrices, multiple frames of third images corresponding to the multiple frames of the second images can be obtained. Therefore, the image set can include one or more frames of third images.

[0043] Step 105: Stitch the third images in the image set with the multiple frames of matte images to obtain a target image sequence.

[0044] In the case of obtaining an image set, the third image in the image set is stitched with multiple frame matte images to obtain a target image sequence through image stitching, and the number of images included in the target image sequence is the same as the number of multiple frame matte images.

[0045] For the case where the image set only includes one frame of the third image, the target image sequence can be determined based on one frame of the third image and multiple frame matte images; for the case where the image set includes multiple frames of the third image, the target image sequence can be determined based on multiple frames of the third image and multiple frame matte images.

[0046] In the above implementation, by detecting the corner points of the target image, a preset number of first corner points corresponding to the target image are determined. By performing matte processing on the images in the first image sequence, multiple frame second images with the target object removed and multiple frame matte images including only the target object are obtained. In the case of determining a preset number of first corner points, a preset number of second corner points are determined based on the first corner points, and the second corner points can be quickly and accurately calibrated based on the pre-determined first corner points; after determining the first corner points and the second corner points, a set of homography matrices is determined according to the first corner points and the second corner points. Based on the set of homography matrices, the target background image is transformed to the image perspective of one frame or multiple frame second images, and an image set including one frame or multiple frame third images is obtained. The images in the image set and the multiple frame matte images are stitched to obtain a target image sequence, and the background of the image can be replaced through image processing. Compared with the scheme of replacing the real background, it has the advantages of convenience, low cost, short shooting cycle, high efficiency, and strong feasibility and practicability.

[0047] The process of determining the first corner points is introduced below. When detecting the corner points of the wall of the target image to determine a preset number of first corner points corresponding to the background part of the target image, it includes:

[0048] After selecting the target image in the first image sequence, based on a preset detection model, corner point detection is performed on the background part of the target image to determine the preset number of first corner points;

[0049] Wherein, in the target image, the area corresponding to the target space is the background part, and the area corresponding to the target object is the main body part.

[0050] After obtaining the first image sequence and selecting the target image in the first image sequence, corner point detection is performed on the background part of the target image based on a preset detection model to determine a preset number of first corner points corresponding to the background part of the target image, and using the model for detection can improve the accuracy of corner point detection.

[0051] The preset detection model in this embodiment is a deep learning model, and the deep learning model is determined after modifying the basic model. The basic model can adopt yoloface. The preset detection model can sequentially output a preset number of first corner points in a specific order. Here, the preset number is related to the preset detection model. For example, the preset detection model outputs 8 first corner points.

[0052] For the target image, the area corresponding to the target space is the background part of the target image, and the area corresponding to the target object is the main body part of the target image; since the target image is any first image in the first image sequence, for each frame of the first image, its background part is the area corresponding to the target space and the main body part is the area corresponding to the target object.

[0053] See Figure 3 As shown, it is a specific example of determining 8 first corner points by performing corner point detection on the target image. In Figure 3 , the target space is a shooting studio, the target object is a car, and the labels corresponding to the 8 first corner points are determined based on the output order of the preset detection model.

[0054] In the above implementation scheme, by adopting the preset detection model to perform corner point detection, the accuracy and efficiency of corner point detection can be improved.

[0055] As an optional embodiment, the method further includes: before performing corner point detection on the target image, obtaining multiple frames of first images captured during the rotation of the target object in the target space, and determining the first image sequence based on the multiple frames of first images; wherein, the multiple frames of first images correspond to the same shooting position, and each frame of the first image corresponds to a rotation state.

[0056] Before determining the target image and performing corner point detection on the target image, obtain multiple frames of first images captured at the same shooting position during the rotation of the target object in the target space, and determine the first image sequence based on the obtained multiple frames of first images. Here, the shooting object is the target object in the target space, or it can also be understood that the shooting object includes the target object and the target space. The target object rotates 360 degrees in the target space, and the target object can be set on a rotating device, and the rotation of the rotating device drives the rotation of the target object, or the target object is connected to a driving device, and the rotation of the target object is controlled based on the driving device.

[0057] The shooting positions corresponding to multiple frames of the first images are the same. That is, when shooting, the shooting device remains stationary. The shooting device can be an independent device. Based on the communication between the terminal device for image processing and the shooting device, the terminal device obtains the first image sequence. The shooting device can also be a shooting component integrated on the terminal device. After the terminal device obtains the first image sequence through the shooting component, it can directly process multiple frames of the first images. The rotation states corresponding to the target object in multiple frames of the first images are different. That is, each frame of the first image corresponds to a rotation state.

[0058] The process of obtaining multiple frames of the first images will be introduced through a specific example below. The target object (car) is located in the target space (shooting studio). A turntable is set on the ground of the target space. The target object is located on the turntable. When the turntable rotates, it drives the target object to rotate 360 degrees. The shooting device remains stationary at a fixed position. The target object is photographed once every 10 degrees of rotation. After the target object completes a 360-degree rotation, 36 frames of the first images can be obtained, and the first image sequence is determined based on the 36 frames of the first images obtained.

[0059] In the above implementation, by shooting to obtain multiple frames of the first images, it is possible to obtain multiple frames of the first images corresponding to different rotation states based on the shooting behavior of the terminal device; by communicating with the shooting device to obtain multiple frames of the first images, it is possible to obtain multiple frames of the first images corresponding to different rotation states based on information interaction.

[0060] The process of determining the second corner points will be introduced below. When determining the preset number of second corner points corresponding to the target background image based on the preset number of first corner points, it includes:

[0061] Based on the preset number of first corner points, perform corner calibration on the target background image to determine the preset number of second corner points corresponding to the target background image;

[0062] Among them, the calibration order of the preset number of second corner points is the same as the calibration order of the preset number of first corner points.

[0063] After determining the preset number of first corner points corresponding to the target image according to the preset detection model, based on the preset number of first corner points, perform corner calibration on the target background image to determine the preset number of second corner points corresponding to the target background image. And when performing corner calibration on the target background image, it is necessary to perform corner calibration on the target background image according to the corner output order of the preset detection model. That is, the calibration order corresponding to the preset number of second corner points is the same as the calibration order corresponding to the preset number of first corner points.

[0064] See Figure 3 and Figure 4 as shown Figure 38 first corner points corresponding to the target image are shown. In Figure 4 the target background image in, 8 second corner points determined based on the 8 first corner points are shown, and the calibration order corresponding to the 8 second corner points is the same as the calibration order corresponding to the 8 first corner points, that is, the calibration labels corresponding to the first corner points and the second corner points match.

[0065] By determining a preset number of second corner points based on a preset number of first corner points, the second corner points can be directly calibrated based on the pre-determined first corner points, simplifying the calibration process of the second corner points.

[0066] The process of determining the set of homography matrices is introduced below. When determining the set of homography matrices according to the preset number of first corner points and the preset number of second corner points, it includes:

[0067] For each of the N regions in the background part of the target image, calculate the homography matrix corresponding to the current region according to the first corner points corresponding to the current region and the second corner points corresponding to the background region in the target background image that matches the current region;

[0068] Determine the set of homography matrices according to the homography matrices corresponding to the N regions respectively;

[0069] Among them, the N regions include the regions corresponding to the first side wall, the second side wall, the top wall, the background wall, and the ground part in the target space in the target image respectively.

[0070] After determining the preset number of first corner points and second corner points, for each of the N regions in the background part of the target image, determine the homography matrix corresponding to the current region. The N regions corresponding to the background part of the target image include the first side wall region corresponding to the first side wall in the target space, the second side wall region corresponding to the second side wall, the top wall region corresponding to the top wall, the background wall region corresponding to the background wall, and the ground region corresponding to the ground part.

[0071] When determining the homography matrix corresponding to any region, calculate the homography matrix corresponding to the current region according to the first corner points corresponding to the current region and the second corner points corresponding to the background region in the target background image that matches the current region. After determining the homography matrices corresponding to the N regions respectively, determine the set of homography matrices according to the N homography matrices.

[0072] The process of calculating the set of homography matrices is introduced below through an example. Refer to Figure 3 and Figure 4 as shown. Through Figure 3 the first corner points (1, 2, 7, 8) in and Figure 4The second corner points (1, 2, 7, 8) in [reference] calculate the homography matrix corresponding to one side wall area (left side wall area). By Figure 3 The first corner points (3, 4, 5, 6) in [reference] and Figure 4 The second corner points (3, 4, 5, 6) in [reference] calculate the homography matrix corresponding to the other side wall area (right side wall area). By Figure 3 The first corner points (1, 2, 3, 4) in [reference] and Figure 4 The second corner points (1, 2, 3, 4) in [reference] calculate the homography matrix corresponding to the top wall area. By Figure 3 The first corner points (2, 3, 6, 7) in [reference] and Figure 4 The second corner points (2, 3, 6, 7) in [reference] calculate the homography matrix corresponding to the background wall area. By Figure 3 The first corner points (5, 6, 7, 8) in [reference] and Figure 4 The second corner points (5, 6, 7, 8) in [reference] calculate the homography matrix corresponding to the ground area. Based on the homography matrices determined above, a set of homography matrices is determined.

[0073] Among them, when calculating the homography matrix corresponding to the current area according to the first corner points corresponding to the current area and the second corner points corresponding to the background area matching the current area in the target background image, it includes:

[0074] Determine the first coordinate set according to the pixel coordinates of the first corner points corresponding to the current area;

[0075] Determine the second coordinate set according to the pixel coordinates of the second corner points corresponding to the background area matching the current area;

[0076] Calculate the homography matrix corresponding to the current area according to the corresponding relationship of each pixel coordinate in the first coordinate set and the second coordinate set.

[0077] When calculating the homography matrix based on corner points for any area, determine the first coordinate set according to the pixel coordinates of the first corner points corresponding to the current area. After determining the background area matching the current area in the target background image, determine the second coordinate set according to the pixel coordinates of the second corner points corresponding to the background area matching the current area. Then, based on the corresponding relationship of each pixel coordinate in the first coordinate set and the second coordinate set, calculate to determine the homography matrix corresponding to the current area.

[0078] In the above implementation process, for each of the N areas, calculate the homography matrix based on the corner pixel coordinates corresponding to the area and the corner pixel coordinates corresponding to the matching background area, and determine a set of homography matrices based on the homography matrices of each area, which can facilitate subsequent homography transformation based on the set of homography matrices.

[0079] The process of obtaining the image set is introduced below. When transforming the target background image to the image perspective of the second image based on the set of homography matrices and obtaining an image set including one or more frames of third images, one of the following solutions is included:

[0080] When the image area corresponding to the ground part of the target space needs to be replaced, for a selected second image, transform the target background image to the image perspective of the current second image based on the set of homography matrices to obtain an image set including one frame of third image;

[0081] When the image area corresponding to the ground part of the target space needs to be retained, for each frame of the second image, transform the target background image to the image perspective of the current second image based on the set of homography matrices to obtain an image set including multiple frames of third images.

[0082] When obtaining the image set through image perspective transformation, it is necessary to determine whether the image area corresponding to the ground part of the target space needs to be replaced. Since the target object is located on the ground part of the target space, and the states of the target object are different, the ground part can be different. For example, the projection of the target object on the ground part is different, and the state of the rotating device on the ground part is different. Therefore, it is necessary to consider whether the area corresponding to the ground part of the target space in the second image needs to be replaced.

[0083] When the image area corresponding to the ground part of the target space needs to be replaced, a second image can be selected from multiple frames of second images. For the selected second image, transform the target background image to the image perspective of the current second image based on the set of homography matrices to obtain the third image corresponding to the target background image under the image perspective of the current second image through image perspective transformation, and obtain an image set including one frame of third image.

[0084] When the image area corresponding to the ground part of the target space needs to be retained, for each frame of the second image, transform the target background image to the image perspective of the current second image based on the set of homography matrices to obtain the third image corresponding to the target background image under the image perspective of the current second image through image perspective transformation. Since each frame of the second image corresponds to a third image, an image set including multiple frames of third images can be obtained. Since the ground areas corresponding to different second images are different, it is necessary to determine the corresponding third image for each frame of the second image.

[0085] In the above embodiments, when obtaining the image set, it is necessary to determine whether the image area corresponding to the ground part of the target space needs to be replaced. In the case where the image area corresponding to the ground part needs to be replaced, an image set including one frame of the third image is determined. In the case where the image area corresponding to the ground part needs to be retained, an image set including multiple frames of the third image is determined, which can realize determining the image set based on actual needs.

[0086] The process of perspective transformation is introduced below. When transforming the target background image to the image perspective of the current second image based on the set of homography matrices, it includes:

[0087] For each area in the current second image, according to the homography matrix corresponding to the current area, perform homography transformation on the matching background area in the target background image to obtain the first background area;

[0088] Generate the third image corresponding to the current second image based on the first background areas respectively matched by at least some of the N areas in the current second image, so as to complete the perspective transformation of the target background image.

[0089] When performing perspective transformation, for each of the N areas corresponding to the current second image, according to the homography matrix corresponding to the current area, perform homography transformation on the background area in the target background image that matches the current area to obtain the first background area. After determining the N first background areas, based on at least some of the N first background areas (that is, based on the first background areas respectively matched by at least some of the N areas in the current second image), generate the third image corresponding to the target background image in the image perspective of the current second image (that is, generate the third image corresponding to the current second image), so as to realize the completion of the perspective transformation of the target background image, making the target background image have the same image perspective as the current second image after perspective transformation.

[0090] Among them, when generating the third image corresponding to the second image, it is necessary to distinguish whether the image area corresponding to the ground part of the target space needs to be replaced. In the case where it needs to be replaced, the generating the third image corresponding to the current second image based on the first background areas respectively matched by at least some of the N areas in the current second image includes:

[0091] Stitch the first background areas respectively matched by the N areas to generate the third image; or;

[0092] Map the current second image according to the first background areas respectively matched by the N areas to generate the third image.

[0093] When the image area corresponding to the ground part needs to be replaced, the third image corresponding to the current second image can be directly generated according to the first background areas respectively matched by the N areas of the current second image. And when generating the third image corresponding to the current second image according to the N first background areas, the N first background areas can be spliced to generate the third image corresponding to the current second image through area splicing; it can also be to paste the map on the current second image according to the N first background areas, and generate the third image corresponding to the current second image through map pasting processing. And when pasting the map, the first background area is attached to the corresponding area in the second image.

[0094] In the case where the image area corresponding to the ground part in the target space needs to be retained, generating the third image corresponding to the current second image according to at least some of the N areas of the current second image respectively matched first background areas includes: pasting the map on the current second image according to the first background areas respectively matched by the first side wall, the second side wall, the top wall and the background wall to generate the third image.

[0095] When the image area corresponding to the ground part needs to be retained, the third image corresponding to the current second image can be generated according to some of the N first background areas. Specifically: paste the map on the current second image according to the first background areas respectively matched by the first side wall, the second side wall, the top wall and the background wall, and generate the third image through map pasting processing. Among them, the first background area matched by the first side wall is the first background area matched by the side wall area corresponding to the first side wall in the current second image.

[0096] The following introduces the process of generating the third image corresponding to the second image based on map pasting processing through an example. See Figure 3 and Figure 4 shown. Based on Figure 3 the homography matrix corresponding to the left wall area in Figure 4 shown, perform homography transformation on the matched background area in the target background image ( Figure 3 shown) to obtain the first background area 1. Based on Figure 4 the homography matrix corresponding to the right wall area in Figure 3 shown, perform homography transformation on the matched background area in the target background image ( Figure 4 shown) to obtain the first background area 2. Based on Figure 3 the homography matrix corresponding to the top wall area in Figure 4Perform a homography transformation on the matched background regions in the figure (as shown), and obtain the first background region 4. Fit the first background region 1, the first background region 2, the first background region 3, and the first background region 4 to the corresponding regions of the current second image. For the first background region 4, when fitting, the matte result needs to be considered. For the regions overlapping with the foreground, the image content is replaced by the target object with the standard background content.

[0097] In the above implementation, when transforming the target background image to the image perspective of the second image based on the set of homography matrices, first determine N first background regions according to the set of homography matrices. When the image region corresponding to the ground part of the target space needs to be replaced, generate the third image corresponding to the second image according to the N first background regions. When the image region corresponding to the ground part of the target space needs to be retained, generate the third image corresponding to the second image according to some of the N first background regions, so as to realize generating the third image by selecting a suitable method based on actual needs.

[0098] The process of generating the target image sequence is introduced below. The method of splicing the third image in the image set with the multiple matte images to obtain the target image sequence includes one of the following solutions:

[0099] When the image set includes one frame of the third image, splice the multiple matte images with the third image respectively to obtain the target image sequence including multiple frames of fourth images;

[0100] When the image set includes multiple frames of the third image, for each frame of the third image, splice the third image with the matched matte image to obtain the target image sequence including multiple frames of fourth images.

[0101] When obtaining the target image sequence according to the image set and the multiple matte images, it is necessary to adopt the corresponding strategy to generate the target image sequence according to the number of frames of the third image in the image set. When the image set includes one frame of the third image, splice the multiple matte images with the third image respectively to obtain the target image sequence including multiple frames of fourth images. Each frame of the fourth image is determined based on the third image and one matte image. Since the multiple matte images are different, the multiple frames of fourth images determined based on the multiple matte images are different.

[0102] When the image set includes multiple frames of the third image, for each frame of the third image, the third image can be spliced with the matched matte image to obtain the target image sequence including multiple frames of fourth images, where the multiple frames of the third image are determined based on the multiple frames of the second image, and each frame of the second image in the multiple frames of the second image corresponds to one matte image. Therefore, the matched matte image can be determined for each frame of the third image.

[0103] When stitching the third image with the matte image, the target position corresponding to the target object can be determined in the third image. According to the matte image matched with the third image, texture mapping is performed at the target position of the third image to generate a fourth image. And before performing the texture mapping, the content at the target position of the third image can be removed, and then the matte image is texture mapped to this position. Refer to Figure 5 As shown, it is a specific example of the fourth image generated based on the third image and the matte image. Figure 5 Adaptive adjustment is performed on the ground area in it.

[0104] Among them, for the case where the image area corresponding to the ground part of the target space needs to be replaced, the image set includes one frame of the third image. At this time, the target image sequence can be generated by repeatedly using the third image; for the case where the image area corresponding to the ground part of the target space needs to be retained, the image set includes multiple frames of the third image. At this time, each frame of the third image and a matched matte image determine the corresponding fourth image.

[0105] For the above implementation solutions, based on the number of the third images in the image set, the corresponding method can be adopted to determine the target image sequence based on the third image and the matte image.

[0106] The above is the overall implementation process of the image processing method provided by the embodiments of this application. By detecting the corner points of the target image, a preset number of first corner points corresponding to the target image are determined. By performing matte processing on the images in the first image sequence, multiple frames of second images with the target object removed and multiple frames of matte images only including the target object are obtained. In the case of determining a preset number of first corner points, a preset number of second corner points are determined based on the first corner points, and the second corner points can be calibrated quickly and accurately based on the pre-determined first corner points; after determining the first corner points and the second corner points, a set of homography matrices is determined according to the first corner points and the second corner points. Based on the set of homography matrices, the target background image is transformed to the image perspective of one frame or multiple frames of the second images, and an image set including one frame or multiple frames of the third image is obtained. The images in the image set and multiple frames of matte images are stitched to obtain the target image sequence. The background of the image can be replaced through image processing. Compared with the solution of replacing the real background, it has the advantages of convenience, low cost, short shooting cycle, high efficiency, and strong feasibility and practicality.

[0107] Furthermore, by using a preset detection model for corner point detection, the accuracy and efficiency of corner point detection can be improved; by determining a preset number of second corner points based on a preset number of first corner points, the second corner points can be directly calibrated based on the pre-determined first corner points, simplifying the calibration process of the second corner points.

[0108] By calculating the homography matrix based on the corner pixel coordinates and determining the set of homography matrices based on multiple homography matrices, it is convenient to perform homography transformation based on the set of homography matrices in the subsequent process; by determining the image set according to the requirement of whether the image area corresponding to the ground part of the target space needs to be replaced, it is possible to determine the image set based on the actual requirement; by determining the target image sequence in a corresponding manner based on the number of the third images in the image set, it is possible to determine the matching strategy based on the number of images to determine the target image sequence.

[0109] In the embodiments of the present application, by fusing multiple algorithms (such as, corner point detection, matting, homography transformation, and image stitching), the background replacement function can be realized through calculation, the background replacement efficiency can be improved, and the cost can be saved.

[0110] The embodiments of the present application further provide an image processing device. Refer to Figure 6 as shown, the device includes:

[0111] A detection and determination module 601, configured to perform corner point detection on a target image to determine a preset number of first corner points corresponding to the background part of the target image, where the target image is an image in a first image sequence, and the first image sequence includes multiple first images obtained by photographing a target object in different states in a target space;

[0112] A processing and acquisition module 602, configured to perform matting processing on each frame of the first image in the first image sequence to obtain multiple second images with the target object removed and multiple matting images including only the target object;

[0113] A determination module 603, configured to determine a preset number of second corner points corresponding to a target background image based on the preset number of first corner points, and determine a set of homography matrices according to the preset number of first corner points and the preset number of second corner points;

[0114] A transformation and acquisition module 604, configured to transform the target background image to the image perspective of the second image based on the set of homography matrices, and obtain an image set including one frame or multiple frames of third images;

[0115] A stitching and acquisition module 605, configured to stitch the third images in the image set with the multiple matting images to obtain a target image sequence.

[0116] Optionally, the detection and determination module is further configured to:

[0117] After selecting the target image in the first image sequence, perform corner point detection on the background part of the target image based on a preset detection model to determine the preset number of first corner points;

[0118] Among them, in the target image, the area corresponding to the target space is the background part, and the area corresponding to the target object is the main part.

[0119] Optionally, the device further includes:

[0120] An acquisition and determination module, configured to, before the detection and determination module performs corner point detection on the target image, acquire multiple first images captured during the rotation of the target object in the target space, and determine the first image sequence based on the multiple first images;

[0121] Among them, the multiple first images correspond to the same shooting position, and each first image corresponds to a rotation state.

[0122] Optionally, the determination module is further configured to:

[0123] Based on the preset number of first corner points, perform corner point calibration on the target background image to determine the preset number of second corner points corresponding to the target background image;

[0124] Among them, the calibration order of the preset number of second corner points is the same as the calibration order of the preset number of first corner points.

[0125] Optionally, the determination module includes:

[0126] A calculation sub-module, configured to, for each of the N regions in the background part of the target image, calculate the homography matrix corresponding to the current region according to the first corner points corresponding to the current region and the second corner points corresponding to the background region in the target background image that matches the current region;

[0127] A determination sub-module, configured to determine a set of homography matrices according to the homography matrices corresponding to the N regions respectively;

[0128] Among them, the N regions include the regions corresponding to the first side wall, the second side wall, the top wall, the background wall, and the ground part in the target space in the target image respectively.

[0129] Optionally, the calculation sub-module includes:

[0130] A first determination unit, configured to determine a first coordinate set according to the pixel coordinates of the first corner points corresponding to the current region;

[0131] A second determination unit, configured to determine a second coordinate set according to the pixel coordinates of the second corner points corresponding to the background region that matches the current region;

[0132] A calculation unit for calculating a homography matrix corresponding to the current area according to the corresponding relationship of each pixel coordinate in the first coordinate set and the second coordinate set.

[0133] Optionally, the transformation acquisition module includes one of the following sub-modules:

[0134] The first transformation acquisition sub-module is used, when the image area corresponding to the ground part of the target space needs to be replaced, for a selected second image, to transform the target background image to the image perspective of the current second image based on the set of homography matrices, so as to obtain an image set including a third image;

[0135] The second transformation acquisition sub-module is used, when the image area corresponding to the ground part of the target space needs to be retained, for each second image, to transform the target background image to the image perspective of the current second image based on the set of homography matrices, so as to obtain an image set including multiple third images.

[0136] Optionally, when the first transformation acquisition sub-module or the second transformation acquisition sub-module transforms the target background image to the image perspective of the current second image based on the set of homography matrices, it includes:

[0137] The transformation acquisition unit is used, for each area in the current second image, to perform a homography transformation on the matching background area in the target background image according to the homography matrix corresponding to the current area, so as to obtain a first background area;

[0138] The generation unit is used to generate a third image corresponding to the current second image according to at least some of the first background areas respectively matched by N areas of the current second image, so as to complete the perspective transformation of the target background image.

[0139] Optionally, when the image area corresponding to the ground part of the target space needs to be replaced, the generation unit includes:

[0140] The first generation sub-unit is used to splice the first background areas respectively matched by the N areas to generate the third image; or;

[0141] The second generation sub-unit is used to perform texture mapping on the current second image according to the first background areas respectively matched by the N areas to generate the third image.

[0142] Optionally, when the image area corresponding to the ground part of the target space needs to be retained, the generation unit is further used to: perform texture mapping on the current second image according to the first background areas respectively matched by the first side wall, the second side wall, the top wall and the background wall to generate the third image.

[0143] Optionally, the splicing acquisition module includes one of the following sub-modules:

[0144] The first splicing acquisition sub-module is configured to, when the image set includes a third image, splice the multiple frame-matted images with the third image respectively to obtain a target image sequence including multiple fourth images;

[0145] The second splicing acquisition sub-module is configured to, when the image set includes multiple third images, for each third image, splice the third image with the matched matted image to obtain a target image sequence including multiple fourth images.

[0146] For the apparatus embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For related parts, please refer to the partial description of the method embodiment.

[0147] On the other hand, an embodiment of the present application further provides an electronic device, including a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the above image processing method are implemented.

[0148] For example, Figure 7 shows a schematic physical structure diagram of an electronic device.

[0149] Such as Figure 7As shown in the figure, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740. Among them, the processor 710, the communications interface 720, and the memory 730 complete mutual communication through the communication bus 740. The processor 710 can call the logical instructions in the memory 730. The processor 710 is used to perform the following steps: Detect the corner points of the target image, and determine a preset number of first corner points corresponding to the background part of the target image. The target image is an image in a first image sequence, and the first image sequence includes multiple first images obtained by photographing a target object in different states in a target space; perform matting processing on each first image in the first image sequence to obtain multiple second images with the target object removed and multiple matted images including only the target object; determine a preset number of second corner points corresponding to the target background image based on the preset number of first corner points, and determine a set of homography matrices according to the preset number of first corner points and the preset number of second corner points; transform the target background image to the image perspective of the second image based on the set of homography matrices to obtain an image set including one or more third images; splice the third images in the image set with the multiple matted images to obtain a target image sequence. The processor 710 is also used to perform other steps in this solution, which will not be further elaborated here.

[0150] In addition, when the logical instructions in the above-mentioned memory 730 can be implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0151] On the other hand, the embodiments of this application also provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps in the image processing method provided in the above-mentioned embodiments.

[0152] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative effort.

[0153] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of each embodiment of the present application.

Claims

1. An image processing method, characterized in that, Including: Performing corner point detection on a target image to determine a preset number of first corner points corresponding to the background part of the target image, where the target image is an image in a first image sequence, and the first image sequence includes multiple first images obtained by photographing a target object in different states in a target space; Performing matte processing on each first image in the first image sequence to obtain multiple second images with the target object removed and multiple matte images including only the target object; Determining a preset number of second corner points corresponding to a target background image based on the preset number of first corner points, and determining a set of homography matrices according to the preset number of first corner points and the preset number of second corner points; Transforming the target background image to the image perspective of the second image based on the set of homography matrices to obtain an image set including one or more third images; Stitching the third images in the image set with the multiple matte images to obtain a target image sequence.

2. The method according to claim 1, wherein The performing corner point detection on the target image to determine a preset number of first corner points corresponding to the background part of the target image includes: After selecting the target image in the first image sequence, performing corner point detection on the background part of the target image based on a preset detection model to determine the preset number of first corner points; Wherein, in the target image, the area corresponding to the target space is the background part, and the area corresponding to the target object is the main body part.

3. The method according to claim 1, wherein Before performing corner point detection on the target image, the method further includes: Obtaining multiple first images obtained by photographing during the rotation of the target object in the target space, and determining the first image sequence based on the multiple first images; Wherein, the multiple first images correspond to the same shooting position, and each first image corresponds to a rotation state.

4. The method according to claim 1, characterized in that The determining a preset number of second corner points corresponding to the target background image based on the preset number of first corner points includes: Performing corner point calibration on the target background image based on the preset number of first corner points to determine the preset number of second corner points corresponding to the target background image; Wherein, the calibration order of the preset number of second corner points is the same as the calibration order of the preset number of first corner points.

5. The method according to claim 1, wherein The determining the set of homography matrices according to the preset number of first corner points and the preset number of second corner points includes: For each of N regions in the background part of the target image, calculating the homography matrix corresponding to the current region according to the first corner point corresponding to the current region and the second corner point corresponding to the background region in the target background image that matches the current region; Determining the set of homography matrices according to the homography matrices corresponding to the N regions respectively; Wherein, the N regions include the regions corresponding to the first side wall, the second side wall, the top wall, the background wall, and the ground part in the target space in the target image respectively.

6. The method according to claim 5, wherein The calculating the homography matrix corresponding to the current region according to the first corner point corresponding to the current region and the second corner point corresponding to the background region in the target background image that matches the current region includes: Determine a first coordinate set according to the pixel coordinates of the first corner point corresponding to the current region; Determine a second coordinate set according to the pixel coordinates of the second corner point corresponding to the background region matching the current region; Calculate the homography matrix corresponding to the current region according to the corresponding relationship of each pixel coordinate in the first coordinate set and the second coordinate set.

7. The method according to claim 5, wherein The obtaining an image set including one or more frames of third images by transforming the target background image to the image perspective of the second image based on the set of homography matrices includes one of the following solutions: When the image region corresponding to the ground part of the target space needs to be replaced, for a selected second image, transform the target background image to the image perspective of the current second image based on the set of homography matrices to obtain an image set including one frame of third image; When the image region corresponding to the ground part of the target space needs to be retained, for each frame of the second image, transform the target background image to the image perspective of the current second image based on the set of homography matrices to obtain an image set including multiple frames of third images.

8. The method according to claim 7, wherein The transforming the target background image to the image perspective of the current second image based on the set of homography matrices includes: For each region in the current second image, perform a homography transformation on the matching background region in the target background image according to the homography matrix corresponding to the current region to obtain a first background region; Generate a third image corresponding to the current second image according to the first background regions respectively matched by at least some of the N regions in the current second image to complete the perspective transformation of the target background image.

9. The method according to claim 8, characterized in that, When the image region corresponding to the ground part of the target space needs to be replaced, the generating a third image corresponding to the current second image according to the first background regions respectively matched by at least some of the N regions in the current second image includes: Stitch the first background regions respectively matched by the N regions to generate the third image; or; Perform texture mapping on the current second image according to the first background regions respectively matched by the N regions to generate the third image.

10. The method according to claim 8, wherein When the image region corresponding to the ground part of the target space needs to be retained, the generating a third image corresponding to the current second image according to the first background regions respectively matched by at least some of the N regions in the current second image includes: Perform texture mapping on the current second image according to the first background regions respectively matched by the first side wall, the second side wall, the top wall and the background wall to generate the third image.

11. The method according to claim 1 or 7, characterized in that, The obtaining a target image sequence by stitching the third image in the image set with the multiple frames of matte images includes one of the following solutions: When the image set includes one frame of third image, stitch the multiple frames of matte images with the third image respectively to obtain a target image sequence including multiple frames of fourth images; In the case where the image set includes multiple frames of third images, for each frame of the third images, the third images are stitched with the matching matte images to obtain a target image sequence including multiple frames of fourth images.

12. An image processing apparatus, characterized in that, Comprising: A detection and determination module, configured to perform corner point detection on a target image to determine a preset number of first corner points corresponding to the background part of the target image, where the target image is an image in a first image sequence, and the first image sequence includes multiple frames of first images obtained by photographing a target object in different states in a target space; A processing and acquisition module, configured to perform matte processing on each frame of the first images in the first image sequence to obtain multiple frames of second images with the target object removed and multiple frames of matte images including only the target object; A determination module, configured to determine a preset number of second corner points corresponding to a target background image based on the preset number of first corner points, and determine a set of homography matrices according to the preset number of first corner points and the preset number of second corner points; A transformation and acquisition module, configured to transform the target background image to the image perspective of the second image based on the set of homography matrices to obtain an image set including one frame or multiple frames of third images; A stitching and acquisition module, configured to stitch the third images in the image set with the multiple frames of matte images to obtain a target image sequence.

13. An electronic device, characterized in that, Comprising a processor, a memory, and a computer program stored on the memory and executable on the processor, where when the computer program is executed by the processor, the steps of the image processing method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 11 are implemented.

Images