Image synthesis processing method and system and computer program product

Abstract

The invention provides an image synthesis processing method and system and a computer program product, and relates to the technical field of computer image processing. The image synthesis processing method comprises the steps of obtaining a target image, distortion data corresponding to the target image and a special effect image used for presenting special effect elements in the target image; performing blurring processing on the special effect image based on the distortion data to obtain a blurred image; and performing merging processing on the blurred image and the target image to obtain a composite image.

Description

Technical Field

[0001] The present invention relates to the field of computer image processing technology, and in particular to an image synthesis processing method, system and computer program product. Background Art

[0002] Visual Effects Compositing (VFX for short) refers to the process of combining different image elements to form one or more composite images. Image elements can be real footage or computer-generated special effects images. When compositing real footage with computer-generated images, if the real footage contains a rotation focus effect, the computer-generated image needs to be blurred to match the real footage.

[0003] Currently, blur processing is performed by manually adding blur effects to computer-generated images through visual effects synthesis software. However, due to different production experiences of users, the processed blur effects are difficult to unify in visual presentation and cannot be streamlined. The accuracy of manually added blur effects cannot be guaranteed, and the production cost is high and the production efficiency is low.

[0004] In view of this, some embodiments of the present specification provide an image synthesis processing method, system and computer program product, which aim to automate blur processing and image synthesis, improve the accuracy of adding blur effects, reduce the production cost of adding blur effects, and improve production efficiency. Summary of the invention

[0005] One or more embodiments of the present specification provide an image synthesis processing method, the method comprising: acquiring a target image, distortion data corresponding to the target image, and a special effect image for presenting special effect elements in the target image; blurring the special effect image based on the distortion data to obtain a blurred image; and merging the blurred image with the target image to obtain a synthesized image.

[0006] According to the method provided in one or more embodiments of the present specification, a special effect image is blurred based on distortion data to obtain a blurred image, including: adding distortion to the special effect image based on the distortion data to obtain a distorted image; blurring the distorted image based on a preset depth of field parameter to obtain a blurred image; and removing the distortion from the blurred image based on the distortion data to obtain a blurred image.

[0007] According to the method provided in one or more embodiments of the present specification, distortion processing is performed on a special effect image based on distortion data to obtain a distorted image, including: generating a distortion map based on the distortion data; performing distortion processing on the special effect image based on the distortion map to obtain a distorted image; wherein the distortion map is used to characterize the mapping relationship of pixels in the special effect image from positions before the distortion is added to positions after the distortion is added.

[0008] According to the method provided in one or more embodiments of the present specification, a blurred image is dedistorted based on distortion data to obtain a blurred image, including: generating a dedistortion map based on the distortion data; dedistorting the blurred image based on the dedistortion map to obtain a blurred image; wherein the dedistortion map is used to characterize the mapping relationship of pixels in the blurred image from the position before the distortion is removed to the position after the distortion is removed.

[0009] According to the method provided in one or more embodiments of the present specification, a distorted image is blurred based on a preset depth of field parameter to obtain a blurred image, including: obtaining a target light spot in a target image; adding the target light spot to the distorted image, and blurring the distorted image based on the preset depth of field parameter and the target light spot to obtain a blurred image.

[0010] According to the method provided by one or more embodiments of the present specification, a target light spot in a target image is obtained, including: extracting the target light spot in the target image based on a position of a special effect element in the special effect image corresponding to the target image.

[0011] According to the method provided by one or more embodiments of the present specification, based on the special effect elements in the special effect image corresponding to the position in the target image, the target light spot is extracted in the target image, including: based on the special effect elements in the special effect image corresponding to the position in the target image, intercepting a target area containing a light spot picture in the target image; and extracting the target light spot in the target area based on brightness information of the target area.

[0012] According to the method provided by one or more embodiments of the present specification, before extracting the target light spot in the target area based on the brightness information of the target area, the method further includes: increasing the brightness information of the target area.

[0013] According to the method provided by one or more embodiments of the present specification, the target image is obtained by photographing with a rotary focus lens; the method further includes: acquiring distortion data based on the rotary focus lens.

[0014] According to the method provided by one or more embodiments of this specification, obtaining distortion data based on a rotary focus lens includes: determining the distortion data based on a grid map photographed by the rotary focus lens; or obtaining the distortion data based on lens parameters of the rotary focus lens.

[0015] According to the method provided by one or more embodiments of this specification, the size of the special effect image is the same as the size of the target image.

[0016] According to the method provided by one or more embodiments of the present specification, the transparency of the pixel area in the special effect image that does not display the special effect element is transparent.

[0017] According to the method provided by one or more embodiments of the present specification, the target image is an image frame in a target video, and the special effect image is an image frame in a special effect video. The method also includes: obtaining a preset number of synthetic images based on a preset number of target images in the target video and a preset number of special effect images in the special effect video, and obtaining a synthetic video based on the preset number of synthetic images.

[0018] One or more embodiments of the present specification also provide an image synthesis processing system, the system comprising: an acquisition module, used to acquire a target image, distortion data corresponding to the target image, and a special effect image for presenting special effect elements in the target image; a blurring processing module, used to blur the special effect image based on the distortion data to obtain a blurred image; and an image synthesis module, used to merge the blurred image with the target image to obtain a synthesized image.

[0019] One or more embodiments of the present specification also provide a computer program product, including a computer program, which, when at least a portion of the computer program is executed by a processor, can implement the image synthesis processing method described in some embodiments of the present specification.

[0020] The beneficial effects that may be brought about by the embodiments of this specification include but are not limited to: (1) by acquiring a target image, distortion data corresponding to the target image, and a special effect image for presenting special effect elements in the target image, and blurring the special effect image based on the distortion data to obtain a blurred image, and further merging the blurred image with the target image to obtain a composite image, thereby automatically performing blurring and composite image processing, improving the accuracy of adding blur effects, reducing the production cost of adding blur effects, and improving production efficiency; (2) by obtaining a preset number of composite images based on a preset number of target images in a target video and a preset number of special effect images in a special effect video, and obtaining a composite video based on the preset number of composite images, thereby automatically synthesizing the target video and the special effect video; (3) by adding distortion processing to the special effect image to obtain a distorted image, and merging the distorted image based on a preset depth of field parameter Blurring is performed to obtain a blurred image. Based on the distortion data, the blurred image is subjected to distortion removal to obtain a blurred image, so that the light spots in the blurred image form a ring-shaped blur effect. The closer the area is to the edge of the image, the more obvious the ring-shaped blur effect is. The shape of the light spots is approximately elliptical, thereby simulating the rotational focus effect. In this way, the blurred image also includes the rotational focus effect. When the blurred image including the rotational focus effect is subsequently synthesized with the target image including the rotational focus effect, the obtained synthesized image can have consistency and coordination in visual presentation, thereby presenting a natural visual effect. (4) By obtaining the target light spot in the target image, the target light spot is added to the distorted image, and the distorted image is blurred based on the preset depth of field parameter and the target light spot, so that the blurred image contains a clearer elliptical light spot, which is closer to the light spot in the real rotational focus effect, thereby obtaining a blurred image that is closer to the rotational focus effect. It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the beneficial effects that may be produced may be any one or a combination of the above, or any other beneficial effects that may be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] This specification will be further explained by way of exemplary embodiments, which will be described in detail by way of the accompanying drawings. The same numbers in the drawings represent the same structures or steps.

[0022] Figure 1 It is an exemplary flow chart of an image synthesis processing method shown in some embodiments of this specification.

[0023] Figure 2 It is a schematic diagram of a target image shown in some embodiments of this specification.

[0024] Figure 3 It is a schematic diagram of a special effect image shown in some embodiments of this specification.

[0025] Figure 4 It is a schematic diagram of a composite image shown in some embodiments of this specification.

[0026] Figure 5 This is an exemplary flow chart of a method for obtaining a blurred image according to some embodiments of this specification.

[0027] Figure 6 It is a schematic diagram of a distorted image according to some embodiments of this specification.

[0028] Figure 7 It is a schematic diagram of a blurred image shown in some embodiments of this specification.

[0029] Figure 8 This is a schematic diagram of a blurred image according to some embodiments of this specification.

[0030] Fig. 9 It is an exemplary flow chart of a blurred image obtaining method shown in some embodiments of this specification.

[0031] Fig.10 It is a schematic diagram of a target area image shown in some embodiments of this specification.

[0032] Fig.11 This is another schematic diagram of a blurred image according to some embodiments of this specification.

[0033] Fig.12 is an exemplary block diagram of an image synthesis processing system according to some embodiments of the present specification. DETAILED DESCRIPTION

[0034] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the embodiments will be described in detail below with reference to the accompanying drawings. Obviously, the contents described below are some examples or embodiments of this specification. For ordinary technicians in this field, without paying creative work, the technical solutions or means disclosed in this specification can also be applied to other scenarios based on these technical contents.

[0035] It should be understood that the "system", "device", "unit" and / or "module" used in this specification is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0036] Unless otherwise specified, technical terms used in this specification to describe components, elements, etc. do not refer to the singular, but may also include the plural. Generally speaking, terms such as "include", "comprise", etc. only indicate that the steps, elements, or components that have been clearly identified are included, and these steps, elements, and components do not constitute an exclusive list, such as the method or device described may also include other steps or components.

[0037] Flowcharts are used in this specification to illustrate the operation steps performed by the device or system of the relevant embodiments, but unless otherwise specified, the order used to describe these steps should not be understood as a limitation on the order in which the steps are performed. A person of ordinary skill in the art can adjust the order in which these steps are performed based on the knowledge and information conveyed by the embodiments of this specification, and the adjustment includes but is not limited to swapping the order, merging multiple steps, and splitting a certain step.

[0038] Visual effects compositing (VFX for short) refers to the process of combining different image elements to form one or more composite images. It is usually used in film and television production, advertising, animation and video games. Visual effects compositing can be used to create various special effects scenes, such as scenes of monsters appearing in the jungle, scenes of falling from cliffs, explosion scenes, etc. The image elements for visual effects compositing can be real filmed materials, computer-generated special effects images, etc. Real filmed materials can be, for example, videos and images shot by cameras, and computer-generated special effects images can be, for example, special effects images generated by 3D animation software.

[0039] When synthesizing the real shot material with the computer-generated image, if the real shot material contains a rotation focus effect, the computer-generated image needs to be blurred to match it with the real shot picture, so that the synthesized image has consistency and coordination in visual presentation and presents a natural visual effect. In some embodiments, the visual effect synthesis can manually add a blur effect to the computer-generated image through the visual effect synthesis software, so that it can match the real shot material containing the rotation focus effect during the visual effect synthesis. For example, in the post-production stage, the computer-generated image is deformed by the deformation tool, and the mask tool and the matte tool are combined to simulate the rotation focus effect. However, due to the different production experience of different users, the processed blur effect is difficult to unify in visual presentation and cannot be streamlined. The accuracy of manually adding the blur effect cannot be guaranteed, and the production cost is high and the production efficiency is low.

[0040] To this end, some embodiments of the present specification propose an image synthesis processing method, which obtains a target image, distortion data corresponding to the target image, and a special effects image for presenting special effects elements in the target image, and blurs the special effects image based on the distortion data to obtain a blurred image, and further merges the blurred image with the target image to obtain a synthesized image, thereby automating the blurring and synthesizing images, improving the accuracy of adding blur effects, reducing the production cost of adding blur effects, and improving production efficiency.

[0041] Figure 1 It is an exemplary flow chart of an image synthesis processing method shown in some embodiments of this specification. Figure 1 The process 100 shown can be executed by a terminal device, for example, it can be implemented by an image synthesis processing system 1200 deployed on a computing device. In some embodiments, the computing device can be a desktop computer, a laptop computer, a mobile phone, a VR device, a tablet computer, a gaming device, etc. In some embodiments, the computing device can be a user-end device, or a server-end device or a cloud device. In some embodiments, part of the process 100 can be executed by a computing device as a user end, and another part can be executed by a computing device as a server end. Figure 1 As shown, process 100 may include the following steps.

[0042] Step 110 , obtaining a target image, distortion data corresponding to the target image, and a special effect image for presenting special effect elements in the target image. In some embodiments, step 110 may be implemented by the obtaining module 1210 .

[0043] In some embodiments, the target image may include a rotational focus effect. The rotational focus effect refers to the effect that when shooting with a larger aperture, the light spot outside the focus and near the edge of the image is compressed into an elliptical image. It was originally caused by defects in the optical structure design, and later became widely loved because of its strong artistic effect and high recognition. In an image with a rotational focus effect, the closer the light spot shape is to the center of the image, the closer it is to a circle, and the closer the light spot shape is to the edge of the image, the closer it is to an ellipse. The light spots in the image together form a ring-shaped blur effect, and the closer the area is to the edge of the image, the more obvious the ring-shaped blur effect is.

[0044] In some embodiments, the target image may be a real-life image material, which may be obtained by shooting with a rotary focus lens. The rotary focus lens is an optical lens that can continuously change the focal length within a certain range to achieve shooting at different viewing angles. The rotary focus lens usually has a relatively large aperture (such as f / 2.8, f / 4, etc.), and an image or video with a rotary focus effect may be shot by adjusting the aperture, focal length, and other parameters of the rotary focus lens.

[0045] Figure 2 is a schematic diagram of a target image according to some embodiments of this specification, such as Figure 2 As shown, the target image may be an image including a rotational focus effect taken by a rotational focus lens, and the light spot of the rotational focus effect may be as shown in FIG. Figure 2 As shown in the dashed box, the closer Figure 2 The closer the spot shape in the center area is to a circle, the closer Figure 2 The closer the spot shape in the edge area is to an ellipse, the more obvious the annular blur effect will be when the spots in the image form a ring-shaped blur effect. Figure 2 As shown, the arrangement direction of the light spots in the upper left corner and the lower right corner of the image is roughly inclined from the lower left to the upper right, and the inclined direction is similar to the inclined direction of the symbol " / ", the arrangement direction of the light spots in the upper right corner and the lower left corner of the image is roughly inclined from the upper left to the lower right, and the inclined direction is similar to the inclined direction of the symbol "\", the arrangement direction of the light spots on the upper and lower sides of the image is roughly horizontal, and the arrangement direction of the light spots on the left and right sides of the image is roughly vertical, and the light spots arranged in various directions together form a ring-shaped blur effect.

[0046] In some embodiments, the special effect image may be an image containing special effect elements. For example, the special effect elements may be three-dimensional models generated by three-dimensional animation software, such as animation characters, objects, props, etc. In some embodiments, the transparency of the pixel area in the special effect image that does not display the special effect elements may be transparent, so that when the special effect image is synthesized with the target image, the pixel area that does not display the special effect elements can present the color value of the corresponding pixel area in the target image, avoiding unnecessary occlusion of the target image. In some embodiments, the color value of the pixel can be represented by three channels of RGB (red, green, and blue), and the transparency can be represented by the Alpha channel, which represents the degree of transparency of the pixel color value. For example, the Alpha channel of the pixel area that does not display the color value can be set to 0, indicating that the transparency of the pixel area is completely transparent.

[0047] Figure 3 is a schematic diagram of a special effect image according to some embodiments of this specification. Figure 3 For example, the special effect element can be an animated character. The Alpha channel of the pixel area where the special effect element is located can be set to 255, indicating that the transparency of the pixel area is opaque and can present a color value; the Alpha channel of the pixel area where no special effect element is displayed can be set to 0, indicating that the transparency of the pixel area is transparent (in Figure 3 It should be understood that Figure 2 The dashed box in Figure 3The pointing lines and explanatory characters in the figure are only identifiers to assist understanding, so as to clarify the annotated objects in the image, and should not be understood as part of the image. In the schematic diagrams of subsequent images, there are also dashed boxes, pointing lines or explanatory characters with similar functions, and their functions are similar to this, so they will not be repeated in the following.

[0048] Distortion refers to changes in shape. In photography or videography, image distortion mainly refers to the phenomenon that the image is distorted or distorted in shape and proportion due to the characteristics of the lens. For example, when shooting with a wide-angle lens, the edges of the image will expand outward and present a barrel-shaped distortion; when shooting with a telephoto lens, the edges of the image will sag inward, causing straight lines to bend inward at the edge of the image and present a pillow shape.

[0049] In some embodiments, the degree of distortion of the image can be characterized by distortion data, and the distortion data can be composed of a series of parameters, which can characterize the degree of distortion of each pixel position in the image. Exemplarily, the distortion data may include radial distortion parameters, tangential distortion parameters, intrinsic parameter matrices (such as focal length, pixel size), extrinsic parameter matrices (such as rotation and translation matrices), etc. Among them, radial distortion is caused by the bending of light radiating outward from the center of the lens. This distortion usually occurs in wide-angle lenses or fisheye lenses, which causes straight lines to appear curved at the edge of the image. The radial distortion parameters may include, for example, radial distortion coefficients k1, k2, k3, etc. Tangential distortion is caused by the lens being not completely parallel to the image sensor plane. The tangential distortion parameters may include, for example, tangential distortion coefficients p1, p2, etc., which are used to characterize the degree of tangential distortion.

[0050] In some embodiments, the distortion data corresponding to the target image may be the distortion data corresponding to the rotary focus lens used to shoot the target image, and thus the distortion data may be acquired based on the rotary focus lens. The following describes how to acquire the distortion data based on the rotary focus lens through two specific solutions.

[0051] In some embodiments, distortion data can be determined based on a grid map captured by a rotary focus lens. The grid map can be, for example, a checkerboard map. For example, a checkerboard map can be obtained by using a rotary focus lens for capturing a target image. When shooting, the entire screen of the checkerboard map can be covered by the checkerboard to improve the accuracy of calculating distortion data. In some embodiments, the grid map can be calibrated based on a camera calibration algorithm to determine distortion data. Among them, the principle of camera calibration is to restore three-dimensional information from a two-dimensional image by establishing a geometric model of the camera. The camera calibration process includes obtaining the camera's intrinsic parameter matrix (such as focal length, pixel size, etc.), extrinsic parameter matrix (such as rotation and translation matrix), radial distortion parameters, tangential distortion parameters, etc. These parameters can help correct image distortion and restore three-dimensional scenes from two-dimensional images. The camera calibration algorithm can include, for example, a linear calibration method, a nonlinear calibration method, a two-step calibration method, and the like. For example, the camera calibration of the checkerboard image can be performed by calling the lens distortion tool in the visual effects synthesis software (such as LensDistortion used in nuke). The user can mark the characteristic lines indicating the straight lines in the checkerboard image, or the lens distortion tool can automatically mark the characteristic lines indicating the straight lines. The computing device calls LensDistortion to perform camera calibration processing on the checkerboard image based on the characteristic lines and obtains the corresponding distortion data.

[0052] In some embodiments, distortion data may also be obtained based on lens parameters of the rotary focus lens. The lens parameters may be lens parameters recorded at the shooting site when shooting the target image. The lens parameters of the rotary focus lens may include, for example, focal length, camera type (such as full frame, APS-C, etc.), model, lens size or specifications, etc. Exemplarily, preset distortion data corresponding to the lens parameters may be searched based on the lens parameters, such as distortion data corresponding to the lens parameters provided by the manufacturer or distortion data corresponding to the lens parameters stored in a related database.

[0053] Step 120 , blurring the special effect image based on the distortion data to obtain a blurred image. In some embodiments, step 120 may be implemented by blurring processing module 1220 .

[0054] In some embodiments, the blurring process may be to blur the special effects image based on the depth of field parameters. In some embodiments, when the target image includes a rotational focus effect, the blurring process of the special effects image must match the rotational focus effect in the target image, so that the special effects image also includes the rotational focus effect. Since the characteristic of the rotational focus effect is that the light spot shape is approximately elliptical and the light spots in the image together form a ring-shaped blur effect, blurring the special effects image directly based on the depth of field parameters will cause the light spots in the blurred image to lack an elliptical shape and fail to present a ring-shaped blur effect, which does not match the rotational focus effect. Therefore, in some embodiments, the special effects image can first be subjected to distortion processing based on the distortion data to obtain a distorted image, and then the distorted image can be blurred based on the preset depth of field parameters to obtain a blurred image, and further, the blurred image can be subjected to dedistortion processing based on the distortion data to obtain a blurred image. By performing distortion and dedistortion operations before and after blur processing, the light spot in the special effect image (i.e., the blurred image) after blur processing can present an elliptical shape, and the light spot in the blurred image presents a ring-shaped blur effect, so that the blurred image includes a rotation focus effect, thereby matching the rotation focus effect in the target image. This part of the content will be described in detail in process 500 later.

[0055] Step 130 , merging the blurred image with the target image to obtain a composite image. In some embodiments, step 130 may be implemented by the image synthesis module 1230 .

[0056] In some embodiments, merging the blurred image with the target image may be superimposing the blurred image with the target image. In some embodiments, the size of the special effect image may be the same as the size of the target image, so that the two images can be directly superimposed when synthesizing the image without further adjusting the position of the special effect element, thereby simplifying the image synthesis process. Figure 4 is a schematic diagram of a composite image according to some embodiments of the present specification. The composite image obtained by merging the blurred image with the target image can be as follows: Figure 4 shown.

[0057] In some embodiments, the target image may be an image frame in the target video, and the special effect image may be an image frame in the special effect video. For example, the target video may include a preset number of target images, and the preset number of target images (also called frame sequences or image sequences) are continuous and arranged in chronological order; the special effect video may include a preset number of special effect images, and the preset number of special effect images (also called frame sequences or image sequences) are continuous and arranged in chronological order.

[0058] In some embodiments, a preset number of composite images can be obtained based on a preset number of target images in a target video and a preset number of special effect images in a special effect video, and a composite video can be obtained based on the preset number of composite images. Exemplarily, the target images in the target video can correspond to the special effect images in the special effect video one by one, and for each target image in the target video, the image synthesis processing method provided in some embodiments of this specification can be used to obtain a composite image, and the preset number of composite images can be processed in a time sequence corresponding to the target image (or special effect image) to obtain a composite video, thereby automatically synthesizing the target video and the special effect video, and the special effect elements in the synthesized video include a rotation and focus effect, so that the composite video has consistency and coordination in visual presentation, presenting a natural visual effect.

[0059] In some embodiments, the image synthesis processing method can be applied to visual effects synthesis software (such as nuke) in the form of a plug-in, and the plug-in can call tools or nodes in the visual effects synthesis software to perform image synthesis processing.

[0060] Figure 5 is an exemplary flow chart of a method for obtaining a blurred image according to some embodiments of this specification. In some embodiments, Figure 5 The process 500 shown may be executed by a computing device, for example, may be implemented by the blur processing module 1220 in the image synthesis processing system 1200 arranged on the computing device. In some embodiments, the process 500 may be a further elaboration of step 120 in the process 100. Figure 5 As shown, process 500 may include the following steps.

[0061] Step 510 , based on the distortion data, a distortion-added image is obtained by performing distortion-added processing on the special effect image. In some embodiments, step 510 may be implemented by the distortion-added unit 1221 in the blur processing module 1220 .

[0062] In some embodiments, distortion processing can be added to the special effect image based on the distortion model and the distortion data. The distortion model can be used to add distortion effects to the image, or to remove distortion effects from the image. The distortion model can be, for example, a radial distortion model, a tangential distortion model, etc. The radial distortion model can be used to perform distortion processing on barrel distortion and pincushion distortion. The characteristic of barrel distortion is that the edge of the image expands outward, causing the image to look like it is shot from a barrel, and the characteristic of pincushion distortion is that the edge of the image is concave inward, causing the image to look like it is compressed. The tangential distortion model is used to process the distortion caused by the lens itself not being parallel to the image plane. Exemplarily, the final position of the pixel in the special effect image can be calculated based on the distortion data, the initial position of the pixel in the special effect image, and the corresponding distortion model, so that the pixel in the special effect image moves from the initial position to the final position, so that the special effect image produces a distortion effect, the special effect element in the special effect image presents a distorted shape, and a distorted image is obtained.

[0063] In order to facilitate the transmission of distortion data, the distortion data can also be stored in the form of a distortion map. In some embodiments, a distortion map can be generated based on the distortion data, and distortion processing can be added to the special effect image based on the distortion map to obtain a distorted image. Exemplarily, the distortion data can be stored in the form of a distortion map by calling a lens distortion tool (such as LensDistortion used in nuke), and the distortion map is used to characterize the mapping relationship of pixels in the special effect image from the position before the distortion is added to the position after the distortion is added. Further, a map mapping tool (such as STmap used in nuke, REmap used in after effects, etc.) is called to add distortion processing to the special effect image based on the mapping relationship in the distortion map, so that the pixel position in the special effect image changes to produce a distortion effect.

[0064] Figure 6 is a schematic diagram of a distorted image according to some embodiments of this specification, by Figure 3 By adding distortion to the special effects image shown in Figure 6 It can be understood that due to the distortion, the special effect elements originally displayed in the picture frame are moved outside the picture frame. Figure 6 As shown in the middle dotted area, the picture within the dotted area is the picture inside the frame, that is, the picture that can be seen, and the picture frame shows one arm and part of the clothes of the special effects character. The picture outside the dotted area is the picture outside the frame, that is, the picture that cannot be seen, and the picture outside the frame shows the rest of the special effects character except one arm and part of the clothes.

[0065] Step 520 , blurring the distorted image based on a preset depth of field parameter to obtain a blurred image. In some embodiments, step 520 may be implemented by the blurring processing unit 1222 in the blurring processing module 1220 .

[0066] In some embodiments, the preset depth of field parameter is used to characterize the degree and range of blurring the distorted image, which may be an adjustable parameter open to user-defined settings. For example, the distorted image may be blurred based on the preset depth of field parameter by calling a defocus blur tool (such as zdefocus or PXF-zdefocus in nuke, or FirschluftLenscare in after effects) to obtain a blurred image.

[0067] Figure 7 is a schematic diagram of a blurred image according to some embodiments of this specification, by Figure 6 The distorted image shown in the figure is blurred to obtain Figure 7 The blurred image shown. Figure 6 Similar to the picture in , due to the distortion, the special effect elements originally displayed in the frame are moved outside the frame. Figure 7 As shown in the dotted area, the image within the dotted area is the image within the frame, and the image outside the dotted area is the image outside the frame. After removing the distortion, the special effect elements can be fully displayed within the frame.

[0068] Step 530 , dedistort the blurred image based on the distortion data to obtain a blurred image. In some embodiments, step 530 may be implemented by the dedistortion unit 1223 in the blur processing module 1220 .

[0069] In some embodiments, the blurred image may be subjected to distortion removal processing based on a distortion model and distortion data. The distortion model may be, for example, a radial distortion model, a tangential distortion model, etc. Exemplarily, the final position of the pixel in the blurred image (i.e., the initial position of the pixel in the special effect image) may be calculated based on the distortion data, the initial position of the pixel in the blurred image (i.e., the final position of the pixel in the special effect image) and the corresponding distortion model, so that the pixel in the blurred image moves from the initial position (i.e., the final position of the pixel in the special effect image) to the final position (i.e., the initial position of the pixel in the special effect image), thereby removing the distortion from the blurred image, so that the shape of the special effect element is restored to the shape before the distortion is added, but the picture of the special effect element includes a rotation and focus effect, thereby obtaining a blurred image including a rotation and focus effect.

[0070] In order to facilitate the transmission of distortion data, the distortion data can also be stored in the form of a dedistortion map. In some embodiments, a dedistortion map can be generated based on the distortion data, and the blurred image can be dedistorted based on the dedistortion map to obtain a blurred image including a rotation focus effect. Exemplarily, the distortion data can be stored in the form of a dedistortion map by calling a lens distortion tool (such as LensDistortion used in nuke), and the dedistortion map is used to characterize the mapping relationship of the pixels in the blurred image from the position before the distortion is removed to the position after the distortion is removed. Further, a map mapping tool (such as STmap used in nuke, REmap used in after effects, etc.) can be called to dedistort the blurred image based on the mapping relationship in the dedistortion map, so that the pixel position in the blurred image changes, thereby removing the distortion effect added in the blurred image. In some embodiments, when the lens distortion tool stores the distortion data in the form of a map, it can simultaneously output a distortion map and a dedistortion map based on the same distortion data.

[0071] Figure 8 is a schematic diagram of a blurred image according to some embodiments of this specification, by Figure 7 The blurred image shown in the figure is dedistorted and the following Figure 8 The blurred image shown, Figure 8 As shown, the light spots in the blurred image together form a ring-shaped blur effect. The closer the area is to the edge of the image, the more obvious the ring-shaped blur effect is, and the shape of the light spots is approximately elliptical, thereby simulating a rotation focus effect.

[0072] By adding distortion processing to the special effects image to obtain a distorted image, blurring the distorted image based on preset depth of field parameters to obtain a blurred image, and removing the distortion of the blurred image based on the distortion data to obtain a blurred image including a rotational focus effect, the light spots in the blurred image together form a ring-shaped blur effect, the closer the area is to the edge of the image, the more obvious the ring-shaped blur effect is, and the shape of the light spot is approximately elliptical, thereby simulating the rotational focus effect, so that the blurred image also includes the rotational focus effect, and when the blurred image including the rotational focus effect is subsequently synthesized with the target image including the rotational focus effect, the obtained synthesized image can have consistency and coordination in visual presentation, thereby presenting a natural visual effect.

[0073] In some embodiments, in order to make the rotation effect in the blurred image more realistic, the distorted image may be blurred based on the target light spot in the target image and a preset depth of field parameter to obtain a blurred image. Fig. 9 It is an exemplary flow chart of a blurred image obtaining method shown in some embodiments of this specification. Fig. 9The process 900 shown can be performed by a computing device. In some embodiments, the process 900 can be performed by Fig.12 The fuzzy processing unit 1222 in the image synthesis processing system 1200 shown in FIG. 9 is implemented in some embodiments. In some embodiments, the process 900 may be a further elaboration of step 520 in the process 500. Fig. 9 As shown, process 900 may include the following steps.

[0074] Step 910: Acquire a target light spot in a target image.

[0075] In some embodiments, the target light spot can be extracted from the target image based on the position of the special effect element in the special effect image corresponding to the target image. When blurring the special effect image, the target light spot is used to indicate the shape of the light spot added to the special effect image. The position of the special effect element corresponding to the target image can be the position where the special effect element is displayed in the target image in the composite image, for example Figure 4 The special effect element is displayed in the lower right corner of the target image. The following describes in detail how to extract the target light spot in the target image.

[0076] In some embodiments, a target area containing a light spot image can be intercepted in the target image based on the position of the special effect element in the special effect image corresponding to the target image. Exemplarily, a region of a preset shape (such as a square or a circle) can be determined in the target image with the position of the special effect element corresponding to the target image as the center and a preset length as the side length or radius. Further, one or more candidate areas containing a light spot image can be identified in the region of the preset shape based on an image recognition algorithm (such as a feature matching algorithm, a target detection algorithm, etc.), and a target area can be determined in one or more candidate areas, for example, the candidate area closest to the special effect element is determined as the target area and intercepted, or the target area is determined in one or more candidate areas based on the brightness information of the light spot and intercepted, or any candidate area is randomly determined as the target area and intercepted. Fig.10 is a schematic diagram of a target area image according to some embodiments of this specification, such as Fig.10 As shown, the target area includes a light spot image.

[0077] In some embodiments, the target area can also be acquired based on a user's capture operation on the target image. For example, when the user captures a target area containing a light spot image in the target image using a capture tool, the computing device can acquire the target area.

[0078] In some embodiments, after determining the target area, the target light spot can be extracted from the target area based on the brightness information of the target area. Exemplarily, the image of the target area can be converted into a grayscale image to eliminate the interference of color information in the target area, and the light spot pattern in the target area can be extracted according to a preset brightness threshold, so as to remove the background in the target area and obtain an image with only the light spot pattern retained as the target light spot.

[0079] In some embodiments, in order to extract a clearer target light spot, before extracting the target light spot in the target area based on the brightness information of the target area, the brightness information of the target area may be increased. For example, the contrast of the target area is increased so that the light spot pattern in the target area is brighter and the area without the light spot pattern is darker.

[0080] In some embodiments, step 910 can be executed at any time before executing step 920. For example, step 920 can be executed before / after step 510, or simultaneously with step 510. For another example, step 920 can be executed before / after step 110, or simultaneously with step 110.

[0081] Step 920: Add the target light spot to the distorted image, and blur the distorted image based on the preset depth of field parameter and the target light spot to obtain a blurred image.

[0082] In some embodiments, the preset depth of field parameter is used to characterize the degree and range of blurring the distorted image, which may be an adjustable parameter open to user-defined settings. Exemplarily, the target spot can be read by calling a defocus blur tool (such as zdefocus or PXF-zdefocus used in nuke, FirschluftLenscare used in after effects), and the target spot distribution is displayed in the distorted image according to the pixel information in the distorted image and the shape of the target spot, and the distorted image with the target spot distributed is blurred based on the preset depth of field parameter to obtain a blurred image.

[0083] By acquiring the target light spot in the target image, adding the target light spot to the distorted image, and blurring the distorted image based on the preset depth of field parameters and the target light spot, the blurred image contains a clearer elliptical light spot, which is closer to the light spot shape in the real rotation focus effect, thereby obtaining a blurred image that is closer to the rotation focus effect.

[0084] In some embodiments, after step 920 , step 530 may be further executed.

[0085] Fig.11is another schematic diagram of a blurred image according to some embodiments of this specification. For example, a target light spot may be added to Figure 6 In the distorted image shown in FIG. 1 , the distorted image is blurred based on the preset depth of field parameter and the target light spot to obtain a blurred image, and then the blurred image is dedistorted based on the distortion data to obtain a Fig.11 The blurred image shown. Fig.11 As shown in the figure, the blurred image contains a clearer elliptical light spot. Figure 8 The blurred image without the target spot is shown in the figure. Fig.11 The shape of the light spot in the image is closer to the shape of the light spot in the real focus effect, making the blurred image present a more natural focus effect.

[0086] This specification also provides an image synthesis processing system. Fig.12 is an exemplary block diagram of an image synthesis processing system according to some embodiments of this specification. Fig.12 As shown, in some embodiments, the image synthesis processing system 1200 may include an acquisition module 1210 , a blur processing module 1220 , and an image synthesis module 1230 .

[0087] The acquisition module 1210 is used to acquire a target image, distortion data corresponding to the target image, and a special effect image for presenting a special effect element in the target image.

[0088] The blur processing module 1220 is used to perform blur processing on the special effect image based on the distortion data to obtain a blurred image.

[0089] The image synthesis module 1230 is used to combine the blurred image with the target image to obtain a synthesized image.

[0090] In some optional embodiments, the blur processing module 1220 may also include a distortion adding unit 1221, a blur processing unit 1222 and a dedistortion unit 1223; the distortion adding unit 1221 may be used to perform distortion processing on the special effect image based on the distortion data to obtain a distorted image; the blur processing unit 1222 may be used to perform blur processing on the distorted image based on a preset depth of field parameter to obtain a blurred image; the dedistortion unit 1223 may be used to perform dedistortion processing on the blurred image based on the distortion data to obtain a blurred image.

[0091] In some optional embodiments, the distortion unit 1221 can also be used to generate a distortion map based on the distortion data; perform distortion processing on the special effect image based on the distortion map to obtain a distorted image; wherein the distortion map is used to characterize the mapping relationship between pixels in the special effect image from the position before the distortion is added to the position after the distortion is added.

[0092] In some optional embodiments, the dedistortion unit 1223 can also be used to generate a dedistortion map based on the distortion data; dedistort the blurred image based on the dedistortion map to obtain a blurred image; wherein the dedistortion map is used to characterize the mapping relationship between pixels in the blurred image from the position before the distortion is removed to the position after the distortion is removed.

[0093] In some optional embodiments, the blur processing unit 1222 can also be used to obtain a target light spot in a target image; add the target light spot to the distorted image, and blur the distorted image based on a preset depth of field parameter and the target light spot to obtain a blurred image.

[0094] In some optional embodiments, the blur processing unit 1222 may also be used to extract a target light spot in the target image based on a position of a special effect element in the special effect image corresponding to the position in the target image.

[0095] In some optional embodiments, the blur processing unit 1222 can also be used to capture a target area containing a light spot image in the target image based on the position of the special effect element in the special effect image corresponding to the target image; and extract the target light spot in the target area based on the brightness information of the target area.

[0096] In some optional embodiments, the blur processing unit 1222 may also be configured to increase the brightness information of the target area before extracting the target light spot in the target area based on the brightness information of the target area.

[0097] In some optional embodiments, the image synthesis processing system 1200 may further include a distortion data acquisition module 1240 for acquiring distortion data based on a rotational focus lens.

[0098] In some optional embodiments, the distortion data acquisition module 1240 may also be used to determine the distortion data based on a grid image captured by the rotary focus lens; or to acquire the distortion data based on lens parameters of the rotary focus lens.

[0099] In some optional embodiments, the image synthesis processing system 1200 may further include a video synthesis module 1250, which is used to obtain a preset number of synthesized images based on a preset number of target images in the target video and a preset number of special effect images in the special effect video, and to obtain a synthesized video based on the preset number of synthesized images.

[0100] For more information about each module, see Figures 1 to 11 The relevant description of will not be repeated here. It should be understood that Fig.12The system and its modules shown can be implemented in various ways. For example, in some embodiments, the system and its modules can be implemented by hardware, software, or a combination of software and hardware. Among them, the hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. Those skilled in the art will understand that the above methods and systems can be implemented using computer executable instructions and / or control codes contained in a processor, such as a disk, CD or DVD-ROM, etc., and such codes are provided in a carrier medium or a memory of a programmable device. The system and its modules of this specification can not only be implemented by hardware circuits such as ultra-large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., but can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above hardware circuits and software (e.g., firmware).

[0101] It should be noted that the above description of the system and its modules is only for convenience of description and does not limit this specification to the scope of the embodiments. It is understandable that for those skilled in the art, after understanding the principle of the system, it is possible to arbitrarily combine the modules without deviating from this principle to form a subsystem connected to other modules. Or some modules may be split to obtain more modules or multiple units under the module. Such variations are all within the scope of the disclosure of this specification.

[0102] Some embodiments of the present specification also provide a computer program product, including computer instructions. When at least part of the computer instructions are executed by a processor, the present specification can be implemented. Figures 1 to 11 In some embodiments, the computer program product may only involve computer instructions, which may be carried by a storage medium or a processing device. In other embodiments, the computer program product may also be a storage medium or a processing device containing the aforementioned computer instructions. The processing device may include one or more processors and a storage medium.

[0103] In some embodiments, the processor may be a combination of one or more of the following processors: a central processing unit (CPU), an application specific integrated circuit (ASIC), an application specific instruction set processor (ASIP), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a programmable logic controller (PLC), a reduced instruction set computer (RISC), and a microprocessor.

[0104] In some embodiments, the storage medium may include a combination of one or more of the following: a mass storage, a removable storage, a volatile read-write memory, and a read-only memory (ROM). Exemplary mass storage may include a magnetic disk, an optical disk, a solid-state hard disk, and the like. Exemplary removable storage may include a flash disk, a floppy disk, an optical disk, a memory card, a compressed hard disk, a magnetic tape, and the like. Exemplary volatile read-write memory may include a random access memory (RAM). Exemplary random access memory may include a dynamic random access memory (DRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), a static random access memory (SRAM), a thyristor random access memory (T-RAM), and a zero capacitance memory (Z-RAM), and the like. Exemplary read-only memory may include a masked read-only memory (MROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a compact disk read-only memory (CD-ROM), and a digital multifunction hard disk read-only memory, and the like.

[0105] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are taught in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

Claims

1. An image synthesis processing method, characterized in that: The method comprises: Acquire a target image, distortion data corresponding to the target image, and a special effect image for presenting a special effect element in the target image; Performing blur processing on the special effect image based on the distortion data to obtain a blur image; The blurred image is combined with the target image to obtain a composite image.

2. The method according to claim 1, characterized in that The blurring the special effect image based on the distortion data to obtain a blurred image includes: Adding distortion processing to the special effect image based on the distortion data to obtain a distorted image; Blurring the distorted image based on a preset depth of field parameter to obtain a blurred image; The blurred image is subjected to distortion removal processing based on the distortion data to obtain the blurred image.

3. The method according to claim 2, characterized in that The adding distortion processing to the special effect image based on the distortion data to obtain the distorted image includes: generating a distortion map based on the distortion data; Adding distortion to the special effect image based on the distortion map to obtain the distorted image; The distortion map is used to represent the mapping relationship between the position of the pixels in the special effect image before the distortion is added and the position after the distortion is added.

4. The method according to claim 2, characterized in that: The step of performing a distortion removal process on the blurred image based on the distortion data to obtain the blurred image includes: generating a dedistortion map based on the distortion data; Performing a distortion removal process on the blurred image based on the distortion removal map to obtain the blurred image; The dedistortion map is used to represent the mapping relationship between the position of the pixels in the blurred image before the distortion is removed and the position after the distortion is removed.

5. The method according to claim 2, characterized in that: The blurring the distorted image based on the preset depth of field parameter to obtain a blurred image includes: Acquire a target light spot in the target image; The target light spot is added to the distorted image, and blurring is performed on the distorted image based on the preset depth of field parameter and the target light spot to obtain the blurred image.

6. The method according to claim 5, characterized in that The step of acquiring a target light spot in the target image comprises: The target light spot is extracted from the target image based on a position in which the special effect element in the special effect image corresponds to the target image.

7. The method according to claim 6, characterized in that The extracting the target light spot in the target image based on the position of the special effect element in the special effect image corresponding to the position in the target image comprises: Based on the position of the special effect element in the special effect image corresponding to the position in the target image, intercepting a target area containing a light spot picture in the target image; The target light spot is extracted in the target area based on brightness information of the target area.

8. The method according to claim 7, characterized in that Before extracting the target light spot in the target area based on the brightness information of the target area, the method further includes: The brightness information of the target area is increased.

9. The method according to claim 1, characterized in that: The target image is obtained by shooting with a rotary focus lens; The method further comprises: The distortion data is acquired based on the rotary focus lens.

10. The method according to claim 9, characterized in that The obtaining the distortion data based on the rotary focus lens includes: Determining the distortion data based on a grid map captured by the rotary focus lens; or The distortion data is acquired based on lens parameters of the rotary focus lens.

11. The method according to claim 1, characterized in that: The size of the special effect image is the same as the size of the target image.

12. The method according to claim 1, characterized in that The transparency of the pixel area in the special effect image that does not display the special effect element is transparent.

13. The method according to claim 1, characterized in that The target image is an image frame in the target video, the special effect image is an image frame in the special effect video, and the method further includes: The preset number of composite images is obtained based on the preset number of target images in the target video and the preset number of special effect images in the special effect video, and the composite video is obtained based on the preset number of composite images.

14. An image synthesis processing system, characterized in that: The system comprises: An acquisition module, used for acquiring a target image, distortion data corresponding to the target image, and a special effect image for presenting a special effect element in the target image; A blur processing module, used for blurring the special effect image based on the distortion data to obtain a blurred image; The image synthesis module is used to merge the blurred image with the target image to obtain a synthesized image.

15. A computer program product, characterized in that The invention comprises a computer program, and when at least a part of the computer program is executed by a processor, the method according to any one of claims 1 to 13 can be implemented.

Images