Special effects image processing method, device, electronic device and storage medium

Abstract

The embodiments of the present disclosure provide a special effect image processing method, device, electronic device and storage medium, the method comprising: determining a fusion model of the special effect to be processed according to the special effect properties of the target special effect to be superimposed; upon receiving a target special effect display instruction, determining a target special effect fusion model according to the special effect fusion model to be processed and a human body segmentation area corresponding to the target object in the image to be processed; writing pixel depth information corresponding to the target special effect fusion model into a rendering engine, so that the rendering engine renders a target image corresponding to the image to be processed based on the pixel depth information; wherein the target image includes the target special effect. The technical solution of the embodiments of the present disclosure improves the accuracy of the fusion of special effects with specific areas in an image, making the final special effect image more realistic and enhancing the user experience.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image processing technology, and in particular to a special effect image processing method, device, electronic device, and storage medium. Background Art

[0002] With the continuous development of image processing technology, the special effects provided by application software to users are becoming more and more abundant. For example, after the user uses the terminal device to take an image, the image can be processed based on the built-in functions of the application to obtain the corresponding special effect image.

[0003] However, in the solutions provided by the existing technology, when the special effects provided by the application are added to certain specific areas in the image (such as certain areas of the human body), the special effects may not be accurately matched with the picture in that area, and the image processing effect needs to be improved; at the same time, the special effects and the picture content may also be interpenetrated, which results in the final generated special effects image being not realistic enough and the user experience being poor when using the special effects. Summary of the Invention

[0004] The present disclosure provides a special effects image processing method, device, electronic device and storage medium, which improve the accuracy of the fusion of special effects with specific areas in an image, make the final special effects image more realistic, and enhance the user experience.

[0005] In a first aspect, an embodiment of the present disclosure provides a special effects image processing method, comprising:

[0006] Determine the special effect fusion model to be processed according to the special effect properties of the target special effect to be superimposed;

[0007] Upon receiving a target special effect display instruction, determining a target special effect fusion model according to the special effect fusion model to be processed and a human body segmentation region corresponding to the target object in the image to be processed;

[0008] The pixel depth information corresponding to the target special effect fusion model is written into the rendering engine, so that the rendering engine renders a target image corresponding to the image to be processed based on the pixel depth information; wherein the target image includes the target special effect.

[0009] In a second aspect, the embodiments of the present disclosure further provide a special effects image processing device, comprising:

[0010] A module for determining a special effect fusion model to be processed, for determining a special effect fusion model to be processed according to special effect properties of a target special effect to be superimposed;

[0011] a target special effect fusion model determination module, configured to determine the target special effect fusion model based on the special effect fusion model to be processed and the human body segmentation area corresponding to the target object in the image to be processed upon receiving a target special effect display instruction;

[0012] A rendering module is used to write the pixel depth information corresponding to the target special effect fusion model into a rendering engine, so that the rendering engine renders a target image corresponding to the image to be processed based on the pixel depth information; wherein the target image includes the target special effect.

[0013] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0014] one or more processors;

[0015] a storage device for storing one or more programs,

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the special effect image processing method as described in any one of the embodiments of the present disclosure.

[0017] In a fourth aspect, an embodiment of the present disclosure further provides a storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to execute the special effects image processing method as described in any one of the embodiments of the present disclosure.

[0018] The technical solution of the embodiment of the present disclosure determines a fusion model of the special effect to be processed according to the special effect properties of the target special effect to be superimposed, that is, determines a model to be fused corresponding to a specific area in the image, and when a target special effect display instruction is received, determines the target special effect fusion model according to the special effect fusion model to be processed and the human body segmentation area corresponding to the target object in the image to be processed, thereby realizing the fusion of the special effect with the picture content of the area, and writing the pixel depth information corresponding to the target special effect fusion model into the rendering engine, so that the rendering engine renders the target image corresponding to the image to be processed based on the pixel depth information, thereby improving the accuracy of the fusion of the special effect with the specific area in the image, and at the same time, avoiding the occurrence of model penetration between the special effect and the picture content, making the final special effect image more realistic, and enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0020] Figure 1 A flowchart of a special effects image processing method provided in the first embodiment of the present disclosure;

[0021] Figure 2 This is a schematic diagram of the special effects fusion model to be processed provided in the first embodiment of the present disclosure;

[0022] Figure 3 This is a schematic diagram of the target reference axis provided in the first embodiment of the present disclosure controlling the joint movement of the human body segmentation region and the special effect fusion model to be processed;

[0023] Figure 4 A schematic structural diagram of a special effects image processing device provided in the second embodiment of the present disclosure;

[0024] Figure 5 This is a structural diagram of an electronic device provided in Example 3 of the present disclosure. DETAILED DESCRIPTION

[0025] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0026] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0027] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0028] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. It should be noted that the modifications of "one" and "a plurality of" mentioned in this disclosure are illustrative and not restrictive. Those skilled in the art should understand that unless the context clearly indicates otherwise, they should be understood as "one or more".

[0029] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0030] Before introducing the present technical solution, an example of the application scenario of the embodiment of the present disclosure can be first explained. For example, after a user uses the camera device on the mobile terminal to capture an image, the image can be imported into the application, and special effects can be added to the image based on the built-in functions of the application. At this time, the added special effects may not accurately match the content of the picture. At the same time, when some dynamic models are preset in the special effects, these dynamic models may also appear to be interpenetrating with the content in the picture, and the effect of the processed special effects image is poor. At this time, according to the technical solution of this embodiment, a special effects fusion model to be processed can be determined in advance, and then the human body segmentation area in the picture can be determined, so as to fuse the two to obtain a target special effects fusion model, and finally the depth information of the pixel points corresponding to the model is written into the rendering engine to obtain the target image, thereby effectively avoiding the situation where the special effects cannot accurately match the content of the picture and are easy to interpenetrate.

[0031] Example 1

[0032] Figure 1 This is a flow chart of a special effects image processing method provided in the first embodiment of the present disclosure. The embodiment of the present disclosure is applicable to situations where special effects are matched with images with high accuracy to obtain special effects images. The method can be performed by a special effects image processing device, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, PC or server, etc.

[0033] like Figure 1 As shown, the method includes:

[0034] S110: Determine a special effect fusion model to be processed according to special effect properties of the target special effect to be superimposed.

[0035] Among them, the device for executing the special effect image processing method provided by the embodiment of the present disclosure can be integrated into the application software that supports the special effect image processing function, and the software can be installed in the electronic device. Optionally, the electronic device can be a mobile terminal or a PC terminal, etc. The application software can be a type of software for image / video processing. The specific application software will not be described here one by one, as long as the image / video processing can be achieved. It can also be a specially developed application program to implement the software for adding special effects and displaying the special effects, or it can be integrated into the corresponding page, and the user can process the special effect video through the page integrated in the PC terminal.

[0036] It should be noted that the technical solution of this embodiment can be executed based on existing images (i.e., images that the user actively imports into the application) or on images taken by the user in real time. That is to say, after determining the image to be processed and the target special effect selected by the user in the application, the target special effect can be fused with the image to be processed based on the solution of this embodiment to obtain the special effect image desired by the user.

[0037] In the present embodiment, the target special effect to be superimposed can be a special effect developed in advance and integrated into the application. For example, the target special effect can be a dynamic fish tank special effect that can be added to the image, and a plurality of dynamic goldfish models are also present in the special effect. Further, the thumbnail corresponding to the target special effect can be associated with a pre-developed control. When it is detected that the user triggers the control, it indicates that the user wishes to add the special effect to the image. At this time, the application needs to retrieve the data associated with the special effect and fuse the special effect with the corresponding picture in the image according to the solution of the present embodiment, so as to obtain a target image containing the special effect. Taking the above-mentioned fish tank image as an example, in the obtained special effect image, the fish tank is fused with the user's head area, thereby presenting the visual effect of the fish tank being put on the user's head in a more interesting form. At the same time, a plurality of goldfish will also be displayed near the user's head in the fish tank.

[0038] Those skilled in the art should understand that, in actual application, the target special effect can be integrated with one image or with pictures in multiple video frames. In this case, the multiple special effect images finally obtained can present a dynamic visual effect. At the same time, the style of the target special effect is not limited to the fish tank in the above example, but can also be a variety of interesting anthropomorphic special effects, such as floating balloons, etc. The embodiments of the present disclosure do not make specific limitations here.

[0039] In this embodiment, after the user selects the corresponding target special effect for a certain image, it is first necessary to determine the corresponding special effect fusion model to be processed based on the special effect properties of the special effect. Among them, the special effect fusion model to be processed can be a pre-developed three-dimensional model. It can be understood that after the special effect fusion model to be processed is developed, it needs to be associated with the target special effect, so that when it is detected that the user triggers the control associated with the target special effect, the model is retrieved based on the application software and fused with the specific area in the picture. Taking the above-mentioned fish tank special effect as an example, the special effect fusion model to be processed can be a fish tank model pre-developed in three-dimensional space. At the same time, the model is also associated with multiple goldfish-style sub-models. When it is detected that the user triggers the control corresponding to the fish tank special effect, the application can call the above-mentioned model and execute the subsequent processing solution.

[0040] Accordingly, the special effect attributes can be parameters that determine the target special effect display shape and special effect display location. It can be understood that these attributes also determine the style of the special effect model to be processed and directly determine the visual effect of the final special effect image. Optionally, based on the special effect display shape and special effect display location of the target special effect, a special effect fusion model corresponding to the target special effect is determined as the special effect fusion model to be processed.

[0041] Among them, the special effects fusion model is a pre-developed model associated with the target special effects, and is also the special effects fusion model to be processed. The display shape is the information that determines the shape of the model, and the display part is the information that determines which part of the picture the model is fused with.

[0042] Continuing with the above example, when the target special effect is a fish tank special effect, the application can determine that the display shape of the special effect is an ellipse. At the same time, it can determine that the display part of the fish tank model is the user's head area. On this basis, the application can call the previously developed fish tank model as the special effect fusion model to be processed. It can be understood that the fish tank model needs to be integrated with the picture corresponding to the user's head in the subsequent process, so that the final special effect image presents the visual effect of an elliptical fish tank on the user's head.

[0043] Those skilled in the art should understand that in actual application, there may be multiple special effect display shapes and special effect display locations of the target special effect. It can be understood that, in addition to the ellipse in the above example, the special effect display shape may also be a triangle or a rectangle, etc. Correspondingly, the display location may also be the user's arm or the user's body, etc. The embodiments of the present disclosure do not make specific limitations here.

[0044] S120. When a target special effect display instruction is received, a target special effect fusion model is determined according to the special effect fusion model to be processed and a human body segmentation region corresponding to the target object in the image to be processed.

[0045] The target special effect display instruction can be generated based on a user trigger operation, or it can be automatically generated when an image is detected to meet preset conditions. For example, a special effect image processing control can be pre-developed in the application. When a user triggers the control, the application can call the pre-written program and perform the special effect image processing operation. Alternatively, when an image containing the user's body is detected in the display interface, a target special effect display instruction is automatically generated. After receiving the instruction, the application can perform the special effect image processing operation.

[0046] In this embodiment, when receiving the target special effect display instruction, the special effect fusion model to be processed can also be displayed. Specifically, the application can transparently display the paper model in the special effect fusion model to be processed according to the target special effect display instruction.

[0047] In this embodiment, the special effects model to be processed can be composed of multiple parts. For example, a paper model can be included in the special effects model to be processed, wherein the paper model can be the part that needs to correspond to a specific area in the picture; at the same time, the model can be pre-built by the staff based on relevant image processing applications (such as non-linear special effects production software). When the user triggers the fish tank special effect, the application can retrieve the pre-built paper model and write the depth information of the image taken by the user into the paper model. Finally, the paper model is rendered to the corresponding display interface. In this embodiment, when the captured image is displayed based on the paper model, other models involved in the fish tank special effect (such as goldfish) can be shielded. At the same time, it also avoids the problem of other models easily "piercing the model" with the captured image when the captured image is modeled separately.

[0048] by Figure 2 For example, for the fish tank special effect, a circular paper model can be constructed in three-dimensional space for the special effect in advance. It can be understood that the paper model needs to be integrated with the image of the user's head area in the subsequent process; further, a fish tank model is created outside the paper model to surround the paper model, and after the relative position of the above-mentioned paper model and the fish tank model is fixed, it is associated with the target special effect. Based on this, when it is detected that the control corresponding to the target special effect is triggered, the application can call and display the above-mentioned paper model and the fish tank model. In the actual application process, in order to make the final special effect image more realistic, it is also necessary to set parameters such as transparency for the paper model of the fish tank special effect, so that when the special effect model corresponding to the fish tank special effect is displayed on the display interface, only the fish tank model outside the paper model is displayed, and the paper model corresponding to the image of the user's head area is hidden, thereby avoiding the influence of the paper model on the special effect image.

[0049] In this embodiment, the advantage of setting a paper model in the special effects image to be processed is that, during the process of generating the special effects image, it is convenient for the application to locate and correspond the special effects to a specific area in the picture, thereby avoiding the problem of poor matching between the special effects and the picture; at the same time, after adopting the paper model, there is no need to construct a 3D model for the area corresponding to the paper model during the process of generating the special effects image, which indirectly improves the special effects image processing capability of the application.

[0050] In this embodiment, when the application receives the target special effect display instruction, it is necessary to determine the human body segmentation area corresponding to the target object in the image to be processed; and then bind the human body segmentation area to the special effect fusion model to be processed to obtain the target special effect fusion model.

[0051] Among them, the image to be processed can be an image taken in real time by the user through the camera device on the mobile terminal, or it can be an image actively uploaded to the application. Furthermore, after a user is pre-determined as the target object in the application, the image to be processed can also include part or all of the user's body. Of course, in actual application, the target object can be one or more specific users or any user. For the second case mentioned above, it can be understood that when a user's body image is detected in the picture, the user is determined as the target user.

[0052] Accordingly, the human body segmentation region of the target object is any part of the torso segmentation region, for example, the region corresponding to the target user's head and the region corresponding to the target user's arms in the displayed image. It is understood that in actual application, one or more human body segmentation regions can be determined as needed, for example, only the region corresponding to the user's head can be used as the human body segmentation region, or the region corresponding to the user's head, the region corresponding to the arms, and the region corresponding to the legs can all be used as human body segmentation regions. This embodiment of the present disclosure does not specifically limit this.

[0053] In the specific process of determining the human body segmentation model, the human body segmentation area in the image to be processed can be determined based on the human body segmentation algorithm or the human body segmentation model. Among them, the human body segmentation algorithm or the human body segmentation model can be a pre-trained neural network model integrated into the application, which is at least used to segment the picture corresponding to the user's body in the image to be processed, so as to determine the human body segmentation area in the image to be processed. It can be understood that the input of the above model is the image to be processed containing part or all of the user's body picture, and the output is the human body segmentation area corresponding to the image to be processed. Those skilled in the art should understand that for the human body segmentation algorithm or the human body segmentation model, it can be trained based on the corresponding training set and verification set before being integrated into the application. When the loss function of the algorithm or model converges, it indicates that the model training is completed and can be deployed in the application. The specific training process will not be repeated in this embodiment.

[0054] Furthermore, once the human body segmentation region of the target object is determined in the image to be processed, the region can be bound to the special effects fusion model to be processed, thereby obtaining the target special effects fusion model. Optionally, the human body segmentation region and the special effects fusion model to be processed are bound to obtain the special effects fusion model to be used; then, the human body segmentation region and the special effects fusion model to be used are intersected to determine the target special effects fusion model.

[0055] Specifically, in the process of determining the special effect fusion model to be used, the target reference axis corresponding to the target object can be determined first; the target reference axis is used to control the joint movement of the human body segmentation area and the special effect fusion model to be processed to obtain the special effect fusion model to be used. The target reference axis of the target object can be an axis corresponding to the user's body in a pre-constructed three-dimensional space coordinate system. Figure 3 For example, after the application constructs a three-dimensional space coordinate system based on the image in the display interface, the y-axis corresponding to the user's head area in the coordinate system can be used as the target reference axis. On this basis, when the user's head in the display interface tilts, the y-axis in the three-dimensional space will also undergo adaptive changes.

[0056] Furthermore, the intersection processing is performed on the human body segmentation area and the special effect fusion model to be used. Specifically, the area corresponding to the special effect model to be used is first determined, and then the common area between the area and the human body segmentation area is further determined, and then the data corresponding to the two areas are associated. In actual application, the above processing process is the process of associating the picture of the user's head area with the special effect fusion model to be used. It can be understood that by associating the y-axis with the special effect fusion model to be processed, the binding of the human body segmentation area and the special effect fusion model to be processed can be achieved. On this basis, when the user's head rotates, that is, when the position or orientation of the human body segmentation area changes, the target reference axis can control the special effect fusion model to be processed and the human body segmentation area to move together. Continue with Figure 3 Taking an example to illustrate, when the display interface shows a picture of the user looking up, that is, when the position and orientation of the user's head area changes, the y-axis in the three-dimensional space will also adaptively change with the change of the area. At the same time, the y-axis will drive the position and orientation of the special effect fusion model to be processed to change, that is, the paper model in the special effect fusion model to be processed is adjusted to face the direction of the virtual camera, so that when the application processes multiple special effect images, the model and the human body segmentation area are linked.

[0057] In this embodiment, the target reference axis is used as the intermediate correlation part, and the advantage of binding the human body segmentation area with the special effect fusion effect to be processed is that when the application continuously processes multiple images and generates corresponding target images, the special effects finally presented in each image will always follow the movement of the specific part of the user's body, thereby presenting a better dynamic visual effect.

[0058] Furthermore, after obtaining the desired special effects model, the segmented human body region and the desired special effects fusion model can be intersected in a fragment shader to obtain the target special effects fusion model. A fragment shader is a programmable program for image processing that runs on hardware with a programmable rendering pipeline. In this embodiment, after determining the segmented human body region and the desired special effects fusion model, the corresponding fragment shader can be run to fuse the two.

[0059] For example, when it is determined that the human body segmentation area is the area corresponding to the user's head, and the special effects fusion model to be used is the model corresponding to the fish tank special effect (that is, the model contains an elliptical fish tank model and a transparent paper model), the application can run the fragment shader to extract the picture corresponding to the user's head area, and combine the picture with the special effects fusion model to be used to obtain the target special effects fusion model. It can be understood that after the target special effects fusion model is rendered in the display interface, a picture of the user's head in the fish tank can be presented.

[0060] Based on the above description, it can be determined that the process of generating the target special effects fusion model can also be understood as a process of performing cutout sampling processing on the picture of the human body segmentation area, and combining the cutout result with the special effects fusion model to be used, so as to obtain an image that needs to be rendered on the display interface.

[0061] S130 , writing pixel depth information corresponding to the target special effect fusion model into a rendering engine, so that the rendering engine renders a target image corresponding to the image to be processed based on the pixel depth information.

[0062] The rendering engine can be a program that controls a graphics processing unit (GPU) to render relevant images. It can be understood that in this embodiment, after the target special effect fusion model is determined, the computer, driven by the rendering engine, can complete the task of drawing the image reflected by the target special effect fusion model onto the target display interface. Accordingly, the rendered target image includes at least the target special effect. Continuing with the above example, after the target special effect fusion model corresponding to the fish tank special effect is determined, the rendering engine, driven by the rendering engine, renders the model and displays the obtained target image on the display interface, which can present a picture of the user's head in the fish tank.

[0063] Optionally, the pixel depth information of each pixel corresponding to the target special effect fusion model is determined based on the rendering camera, and the pixel depth information is written into the rendering engine, so as to write the pixel depth information into the paper model based on the rendering engine to obtain the target image.

[0064] Among them, the rendering camera can be a program used to determine the relevant parameters of each pixel point in the 3D virtual space, and the pixel depth information can be the depth value corresponding to each pixel point in the final rendered image. Those skilled in the art should understand that the depth value of each pixel point is at least used to reflect the depth of each pixel point in the image (that is, the distance between the virtual rendering camera lens and the pixel point). At the same time, in the pre-constructed three-dimensional space, these depth values ​​can also determine the distance between the corresponding pixel point and the viewpoint.

[0065] For example, when the target special effect is a fish tank special effect and the target special effect fusion model corresponding to the special effect is determined, the application needs to use the rendering camera to first obtain the depth value of each pixel on the user's head screen; further, the depth value of each pixel is written into the rendering engine, so that the rendering engine writes these depth values ​​into the paper model corresponding to the user's head area according to the relative position relationship. Finally, the paper model is rendered to obtain the target image. It can be understood that the target image not only includes the picture of the fish tank as a special effect, but also presents the picture of the user's head.

[0066] It should be noted that in actual applications, the rendering engine's relevant parameters (color writemask) can be set to 0. That is, when rendering the target image, only the depth value of each pixel is written into the paper model, without writing the color information of each pixel into the paper model. On this basis, when the target special effect is the fish tank special effect containing multiple goldfish models as in the above example, in the final rendered image, the user's head image can block the goldfish image, avoiding the phenomenon of the goldfish image and the user's head intersecting the model.

[0067] It should also be noted that in actual application, if the target special effects display instruction is not received again and the image to be processed is collected again, the target special effects fusion model is determined based on the binding of the special effects fusion model to be processed and the human body segmentation area.

[0068] It can be understood that in the process of determining the target special effects fusion model and rendering the corresponding target image on the display interface, the application can also store the information determined in the above process, as well as the binding relationship between the special effects fusion model to be processed and the human body segmentation area of ​​the target object, so that when the image containing the human body segmentation area of ​​the target object is collected again, the above data can be directly called, and the corresponding picture can be rendered on the target display interface. Continuing with the above example, for the fish tank model, the application can store the binding relationship between the picture of the target user's head area and the special effects fusion model to be processed, as well as the target special effects fusion model that reflects the picture of the fish tank on the user's head. Furthermore, if the picture of the target user's head area is detected again in the image taken by the user or the image actively uploaded by the user, the application can directly call the corresponding target special effects fusion model, and then render the picture reflected by the model to the display interface based on the rendering engine.

[0069] By storing the binding relationship between the special effects fusion model to be processed and the human body segmentation area, as well as the corresponding target special effects fusion model, when the application collects the image to be processed again and detects the human body segmentation area of ​​the target object from it, it can directly call the relevant data for rendering, avoiding the waste of computing resources and improving the special effects image processing efficiency of the application.

[0070] The technical solution of the embodiment of the present disclosure determines a fusion model of the special effect to be processed according to the special effect properties of the target special effect to be superimposed, that is, determines a model to be fused corresponding to a specific area in the image, and when a target special effect display instruction is received, determines the target special effect fusion model according to the special effect fusion model to be processed and the human body segmentation area corresponding to the target object in the image to be processed, thereby realizing the fusion of the special effect with the picture content of the area, and writing the pixel depth information corresponding to the target special effect fusion model into the rendering engine, so that the rendering engine renders the target image corresponding to the image to be processed based on the pixel depth information, thereby improving the accuracy of the fusion of the special effect with the specific area in the image, and at the same time, avoiding the occurrence of model penetration between the special effect and the picture content, making the final special effect image more realistic, and enhancing the user experience.

[0071] Example 2

[0072] Figure 4 This is a structural diagram of a special effects image processing device provided in the second embodiment of the present disclosure, such as Figure 4 As shown, the device includes: a to-be-processed special effect fusion model determination module 210 , a target special effect fusion model determination module 220 and a rendering module 230 .

[0073] The special effect fusion model to be processed determining module 210 is used to determine the special effect fusion model to be processed according to the special effect properties of the target special effect to be superimposed.

[0074] The target special effect fusion model determination module 220 is used to determine the target special effect fusion model according to the special effect fusion model to be processed and the human body segmentation area corresponding to the target object in the image to be processed when receiving the target special effect display instruction.

[0075] The rendering module 230 is used to write the pixel depth information corresponding to the target special effect fusion model into the rendering engine, so that the rendering engine renders a target image corresponding to the image to be processed based on the pixel depth information; wherein the target image includes the target special effect.

[0076] Optionally, the special effect fusion model to be processed determining module 210 is further used to determine a special effect fusion model corresponding to the target special effect according to the special effect display shape and special effect display position of the target special effect, as the special effect fusion model to be processed.

[0077] On the basis of the above technical solutions, the special effects image processing device also includes a special effects fusion model display module to be processed.

[0078] The special effect fusion model display module to be processed is used to display the special effect fusion model to be processed when receiving the target special effect display instruction; wherein, the paper model in the special effect fusion model to be processed is displayed transparently.

[0079] On the basis of the above technical solutions, the target special effect fusion model determination module 220 includes a human body segmentation region determination unit and a target special effect fusion model determination unit.

[0080] The human body segmentation region determining unit is used to determine the human body segmentation region corresponding to the target object in the image to be processed.

[0081] The target special effect fusion model determination unit is used to bind the human body segmentation area with the special effect fusion model to be processed to obtain the target special effect fusion model.

[0082] Optionally, the human body segmentation region determining unit is further configured to determine the human body segmentation region in the image to be processed based on a human body segmentation algorithm or a human body segmentation model.

[0083] On the basis of the above technical solutions, the human body segmentation area is any part in the torso segmentation area.

[0084] Optionally, the target special effects fusion model determination unit is also used to bind the human body segmentation area with the special effects fusion model to be processed to obtain the special effects fusion model to be used; and perform intersection processing on the human body segmentation area and the special effects fusion model to be used to determine the target special effects fusion model.

[0085] Optionally, the target special effect fusion model determination unit is further used to determine a target reference axis corresponding to the target object; and to control the joint movement of the human body segmentation area and the special effect fusion model to be processed using the target reference axis to obtain the special effect fusion model to be used.

[0086] Optionally, the rendering module 230 is also used to determine the pixel depth information of each pixel corresponding to the target special effect fusion model based on the rendering camera, and write the pixel depth information into the rendering engine, so as to write the pixel depth information into the paper model based on the rendering engine to obtain the target image.

[0087] Optionally, the target special effect fusion model determination module 220 is also used to determine the target special effect fusion model based on the binding of the special effect fusion model to be processed and the human body segmentation area if the target special effect display instruction is not received again and the image to be processed is collected again.

[0088] The technical solution provided in this embodiment determines the fusion model of the special effect to be processed according to the special effect properties of the target special effect to be superimposed, that is, determines the model to be fused corresponding to the specific area in the image, and when receiving the target special effect display instruction, determines the target special effect fusion model according to the special effect fusion model to be processed and the human body segmentation area corresponding to the target object in the image to be processed, thereby realizing the fusion of the special effect and the picture content of the area, and writing the pixel depth information corresponding to the target special effect fusion model into the rendering engine, so that the rendering engine renders the target image corresponding to the image to be processed based on the pixel depth information, thereby improving the accuracy of the fusion of the special effect and the specific area in the image, and at the same time, avoiding the occurrence of model penetration between the special effect and the picture content, making the final special effect image more realistic, and enhancing the user experience.

[0089] The special effects image processing device provided by the embodiments of the present disclosure can execute the special effects image processing method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0090] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0091] Example 3

[0092] Figure 5 This is a structural diagram of an electronic device provided by the third embodiment of the present disclosure. Figure 5 , which shows an electronic device (eg Figure 5 The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0093] like Figure 5 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a pattern processor, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 306 into a random access memory (RAM) 303. Various programs and data required for the operation of the electronic device 300 are also stored in the RAM 303. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An edit / output (I / O) interface 305 is also connected to the bus 304.

[0094] Typically, the following devices may be connected to the I / O interface 305: an editing device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 5 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0095] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 306, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0096] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0097] The electronic device provided by the embodiment of the present disclosure and the special effect image processing method provided by the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0098] Example 4

[0099] An embodiment of the present disclosure provides a computer storage medium having a computer program stored thereon. When the program is executed by a processor, the special effect image processing method provided in the above embodiment is implemented.

[0100] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0101] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0102] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0103] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0104] Determine the special effect fusion model to be processed according to the special effect properties of the target special effect to be superimposed;

[0105] Upon receiving a target special effect display instruction, determining a target special effect fusion model according to the special effect fusion model to be processed and a human body segmentation region corresponding to the target object in the image to be processed;

[0106] The pixel depth information corresponding to the target special effect fusion model is written into the rendering engine, so that the rendering engine renders a target image corresponding to the image to be processed based on the pixel depth information; wherein the target image includes the target special effect.

[0107] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0108] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0109] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0110] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0111] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0112] According to one or more embodiments of the present disclosure, [Example 1] provides a special effects image processing method, the method comprising:

[0113] Determine the special effect fusion model to be processed according to the special effect properties of the target special effect to be superimposed;

[0114] Upon receiving a target special effect display instruction, determining a target special effect fusion model according to the special effect fusion model to be processed and a human body segmentation region corresponding to the target object in the image to be processed;

[0115] The pixel depth information corresponding to the target special effect fusion model is written into the rendering engine, so that the rendering engine renders a target image corresponding to the image to be processed based on the pixel depth information; wherein the target image includes the target special effect.

[0116] According to one or more embodiments of the present disclosure, [Example 2] provides a special effects image processing method, the method further comprising:

[0117] Optionally, based on the special effect display shape and special effect display position of the target special effect, a special effect fusion model corresponding to the target special effect is determined as the special effect fusion model to be processed.

[0118] According to one or more embodiments of the present disclosure, [Example 3] provides a special effects image processing method, the method further comprising:

[0119] Optionally, when a target special effect display instruction is received, the special effect fusion model to be processed is displayed; wherein, the paper model in the special effect fusion model to be processed is displayed transparently.

[0120] According to one or more embodiments of the present disclosure, [Example 4] provides a special effects image processing method, the method further comprising:

[0121] Optionally, determining a human body segmentation region corresponding to the target object in the image to be processed;

[0122] The human body segmentation area is bound to the special effect fusion model to be processed to obtain the target special effect fusion model.

[0123] According to one or more embodiments of the present disclosure, [Example 5] provides a special effects image processing method, the method further comprising:

[0124] Optionally, based on a human body segmentation algorithm or a human body segmentation model, a human body segmentation area in the image to be processed is determined.

[0125] According to one or more embodiments of the present disclosure, [Example 6] provides a special effects image processing method, the method further comprising:

[0126] Optionally, the human body segmentation region is any part in the torso segmentation region.

[0127] According to one or more embodiments of the present disclosure, [Example 7] provides a special effects image processing method, the method further comprising:

[0128] Optionally, the human body segmentation region is bound to the special effect fusion model to be processed to obtain the special effect fusion model to be used;

[0129] The human body segmentation area and the special effect fusion model to be used are intersected to determine a target special effect fusion model.

[0130] According to one or more embodiments of the present disclosure, [Example 8] provides a special effects image processing method, the method further comprising:

[0131] Optionally, determining a target reference axis corresponding to the target object;

[0132] The target reference axis is used to control the joint movement of the human body segmentation area and the special effect fusion model to be processed to obtain the special effect fusion model to be used.

[0133] According to one or more embodiments of the present disclosure, [Example 9] provides a special effects image processing method, the method further comprising:

[0134] Optionally, the pixel depth information of each pixel corresponding to the target special effect fusion model is determined based on the rendering camera, and the pixel depth information is written into the rendering engine, so as to write the pixel depth information into the paper model based on the rendering engine to obtain the target image.

[0135] According to one or more embodiments of the present disclosure, [Example 10] provides a special effects image processing method, the method further comprising:

[0136] Optionally, if the target special effect display instruction is not received again and the image to be processed is collected again, the target special effect fusion model is determined based on the binding of the special effect fusion model to be processed and the human body segmentation area.

[0137] According to one or more embodiments of the present disclosure, [Example 11] provides a special effects image processing device, the device comprising:

[0138] A module for determining a special effect fusion model to be processed, for determining a special effect fusion model to be processed according to special effect properties of a target special effect to be superimposed;

[0139] a target special effect fusion model determination module, configured to determine the target special effect fusion model based on the special effect fusion model to be processed and the human body segmentation area corresponding to the target object in the image to be processed upon receiving a target special effect display instruction;

[0140] A rendering module is used to write the pixel depth information corresponding to the target special effect fusion model into a rendering engine, so that the rendering engine renders a target image corresponding to the image to be processed based on the pixel depth information; wherein the target image includes the target special effect.

[0141] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0142] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0143] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A special effects image processing method, characterized in that: include: Determine the special effect fusion model to be processed according to the special effect properties of the target special effect to be superimposed; Upon receiving a target special effect display instruction, a target special effect fusion model is determined based on the special effect fusion model to be processed and a human body segmentation region corresponding to the target object in the image to be processed; wherein the target special effect fusion model is determined based on a target reference axis and the special effect fusion model to be processed; the target reference axis is used to control the joint movement of the special effect fusion model to be processed and the human body segmentation region; the human body segmentation region is a region corresponding to the user's head; and the special effect fusion model to be processed includes a paper model; Writing pixel depth information corresponding to the target special effect fusion model into a rendering engine, so that the rendering engine renders a target image corresponding to the image to be processed based on the pixel depth information, including: determining pixel depth information of each pixel corresponding to the target special effect fusion model based on a rendering camera, and writing the pixel depth information into the rendering engine, so that the rendering engine writes the pixel depth information into a paper model corresponding to the user's head area to obtain the target image; The target image includes the target special effects and the image of the user's head; and the pixel depth information is a depth value corresponding to at least one pixel in the target image.

2. The method according to claim 1, characterized in that The determining of a special effect fusion model to be processed according to the special effect attributes of the target special effect to be superimposed includes: According to the special effect display shape and special effect display position of the target special effect, a special effect fusion model corresponding to the target special effect is determined as the special effect fusion model to be processed.

3. The method according to claim 1, characterized in that Also includes: When a target special effect display instruction is received, the special effect fusion model to be processed is displayed; wherein the paper model in the special effect fusion model to be processed is displayed transparently.

4. The method according to claim 1, wherein The step of determining a target special effects fusion model according to the special effects fusion model to be processed and a human body segmentation region corresponding to a target object in the image to be processed includes: Determining a human body segmentation region corresponding to a target object in the image to be processed; The human body segmentation area is bound to the special effect fusion model to be processed to obtain the target special effect fusion model.

5. The method according to claim 4, characterized in that Determining a human body segmentation region corresponding to a target object in the image to be processed includes: Based on a human body segmentation algorithm or a human body segmentation model, a human body segmentation region in the image to be processed is determined.

6. The method according to claim 5, characterized in that The human body segmentation region is any part in the torso segmentation region.

7. The method according to claim 4, characterized in that The step of binding the human body segmentation region with the special effect fusion model to be processed to obtain the target special effect fusion model includes: Binding the human body segmentation area with the special effect fusion model to be processed to obtain the special effect fusion model to be used; The human body segmentation area and the special effect fusion model to be used are intersected to determine a target special effect fusion model.

8. The method according to claim 7, characterized in that The step of binding the human body segmentation region with the special effect fusion model to be processed to obtain the special effect fusion model to be used includes: determining a target reference axis corresponding to the target object; The target reference axis is used to control the joint movement of the human body segmentation area and the special effect fusion model to be processed to obtain the special effect fusion model to be used.

9. The method according to claim 1, characterized in that Also includes: If the target special effect display instruction is not received again and the image to be processed is collected again, the target special effect fusion model is determined based on the binding of the special effect fusion model to be processed and the human body segmentation area.

10. A special effects image processing device, characterized in that: include: A module for determining a special effect fusion model to be processed, for determining a special effect fusion model to be processed according to special effect properties of a target special effect to be superimposed; a target special effect fusion model determination module, configured to, upon receiving a target special effect display instruction, determine a target special effect fusion model based on the special effect fusion model to be processed and a human body segmentation region corresponding to a target object in the image to be processed; wherein the target special effect fusion model is determined based on a target reference axis and the special effect fusion model to be processed; the target reference axis is used to control the joint movement of the special effect fusion model to be processed and the human body segmentation region; the human body segmentation region is the region corresponding to the user's head; and the special effect fusion model to be processed includes a paper model; A rendering module, configured to write pixel depth information corresponding to the target special effect fusion model into a rendering engine, so that the rendering engine renders a target image corresponding to the image to be processed based on the pixel depth information; wherein the target image includes the target special effect and a picture of the user's head; and the pixel depth information is a depth value corresponding to at least one pixel in the target image; The rendering module is specifically used to determine the pixel depth information of each pixel corresponding to the target special effect fusion model based on the rendering camera, and write the pixel depth information into the rendering engine, so as to write the pixel depth information into the paper model corresponding to the user's head area based on the rendering engine to obtain the target image.

11. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the special effect image processing method according to any one of claims 1 to 9.

12. A storage medium comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, are used to execute the special effect image processing method according to any one of claims 1 to 9.

Images