Image processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image processing method and device, electronic equipment and a storage medium, and the method comprises the steps: collecting a to-be-processed image comprising a target object in response to a special effect triggering operation; wherein the to-be-processed image comprises at least one to-be-drawn part of the target object; determining an outline drawing of the to-be-drawn part; and determining a display special effect graph corresponding to the to-be-processed image based on the outline drawing graph. According to the technical scheme provided by the embodiment of the invention, the problems that in the prior art, when a neural network is adopted to carry out special effect processing on the image, due to the fact that only brightness information is included in the image, the processing is easily influenced by noisy points, the edge contour of the obtained stylized image is not clear, and the display effect of the stylized image is poor are solved; the effect of improving the processing pertinence of the to-be-delineated part and the trueness of the special effect image when the to-be-delineated part is processed by adopting different processing modes is achieved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image processing technology, and in particular, to an image processing method, device, electronic device, and storage medium. Background Art

[0002] With the development of video creation, users have great demands for the diversity of video content. At present, the processing of video special effects is mostly based on neural networks, and corresponding algorithms can be integrated into neural networks.

[0003] However, there is a problem when using the above method to process images: the images captured by the camera device usually only include brightness information of the image, which is easily affected by noise during special effects processing, resulting in poor stylized images, which in turn causes poor user experience. Summary of the invention

[0004] The present disclosure provides an image processing method, device, electronic device and storage medium, so as to obtain a special effect image under a target style by performing special effect processing on different parts respectively, thereby achieving a targeted effect of processing the special effect image.

[0005] In a first aspect, an embodiment of the present disclosure provides an image processing method, the method comprising:

[0006] In response to a special effect triggering operation, an image to be processed including a target object is collected; wherein the image to be processed includes at least one part to be outlined of the target object;

[0007] Determining a contour outline of the part to be outlined according to at least two processing methods corresponding to the part to be outlined;

[0008] Based on the outline drawing, a display special effect drawing corresponding to the image to be processed is determined.

[0009] In a second aspect, an embodiment of the present disclosure further provides an image processing device, the device comprising:

[0010] An image acquisition module, configured to acquire an image to be processed including a target object in response to a special effect trigger operation; wherein the image to be processed includes at least one part to be outlined of the target object;

[0011] An image outlining module, configured to determine a contour outlining image of the part to be outlined according to at least two processing methods corresponding to the part to be outlined;

[0012] The special effect image generation module is used to determine a display special effect image corresponding to the image to be processed based on the outline sketch.

[0013] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0014] one or more processors;

[0015] a storage device for storing one or more programs,

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any one of the embodiments of the present disclosure.

[0017] In a fourth aspect, the embodiments of the present disclosure further provide a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the image processing method as described in any one of the embodiments of the present disclosure.

[0018] The technical solution provided by the embodiments of the present disclosure can, when the special effect generation conditions are detected, collect an image to be processed that includes a target object. If the image to be processed includes at least one part to be outlined, a contour outline corresponding to the part to be outlined can be determined. Through each contour outline, a display special effect map of the corresponding image to be processed can be determined. This solves the problem in the prior art that when using neural networks to process images, the image is easily affected by noise, resulting in poor stylized image effects, which in turn causes a poor user experience. This achieves the technical effect of processing the image to be processed based on the characteristics of each part to be outlined, thereby improving the pertinence and effectiveness of special effect image processing. Furthermore, it enriches special effect props and achieves the effect of enriching video content. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0020] Figure 1 is a flowchart of an image processing method provided by an embodiment of the present disclosure;

[0021] Figure 2 is a flowchart of an image processing method provided by an embodiment of the present disclosure;

[0022] Figure 3 It is a schematic diagram showing the process of contour processing of a hair part provided by an embodiment of the present disclosure;

[0023] Figure 4 It is a schematic diagram showing the process of contour processing of a hair part provided by an embodiment of the present disclosure;

[0024] Figure 5 It is a flowchart of an image processing method provided by an embodiment of the present invention;

[0025] Figure 6 is a schematic diagram of the effect of processing at least one facial part and facial contour provided by an embodiment of the present disclosure;

[0026] Figure 7 is a flowchart of an image processing method provided by an embodiment of the present disclosure;

[0027] Figure 8 is a schematic diagram of the iris effect provided by an embodiment of the present disclosure;

[0028] Fig. 9 is a flowchart of an image processing method provided by an embodiment of the present disclosure;

[0029] Fig.10 is a schematic diagram of a stroke template of a nose part provided in an embodiment of the present disclosure;

[0030] Fig.11 is a schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure;

[0031] Fig.12 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0033] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0034] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0035] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0036] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0037] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0038] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0039] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0040] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0041] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0042] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0043] Before introducing the present technical solution, an exemplary description of the application scenario can be given first. The technical solution provided by the embodiment of the present disclosure can be applied to any scene that requires special effects processing, and the scene of the special effects processing can be a live broadcast scene, a short video shooting scene, a video communication scene, and the like. The live broadcast scene can be a live broadcast of selling goods, a live broadcast of beauty makeup, or a live broadcast of chatting, and the like. The short video shooting scene can be any scene that can realize video shooting and perform special effects processing on the video content. The special effects processing of short videos can be real-time, or it can be performed after the shooting is completed. The video communication scene can be a video chat scene, where special effects processing is performed on the opposite user, or a scene where special effects processing is performed on the user corresponding to the current terminal. In the above scenarios, as long as the special effects operation is triggered, the solution provided by the embodiment of the present disclosure can be executed to determine and display the special effects image corresponding to each frame, that is, to determine the display of the special effects map.

[0044] It should be noted that the application scenarios described above are only simple explanations of the application scenarios of the solutions, and are not limitations on the application scenarios. The solutions can also be applied in other application scenarios.

[0045] It should also be noted that the image processing device provided by the embodiment of the present disclosure can be integrated into an application software that supports video image processing functions, and the software can be installed in an electronic device, and optionally, the electronic device can be a mobile terminal or a PC, etc. The application software can be software for image processing, and its specific application software will not be described one by one here, as long as it can realize image processing.

[0046] Figure 1 It is a flow chart of an image processing method provided by an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to any situation where special effects processing is required on the collected audio and video frames. The method can be executed by an image processing device, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, a PC or a server, etc.

[0047] like Figure 1 As shown, the method includes:

[0048] S110. In response to a special effect triggering operation, collecting an image to be processed including a target object; wherein the image to be processed includes at least one part of the target object to be outlined.

[0049] In this embodiment, in application software or applications that support special effect image processing functions, controls for triggering special effects can be developed in advance. When it is detected that the user triggers the control, the special effect triggering operation can be responded to, thereby starting to collect images to be processed including the target object.

[0050] It should be noted that the technical solution provided by the embodiment of the present disclosure can process the video frames shot in real time in sequence, and each audio and video frame being processed can be used as an image to be processed. It can also be used to post-process the shot video, that is, each video frame in the video can be used as an image to be processed in sequence. In other words, the image to be processed is the image that is currently being processed with special effects.

[0051] Among them, the target object can be a pre-set acquisition object, and optionally, the target object can be a user or an animal, etc. The target object may or may not be included in the image to be processed. If the image collected in real time is processed, it can be determined whether the target object is included in each video frame (image to be processed). If the target object is not included, it may not be processed; if the target object is included, the solution provided in the embodiment of the present disclosure can be executed to determine the corresponding display special effect map.

[0052] The number of at least one outlined part can be one or more, and the specific outlined part is set according to actual needs, that is, the outlined part can be set in advance according to the target style that is ultimately desired to be presented. The part to be outlined can be understood as a part that needs to be drawn with lines.

[0053] The part to be outlined can be any part whose edge contour needs to be drawn. The part to be outlined matches the object type. Optionally, if the object type is a user type, the part to be outlined can be any one or more parts of the head, torso, and limbs; if the object type is an animal or plant landscape painting, the part to be outlined can be any part of the picture.

[0054] In order to clearly introduce the solution provided by the embodiment of the present disclosure, the target object may be a user as an example, and accordingly, the part to be outlined may be at least one of hair, at least one facial part, and a facial contour. The facial part may be at least one of an eye part, a nose part, and an eyebrow part.

[0055] Specifically, when it is detected that the conditions for generating a display special effect image are met, an image to be processed including the target object may be acquired, and the image to be processed may include at least one part to be outlined.

[0056] In this embodiment, the special effect triggering operation may include at least one of the following: triggering special effect generation props, detecting that the voice information includes special effect image generation instructions; the display interface includes the target object; the display interface includes the part to be outlined.

[0057] It can be understood that the method provided by the embodiment of the present disclosure can be integrated into a special effect prop. When the triggering of the special effect prop is detected, it means that the collected audio and video frames need to be processed, and each collected audio and video frame is used as an image to be processed. Alternatively, voice information is collected based on the microphone array deployed on the terminal device, and the voice information is analyzed and processed. If the processing result includes a special effect adding vocabulary matching the embodiment of the present disclosure, it means that the props for generating and displaying special effect images are triggered. The advantage of determining whether to add special effects based on the content of the voice information is that it avoids the interaction between the user and the display page, and improves the intelligence of adding special effects. Another implementation method can be to determine whether the user's facial image is included in the field of view of the mobile terminal according to the shooting field of view. When the user's facial image is detected, the application software can use the event of detecting the facial image as a triggering operation for generating a special effect image. It can also be that if the field of view includes at least one pre-set part to be outlined, it means that the operation of generating and displaying a special effect image is triggered. It should be understood by those skilled in the art that the specific event selected as the page turning condition for the special effect can be set according to the actual situation, and the embodiment of the present disclosure does not make specific limitations here.

[0058] S120: Determine a contour outline of the part to be outlined according to at least two processing methods corresponding to different parts to be outlined.

[0059] The part to be outlined can be understood as a part whose outline is drawn with lines. The image corresponding to the part whose outline is drawn with lines is used as the outline drawing. The number of the parts to be outlined matches the number of outline drawings, that is, the number of outline drawings can be drawn according to the number of the parts to be outlined set in advance.

[0060] For different parts to be outlined, the same or different methods can be used to determine their contour outline images, that is, the contour outline images of the corresponding parts to be outlined are determined according to at least two processing methods corresponding to the different parts to be outlined.

[0061] Optionally, the pixel points corresponding to the part to be outlined can be determined, and their pixel values ​​can be converted into preset pixel values ​​to determine its outline map. Alternatively, a corresponding neural network can be trained for each part to be outlined, that is, the neural network processes different parts to be outlined to obtain corresponding outline maps.

[0062] In this embodiment, the processing of the portion not to be outlined can be implemented in at least two processing modes.

[0063] In this embodiment, at least two processing methods are determined based on the part display type corresponding to the part to be outlined. Among them, the part display type may include a first display type or a second display type. The first display type may be a local display type, and the second type may be a global display type, wherein the global display type and the local display type may be determined based on the area covered by the part to be outlined. The part type corresponding to a larger coverage area may be used as a global display type, and the type corresponding to a smaller coverage area may be used as a local display type. The part type of the part to be outlined may also be determined based on whether there is an occlusion phenomenon when the part to be outlined is displayed in the display interface. For example, when the user displays hair in the display interface, it will be blocked by the face. In this case, it can be called a local display type; for example, when the eye part is displayed, the iris may be blocked. At this time, it can also be called a local display type. The global display type is a type that all parts are to be displayed. For example, the nose, mouth, eyebrows, etc. in the face are all displayed, and it can be used as a global display type.

[0064] It can be understood that the parts to be outlined under the first display type include the hair part and the eye part, and the second display type may include the nose part, the mouth part, the eyebrow part and the mouth part.

[0065] Correspondingly, under different conditions of part display types, there are certain differences in processing methods. Optionally, the part type corresponding to the at least one part to be outlined includes a first type or a second type, and the first type and the second type correspond to different processing methods.

[0066] Furthermore, the processing methods corresponding to the first type include at least two of a mask processing method, a key point processing method, and a binarization processing method, and the second type includes at least two of a mask processing method, a key point processing method, a linear fitting method, and a template adaptation method. Among them, the key point processing method can be understood as a method for determining the key points corresponding to the part to be outlined. The mask processing method is a method for determining a mask image processing method corresponding to the part to be outlined. The linear fitting method is a method for fitting the key points into a straight line. The method of outlining based on the template and the key points is called the template adaptation method.

[0067] S130: Based on the outline drawing, determine a display special effect drawing corresponding to the image to be processed.

[0068] The display special effects image is an image obtained after the special effects processing of the image to be processed.

[0069] It can be understood that after obtaining the contour outline corresponding to each part to be outlined, all the contour outlines can be placed at the positions corresponding to the parts to be outlined, thereby obtaining a display special effects image.

[0070] In this embodiment, not only a certain image to be processed can be processed, but each video frame in the captured video can also be processed, and each video frame can be used as the image to be processed. If the video is processed, the corresponding display special effect images can be spliced ​​according to the shooting timestamp corresponding to each image to be processed to obtain a special effect video corresponding to the captured video.

[0071] The technical solution provided by the embodiments of the present disclosure can, when the special effect generation conditions are detected, collect an image to be processed that includes a target object. If the image to be processed includes at least one part to be outlined, a contour outline corresponding to the part to be outlined can be determined. Through each contour outline, a display special effect image of the corresponding image to be processed can be determined. This solves the problem that the use of neural networks to process images in the prior art is easily affected by noise, resulting in poor stylized image effects. For each part to be outlined, at least two processing methods can be used to process it according to the characteristics of the part to be outlined, so that the lines of the part to be outlined are smoother, thereby achieving the technical effect of targeted and effective processing of each part to be outlined in the special effect image.

[0072] Figure 2 It is a flowchart of an image processing method provided by an embodiment of the present invention. On the basis of the above-mentioned embodiment, it can first introduce how to determine the contour outline corresponding to the hair part. The specific implementation method can refer to the detailed description of this embodiment, among which the technical terms that are the same as or corresponding to the above-mentioned embodiment are not repeated here.

[0073] like Figure 2 As shown, the method includes:

[0074] S210. In response to a special effect triggering operation, collecting an image to be processed including a target object; wherein the image to be processed includes at least one part to be outlined of the target object.

[0075] It should be noted that the part to be outlined can be any part whose edge contour is determined. In this embodiment, the part to be outlined is described as a hair part.

[0076] S220: Process the image to be processed according to the mask processing method to determine a hair mask image corresponding to the hair part.

[0077] It should be noted that the image to be processed may include the hair part or may not include the hair part. If the hair part is included, the technology provided in the embodiment of the present disclosure may be used to determine the outline corresponding to the hair part.

[0078] A segmentation algorithm may be used to determine the hair part in the image to be processed to obtain a hair mask image corresponding to the hair part. The segmentation algorithm may be any algorithm capable of implementing hair part segmentation, and the specific algorithm content is not limited in this embodiment.

[0079] For example, see Figure 3 If the target object in the image to be processed is a user, the existing hair segmentation model can be used to segment the hair parts of the target object to obtain a hair mask image corresponding to the hair parts.

[0080] S230, masking the hair according to the binary processing method Figure 2 After value processing, a binary image of the hair is obtained.

[0081] The hair binary image includes two types of pixel values, one type of pixel value is 0, and the other type of pixel value is 255. After the mask image is determined according to the mask processing method, since only the mask processing is performed, there may be a situation where the edge contour is unclear, that is, the pixel values ​​of the edge pixels of the hair mask image are different, resulting in unclear edges of the mask image. In order to obtain an edge contour image with clear edges, the hair mask image can be further binarized. The reason and benefit of this processing is that the edge contour of the hair mask image is unclear, and binarization processing can obtain a contour outline image with clear edge contours.

[0082] Binarization processing can be understood as adjusting the pixel values ​​of some pixels in the hair mask image to the first preset pixel value, and adjusting the pixel values ​​of some pixels to the second preset pixel value. Optionally, the pixel values ​​of pixels greater than the preset pixel value can be adjusted to the first preset pixel value, and the pixel values ​​of pixels less than or equal to the preset pixel value can be adjusted to the second preset pixel value, and a hair contour image with clear edge contours can be obtained.

[0083] Specifically, the mask image is binarized based on preset pixel values ​​to obtain a hair binarization image.

[0084] S240: Determine a contour outline corresponding to the hair part based on the hair binary image.

[0085] Specifically, the outline drawing may be a line outline drawing of the hair part, and the outline drawing of the hair part may be obtained by performing a stroke process based on the hair binary drawing.

[0086] On the basis of the above technical solution, in order to make the outline of the hair part vivid and highly matched with the user, the hair binary image can be further processed to obtain an outline with less harsh lines and a higher match with the target object.

[0087] Optionally, based on the hair binary image, determining an outline drawing corresponding to the hair includes: dilating the hair binary image to obtain a hair edge drawing; and determining the hair outline drawing based on the hair edge drawing and the hair binary image.

[0088] The dilation process may be a Gaussian blur process on the binary image. The hair stroke image is an image obtained by tracing the edge of the binary hair image after the dilation process. During the dilation process, the dilation intensity may be controlled, that is, the thickness of the edge contour line may be controlled.

[0089] Specifically, the hair binary image can be expanded to obtain a hair stroke image. Since the expansion process will expand the hair edge outward, the hair outline image can be obtained based on the hair stroke image and the pixel points in the hair binary image. Optionally, the hair stroke image and the pixel points in the hair binary image can be difference processed to obtain the outline image corresponding to the hair part.

[0090] The advantage of using the above-mentioned processing method is that not only a contour outline with clear edge contours can be obtained, but also a stylized image can be obtained to a large extent.

[0091] For example, see Figure 4 After obtaining the hair binary image (a), the hair binary image can be expanded, and a more efficient Gaussian blur can be used here to obtain the hair stroke image (b). The pixel value difference of the pixel point at the same position in the hair stroke image and the hair binary image is processed to obtain the outline of the hair part (c).

[0092] S250: Based on the outline drawing, determine a display special effect drawing corresponding to the image to be processed.

[0093] According to the technical solution provided by the embodiment of the present disclosure, if the part to be outlined is the hair part, a hair mask image corresponding to the hair part can be determined. At this time, the edge boundary of the mask image is not clear, and the hair mask image can be further binarized to obtain a binary image with clear edge contours. Furthermore, in order to obtain a style image in a certain style, the binary image can be expanded, and the degree of expansion can be controlled during the expansion process to obtain a contour outline with different line thicknesses. That is, the mask processing method and the binarization processing method are used to perform fine processing on the image of the hair part, thereby improving the technical effect of image stylization.

[0094] Figure 5 It is a flowchart of an image processing method provided by an embodiment of the present invention. On the basis of the aforementioned embodiment, it can be introduced again how to determine the contour outline corresponding to the mouth area, eyebrow area and face shape. The specific implementation method can refer to the detailed description of this embodiment, among which the technical terms that are the same as or corresponding to the aforementioned embodiment are not repeated here.

[0095] like Figure 5 As shown, the method includes:

[0096] S310. In response to a special effect triggering operation, collecting an image to be processed including a target object; wherein the image to be processed includes at least one part to be outlined of the target object.

[0097] It should be noted that at least one part to be outlined may be at least one facial part. The processing methods for the mouth part and the eyebrow part of the facial part are the same, and the processing method for the facial contour is also the same, which can be elaborated in detail at this time.

[0098] S320, determining key points of the mouth area, the eyebrow area, and the facial contour in the image to be processed based on a key point processing method.

[0099] Generally, the mouth, eyebrows and facial contours are the parts of the target object that can better reflect individual differences, and the effect of such processing may be as close as possible to the outline effect of the target object. The key point recognition algorithm can be any existing algorithm that can recognize the key points of the corresponding parts, and its key point recognition algorithm is used as the key point processing method.

[0100] Specifically, a key point recognition algorithm may be used to determine key points of the mouth, eyebrow, and facial contour in the image to be processed.

[0101] S330 , performing linear fitting processing on key points of the same part to be outlined based on a linear fitting method to obtain a contour outline of the corresponding part to be outlined.

[0102] There may be multiple key points corresponding to each part to be outlined. In this case, the key points are multiple points constituting the part to be outlined. In order to obtain the contour outline of the corresponding part, linear fitting can be performed on the key points of the same part to be outlined to obtain the contour outline of the part to be outlined, see Figure 6 .

[0103] S340: Based on the outline drawing, determine a display special effect drawing corresponding to the image to be processed.

[0104] The technical solution provided by the disclosed embodiment can fully display the mouth area, facial contour, and eyebrow area, and if special effects images are to be obtained in a stick figure style, the key points of the above-mentioned areas can be identified and linearly fitted. For example, a key point identification algorithm can be used to determine multiple key points corresponding to the area to be outlined. Next, a linear fitting method is used to fit the key points of the same area to be outlined to obtain an outline map corresponding to each area to be outlined, thereby achieving a targeted technical effect of processing the area to be processed with special effects.

[0105] Figure 7 It is a flowchart of an image processing method provided by an embodiment of the present disclosure. On the basis of the aforementioned embodiment, it continues to introduce how to determine the contour outline corresponding to the eye part. The specific implementation method can be found in the detailed description of this embodiment, wherein the technical terms that are the same as or corresponding to the aforementioned embodiment are not repeated here.

[0106] like Figure 7 As shown, the method includes:

[0107] S410. In response to a special effect triggering operation, an image to be processed including a target object is collected; wherein the image to be processed includes at least one part to be outlined of the target object.

[0108] Among them, at least one part to be outlined may be an eye part.

[0109] S420: Determine iris key points of the eye based on the key point processing method, and determine an iris schematic diagram based on the iris key points.

[0110] Generally, the differences in the eye parts are mainly reflected in the shape of the eye and the iris. The key point recognition algorithm can be used to identify the iris key points and the key points of the eye parts. The iris schematic diagram can be determined based on the iris key points. The special effects style of the special effects image is a black and white stick figure style, so the difference in the iris is mainly reflected in the size. Based on the iris key points, the size of the iris and the iris center of gravity can be calculated. Figure 8 The iris schematic diagram shown is drawn at the corresponding position to obtain a preliminary iris schematic diagram.

[0111] S430: Determine an eye mask image according to the eye key points and the mask processing method of the eye area.

[0112] Linear fitting can be performed based on the key points of the eye to obtain a schematic diagram of the eye contour. In order to simulate the eye in different states, an eye mask map can be determined first, and the iris of the part can be blocked based on the eye mask map to simulate a simple eye map of the user in the actual state.

[0113] S440: Determine an eye contour outline based on the eye mask image and the iris schematic diagram.

[0114] The iris texture can be masked according to the eye mask map to simulate the covering of the iris by the eyelid, and in particular, the blinking situation can be simulated, that is, the schematic diagram of the user's eye parts in different states can be simulated.

[0115] S450: Determine a display special effect image corresponding to the image to be processed based on the outline sketch.

[0116] The technical solution provided by the embodiments of the present disclosure can use a key point processing method to first determine the eye key points and iris key points corresponding to the eye part and the iris part, and the iris schematic diagram can be determined based on the iris key points. Further, a mask processing method is used to process the eye key points to determine an eye mask map, and the iris key points are masked based on the eye mask map to obtain the contour outline map corresponding to the eye part in different states, thereby achieving the technical effect of determining the accuracy and authenticity of the eye part contour outline map.

[0117] Fig. 9 It is a flowchart of an image processing method provided by an embodiment of the present disclosure. On the basis of the aforementioned embodiment, it can finally introduce how to determine the contour outline corresponding to the nose part. The specific implementation method can refer to the detailed description of this embodiment, among which the technical terms that are the same as or corresponding to the aforementioned embodiment are not repeated here.

[0118] like Fig. 9 As shown, the method includes:

[0119] S510. In response to a special effect triggering operation, an image to be processed including a target object is collected; wherein the image to be processed includes at least one part to be outlined of the target object.

[0120] It should be noted that at least one part to be outlined may be a nose part.

[0121] S520: Determine nose wing key points, nose bridge key points, and nostril key points corresponding to the nose part according to the preset stroke information and key point processing method.

[0122] The stroke information may be a pre-set sketch of the nose. For different users, the nose is a part with a strong style. If the stroke is simply done by image post-processing or by connecting key points, the result will be too realistic, which will have a huge impact on the final effect, that is, there will be no stick figure comic effect. Based on this, a hand-drawn method with higher degree of freedom can be used to perform stylized stroke processing on the nose.

[0123] Considering the stick figure style and the difference of each user's nose part, only certain feature points of the nose part can be drawn. Certain feature points can be nose wing key points, nose bridge key points and nostril key points. At this time, the stroke information can be certain part information set in advance. At this time, a key point processing method can be used to determine multiple key points corresponding to the nose part.

[0124] S530: Determine a contour outline of the nose based on nose wing key points, nose bridge key points, nostril key points, a pre-set stroke template, and a template adaptation method.

[0125] The stroke module may be a pre-set schematic diagram of a nose in a stick figure style, see Fig.10 The key points of the bridge of the nose are composed of multiple points, but the stroke template corresponds to a straight line, that is, the bridge of the nose can be represented by a straight line. Furthermore, the key points of the nostrils can be connected in sequence to obtain a simplified schematic diagram of the nostrils. Correspondingly, the key points of the nose wings can be processed according to the stroke template and the template adaptation method to obtain the outline of the nose.

[0126] S540: Determine a display special effect image corresponding to the image to be processed based on the outline sketch.

[0127] The technical solution provided by the embodiment of the present disclosure can adopt a key point recognition algorithm to determine the key points corresponding to the nose wings, nostrils and nose bridge. Based on the above key points and a pre-set stroke template, an outline outline corresponding to the nose part can be obtained, and a nose special effects image in the style of a comic stick figure can be obtained. Furthermore, based on the special effects image of the nose part, a display special effects image can be obtained, thereby achieving the effect of making the special effects image interesting.

[0128] Fig.11 is a schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure. The device is configured in a client, such as Fig.11 As shown, the device includes: an image acquisition module 610, an image delineation module 620 and a special effect image generation module 630.

[0129] The image acquisition module 610 is used to collect an image to be processed including a target object in response to a special effect trigger operation; wherein the image to be processed includes at least one part to be outlined of the target object; the image outline module 620 is used to determine a contour outline diagram of the part to be outlined; the special effect image generation module 630 is used to determine a display special effect diagram corresponding to the image to be processed based on the contour outline diagram.

[0130] The technical solution provided by the embodiments of the present disclosure can, when the special effect generation conditions are detected, collect an image to be processed that includes a target object. If the image to be processed includes at least one part to be outlined, a contour outline corresponding to the part to be outlined can be determined. Through each contour outline, a display special effect image of the corresponding image to be processed can be determined. This solves the problem in the prior art that when using neural networks to process images, the image is easily affected by noise, resulting in poor stylized image effects and further causing poor user experience. This realizes processing of the image to be processed based on the characteristics of each part to be outlined, thereby improving the technical effect of targeted and effective special effect image processing.

[0131] Based on the above technical solution, the part to be outlined includes at least one of a hair part, at least one facial part and a facial contour.

[0132] On the basis of the above technical solutions, the part to be outlined is the hair part, and the image outline module includes:

[0133] A mask image determining unit is used to determine a hair mask image corresponding to the hair according to the image to be processed; a binary image determining unit is used to determine the hair mask image corresponding to the hair Figure 2 The hair is subjected to a value processing to obtain a hair binary image; and a first outlining unit is used to determine a contour outlining image corresponding to the hair based on the hair binary image.

[0134] On the basis of the above technical solutions, the first outlining unit includes:

[0135] The stroke subunit is used to dilate the hair binary image to obtain a hair stroke image; the first outline subunit is used to determine the hair outline image based on the hair stroke image and the hair binary image.

[0136] On the basis of the above technical solutions, the part to be outlined includes at least one facial part, the at least one facial part includes a mouth part and an eyebrow part, and the image outline module includes:

[0137] A key point recognition unit is used to determine the key points of the mouth, the eyebrow and the facial contour in the image to be processed based on a key point recognition algorithm; a second outlining unit is used to perform linear fitting processing on the key points of the same part to be outlined to obtain a contour outline of the corresponding part to be outlined.

[0138] On the basis of the above technical solutions, the part to be outlined includes the eye part, and the image outline module includes:

[0139] An iris schematic diagram determining subunit is used to determine the iris key points of the eye area and determine the iris schematic diagram based on the iris key points; an eye mask map determining subunit is used to determine the eye mask map based on the eye key points of the eye area; a third outlining unit is used to determine the outline map of the eye area based on the eye mask map and the iris schematic diagram.

[0140] On the basis of the above technical solutions, the part to be outlined includes the nose part, and the image outline module includes:

[0141] a pixel point determination unit, for determining the nose wing key points, nose bridge key points and nostril pixel points corresponding to the nose part according to the preset stroke information; a fourth outlining unit, for determining the contour outline of the nose part based on the nose wing key points, the nose bridge key points, the nostril pixel points and the preset stroke template.

[0142] Based on the above technical solutions, the method is applied in a live broadcast scene or integrated into special effects props.

[0143] The image processing device provided in the embodiments of the present disclosure can execute the image processing method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0144] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0145] Fig.12 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Fig.12 , which shows an electronic device (eg, Fig.12 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.12 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0146] like Fig.12As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An edit / output (I / O) interface 605 is also connected to the bus 604.

[0147] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.12 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0148] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0149] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0150] The electronic device provided by the embodiment of the present disclosure and the image processing method provided by the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0151] The embodiments of the present disclosure provide a computer storage medium on which a computer program is stored. When the program is executed by a processor, the image processing method provided by the above embodiments is implemented.

[0152] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0153] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0154] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0155] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0156] In response to a special effect triggering operation, an image to be processed including a target object is collected; wherein the image to be processed includes at least one part to be outlined of the target object;

[0157] Determine a contour outline of the part to be outlined;

[0158] Based on the outline drawing, a display special effect drawing corresponding to the image to be processed is determined.

[0159] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0160] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0161] According to one or more embodiments of the present disclosure, [Example 1] provides an image processing method, the method comprising:

[0162] In response to a special effect triggering operation, an image to be processed including a target object is collected; wherein the image to be processed includes at least one part to be outlined of the target object;

[0163] Determine a contour outline of the part to be outlined according to at least two processing methods corresponding to the part to be outlined;

[0164] Based on the outline drawing, a display special effect drawing corresponding to the image to be processed is determined.

[0165] According to one or more embodiments of the present disclosure, [Example 2] provides an image processing method, the method comprising:

[0166] Optionally, the part type corresponding to the at least one part to be outlined includes a first type or a second type, and the first type and the second type correspond to different processing methods.

[0167] According to one or more embodiments of the present disclosure, [Example 3] provides an image processing method, the method comprising:

[0168] Optionally, the processing methods corresponding to the first type include at least two of a mask processing method, a key point processing method, and a binarization processing method, and the processing methods corresponding to the second type include at least two of a mask processing method, a key point processing method, a linear fitting method, and a template adaptation method.

[0169] According to one or more embodiments of the present disclosure, [Example 4] provides an image processing method, the method comprising:

[0170] Optionally, the part to be outlined includes at least one of limbs, a trunk, a hair part, at least one facial part and a facial contour.

[0171] According to one or more embodiments of the present disclosure, [Example 5] provides an image processing method, the method comprising:

[0172] Optionally, the part to be outlined is hair in the head, the processing method is a binarization processing method and a mask processing method, and the determining of the outline of the part to be outlined includes:

[0173] Processing the image to be processed according to the mask processing method to determine a hair mask image corresponding to the hair part;

[0174] Mask the hair according to the binary processing method Figure 2 Value processing is performed to obtain a hair binary image;

[0175] Based on the hair binary image, a contour outline image corresponding to the hair part is determined.

[0176] According to one or more embodiments of the present disclosure, [Example 6] provides an image processing method, the method comprising:

[0177] Optionally, determining a contour outline image corresponding to the hair part based on the hair binary image includes:

[0178] dilating the hair binary image to obtain a hair stroke image;

[0179] Based on the hair stroke map and the hair binarization map, a contour outline map of the hair part is determined.

[0180] According to one or more embodiments of the present disclosure, [Example 7] provides an image processing method, the method comprising:

[0181] Optionally, the part to be outlined includes at least one facial part, the at least one facial part includes a mouth part and an eyebrow part, the part to be outlined also includes a facial contour, the processing method includes a key point processing method and a linear fitting method, and the determining of the contour outline of the part to be outlined includes:

[0182] Determine the key points of the mouth, the eyebrow and the facial contour in the image to be processed based on the key point processing method;

[0183] Based on the linear fitting method, the key points of the same part to be outlined are linearly fitted to obtain the contour outline of the corresponding part to be outlined.

[0184] According to one or more embodiments of the present disclosure, [Example 8] provides an image processing method, the method comprising:

[0185] Optionally, the part to be outlined includes an eye part, the processing method includes a key point processing method and a mask processing method, and the determining of a contour outline of the part to be outlined includes:

[0186] Based on the key point processing method, determine the iris key points of the eye part, and determine the iris schematic diagram according to the iris key points;

[0187] Determining an eye mask map according to eye key points and a mask processing method of the eye area;

[0188] Based on the eye mask image and the iris schematic diagram, a contour outline image of the eye area is determined.

[0189] According to one or more embodiments of the present disclosure, [Example 9] provides an image processing method, the method comprising:

[0190] Optionally, the part to be outlined includes a nose part, and determining a contour outline of the part to be outlined includes:

[0191] Determine nose wing key points, nose bridge key points, and nostril key points corresponding to the nose part according to the preset stroke information and key point processing method;

[0192] Based on the nose wing key points, the nose bridge key points, the nostril key points, a preset stroke template and a stroke adaptation method, a contour outline of the nose area is determined.

[0193] According to one or more embodiments of the present disclosure, [Example 10] provides an image processing method, the method comprising:

[0194] Optionally, the method is applied in a live broadcast scene or integrated into special effects props.

[0195] According to one or more embodiments of the present disclosure, [Example 1] provides an image processing device, characterized in that it includes:

[0196] An image acquisition module, configured to acquire an image to be processed including a target object in response to a special effect trigger operation; wherein the image to be processed includes at least one part to be outlined of the target object;

[0197] An image outlining module, configured to determine a contour outlining image of the part to be outlined according to at least two processing methods corresponding to different parts to be outlined;

[0198] The special effect image generation module is used to determine a display special effect image corresponding to the image to be processed based on the outline sketch.

[0199] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0200] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0201] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An image processing method, characterized in that: include: In response to a special effect triggering operation, an image to be processed including a target object is collected; wherein the image to be processed includes at least one part to be outlined of the target object; Determine a contour outline of the part to be outlined according to at least two processing methods corresponding to the part to be outlined; Based on the outline drawing, a display special effect drawing corresponding to the image to be processed is determined.

2. The method according to claim 1, characterized in that The part type corresponding to the at least one part to be outlined includes a first type or a second type, and the first type and the second type correspond to different processing methods.

3. The method according to claim 2, characterized in that The processing methods corresponding to the first type include at least two of a mask processing method, a key point processing method, and a binarization processing method, and the processing methods corresponding to the second type include at least two of a mask processing method, a key point processing method, a linear fitting method, and a template adaptation method.

4. The method according to claim 2, characterized in that: The part to be outlined includes at least one of limbs, a trunk, a hair part, at least one facial part, and a facial contour.

5. The method according to claim 4, characterized in that The part to be outlined is hair in the head part, the processing method is a binarization processing method and a mask image processing method, and the determining of the outline image of the part to be outlined includes: Processing the image to be processed according to the mask processing method to determine a hair mask image corresponding to the hair part; Binarizing the hair mask image according to a binarization processing method to obtain a hair binarization image; Based on the hair binary image, a contour outline image corresponding to the hair part is determined.

6. The method according to claim 5, characterized in that The step of determining a contour outline image corresponding to the hair part based on the hair binary image comprises: dilating the hair binary image to obtain a hair stroke image; Based on the hair stroke map and the hair binarization map, a contour outline map of the hair part is determined.

7. The method according to claim 4, characterized in that The part to be outlined includes at least one facial part, the at least one facial part includes a mouth part and an eyebrow part, the part to be outlined also includes a facial contour, the processing method includes a key point processing method and a linear fitting method, and the determining of the contour outline of the part to be outlined includes: Determine the key points of the mouth, the eyebrow and the facial contour in the image to be processed based on the key point processing method; Based on the linear fitting method, the key points of the same part to be outlined are linearly fitted to obtain the contour outline of the corresponding part to be outlined.

8. The method according to claim 4, characterized in that The part to be outlined includes an eye part, the processing method includes a key point processing method and a mask processing method, and the determining of the outline of the part to be outlined includes: Based on the key point processing method, determine the iris key points of the eye part, and determine the iris schematic diagram according to the iris key points; Determining an eye mask map according to eye key points and a mask processing method of the eye area; Based on the eye mask image and the iris schematic diagram, a contour outline image of the eye area is determined.

9. The method according to claim 4, characterized in that The part to be outlined includes a nose part, and determining a contour outline of the part to be outlined includes: Determine nose wing key points, nose bridge key points, and nostril key points corresponding to the nose part according to the preset stroke information and key point processing method; Based on the nose wing key points, the nose bridge key points, the nostril key points, a preset stroke template and a stroke adaptation method, a contour outline of the nose area is determined.

10. The method according to claim 1, characterized in that The method is applied in a live broadcast scene or integrated into special effects props.

11. An image processing device, characterized in that: include: An image acquisition module, configured to acquire an image to be processed including a target object in response to a special effect trigger operation; wherein the image to be processed includes at least one part to be outlined of the target object; An image outlining module, configured to determine a contour outlining image of the part to be outlined according to at least two processing methods corresponding to different parts to be outlined; The special effect image generation module is used to determine a display special effect image corresponding to the image to be processed based on the outline sketch.

12. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any one of claims 1 to 10.

13. A storage medium comprising computer executable instructions, wherein the computer executable instructions are used to perform the image processing method according to any one of claims 1 to 10 when executed by a computer processor.

Images