Image processing method and related equipment

Abstract

The invention provides an image processing method and related equipment. The method comprises the steps of obtaining a to-be-processed image and an adjustment instruction for the to-be-processed image; establishing corresponding three-dimensional data based on the to-be-processed image; determining a target position and a target color of a target object and a target position of a non-target object area based on the adjustment instruction and the three-dimensional data; and processing the three-dimensional data based on the target position and the target color of the target object and the target position of the non-target object area to obtain a target image.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image processing method and related equipment. Background Art

[0002] The effect of target object special effects in existing image processing is single, and the target object special effects often cause deformation of local areas, which does not conform to the characteristics of other areas, resulting in the target object area after the target object special effects are performed in the overall image being inconsistent with the overall image, resulting in poor image processing effect. Summary of the invention

[0003] The present disclosure proposes an image processing method, device, equipment, storage medium and program product to solve the technical problem of single and uncoordinated eye effects in image processing to a certain extent.

[0004] In a first aspect, the present disclosure provides an image processing method, comprising:

[0005] Acquire an image to be processed and an adjustment instruction for the image to be processed;

[0006] Establishing corresponding three-dimensional data based on the image to be processed;

[0007] Determine a target position, a target color, and a non-target object region target position of a target object based on the adjustment instruction and the three-dimensional data;

[0008] The three-dimensional data is processed based on the target position and the target color of the target object and the target position of the non-target object area to obtain a target image.

[0009] In a second aspect of the present disclosure, there is provided an image processing device, comprising:

[0010] A user interaction module, used to obtain an image to be processed and an adjustment instruction for the image to be processed;

[0011] A three-dimensional reconstruction module, used to establish corresponding three-dimensional data based on the image to be processed;

[0012] The special effects generation module is used to determine the target position, target color and target position of the non-target object area of ​​the target object based on the adjustment instruction and the three-dimensional data; and to process the three-dimensional data based on the target position, target color and target position of the non-target object area to obtain a target image.

[0013] According to a third aspect of the present disclosure, an electronic device is provided, characterized in that it includes one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the program includes instructions for executing the method described in the first aspect or the second aspect.

[0014] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the processors execute the method described in the first aspect or the second aspect.

[0015] In a fifth aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the method described in the first aspect.

[0016] From the above, it can be seen that the image processing method and related equipment provided by the present disclosure can enrich and improve the effect of the special effects of the target object by converting the image to be processed into three-dimensional data and adjusting the shape and color of the target object based on the adjustment instruction. At the same time, the non-target object area is processed based on the adjustment instruction, which can make the target object more coordinated with other areas, further improving the effect of image processing. Users can flexibly set the special effects of the target object through adjustment instructions, which improves the efficiency of the special effects processing of the target object and reduces the production cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 Schematic diagram of an image processing architecture according to an embodiment of the present disclosure.

[0019] Figure 2 The figure is a schematic diagram of the hardware structure of an exemplary electronic device according to an embodiment of the present disclosure.

[0020] Figure 3 The figure is a schematic flowchart of the image processing method according to the embodiment of the present disclosure.

[0021] Figure 4 Schematic diagram of an image processing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0023] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0024] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0025] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0026] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0027] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0028] Figure 1 FIG. 1 is a schematic diagram showing an image processing architecture of an embodiment of the present disclosure. Figure 1, the image processing architecture 100 may include a server 110, a terminal 120, and a network 130 providing a communication link. The server 110 and the terminal 120 may be connected via a wired or wireless network 130. The server 110 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, security services, and CDN.

[0029] The terminal 120 may be implemented in hardware or software. For example, when the terminal 120 is implemented in hardware, it may be various electronic devices having a display screen and supporting page display, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers, and desktop computers, etc. When the terminal 120 device is implemented in software, it may be installed in the electronic devices listed above; it may be implemented as multiple software or software modules (such as software or software modules used to provide distributed services), or it may be implemented as a single software or software module, which is not specifically limited here.

[0030] It should be noted that the image processing method provided in the embodiment of the present application can be executed by the terminal 120 or by the server 110. It should be understood that Figure 1 The number of terminals, networks and servers in the embodiment is only for illustration and is not intended to limit the number of terminals, networks and servers. Any number of terminals, networks and servers may be provided as required.

[0031] Figure 2 FIG. 2 shows a schematic diagram of the hardware structure of an exemplary electronic device 200 provided in an embodiment of the present disclosure. Figure 2 As shown, the electronic device 200 may include: a processor 202, a memory 204, a network module 206, a peripheral interface 208 and a bus 210. The processor 202, the memory 204, the network module 206 and the peripheral interface 208 are connected to each other in communication within the electronic device 200 through the bus 210.

[0032] Processor 202 may be a central processing unit (CPU), an image processor, a neural network processor (NPU), a microcontroller (MCU), a programmable logic device, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or one or more integrated circuits. Processor 202 may be used to perform functions related to the technology described in this disclosure. In some embodiments, processor 202 may also include multiple processors integrated into a single logical component. For example, Figure 2 As shown, processor 202 may include a plurality of processors 202a, 202b, and 202c.

[0033] The memory 204 may be configured to store data (eg, instructions, computer code, etc.). Figure 2 As shown, the data stored in the memory 204 may include program instructions (e.g., program instructions for implementing the image processing method of the embodiment of the present disclosure) and data to be processed (e.g., the memory may store configuration files of other modules, etc.). The processor 202 may also access the program instructions and data stored in the memory 204, and execute the program instructions to operate on the data to be processed. The memory 204 may include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 204 may include a random access memory (RAM), a read-only memory (ROM), an optical disk, a magnetic disk, a hard disk, a solid-state drive (SSD), a flash memory, a memory stick, etc.

[0034] The network module 206 can be configured to provide communication with other external devices to the electronic device 200 via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC) etc.), a cellular network, the Internet or a combination thereof. It is understood that the type of network is not limited to the above specific examples. In some embodiments, the network module 306 can include any number of network interface controllers (NICs), radio frequency modules, transceivers, modems, routers, gateways, adapters, cellular network chips, etc., in any combination.

[0035] The peripheral interface 208 can be configured to connect the electronic device 200 to one or more peripheral devices to achieve information input and output. For example, the peripheral devices can include input devices such as a keyboard, a mouse, a touch pad, a touch screen, a microphone, and various sensors, and output devices such as a display, a speaker, a vibrator, and an indicator light.

[0036] The bus 210 can be configured to transmit information between various components of the electronic device 200 (e.g., the processor 202, the memory 204, the network module 206, and the peripheral interface 208), such as an internal bus (e.g., a processor-memory bus), an external bus (USB port, PCI-E bus), etc.

[0037] It should be noted that, although the architecture of the electronic device 200 only shows the processor 202, the memory 204, the network module 206, the peripheral interface 208 and the bus 210, in the specific implementation process, the architecture of the electronic device 200 may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the architecture of the electronic device 200 may also only include the components necessary for implementing the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0038] Existing target object special effects are often implemented based on 2D key points. For example, in eye special effects, users can define pupil texture materials and detect 2D pupil key points according to algorithms, and draw pupil texture to the appropriate area to obtain output face images. However, this kind of eye special effect cannot simulate the physical eyeball reflection effect, and the special effects that can be achieved are relatively simple. Moreover, the deformation of the eyeball and eye socket as a whole in the output image will cause the deformed result to not conform to the overall characteristics, such as causing pupil distortion, squareness and other problems, resulting in poor image processing effect. Although there are some technologies for implementing target object special effects based on 3D reconstruction, it is difficult for users to simulate the thickness of the cornea in the existing illumination simulation method, and it is also difficult to adjust the shape of the target object. Another target object special effect method is to train the target object special effect based on the network, which can realize the input of the original image and the output of a target object special effect of a certain shape. However, if the user wants to readjust the texture, shape and other effects of the target object special effect, the model needs to be retrained, which not only has a high production cost, but also cannot adjust the target object special effect in real time, and the adjustment is not flexible. Therefore, how to improve the effect and efficiency of the special effects of the target object while reducing the production cost has become a technical problem that needs to be solved urgently.

[0039] In view of this, the embodiments of the present disclosure provide an image processing method and related devices. By converting the image to be processed into three-dimensional data and adjusting the shape and color of the target object based on the adjustment instruction, the effect of the special effect of the target object can be enriched and improved. At the same time, the non-target object area is processed based on the adjustment instruction, which can make the target object more coordinated with other areas, further improving the effect of image processing. Users can flexibly set the special effects of the target object through the adjustment instruction, which improves the efficiency of the special effect processing of the target object and reduces the production cost.

[0040] See also Figure 3 , Figure 3 A schematic flowchart of an image processing method according to an embodiment of the present disclosure is shown. Figure 3 In the embodiment, the image processing method 300 may further include the following steps.

[0041] In step S310, an image to be processed and an adjustment instruction for the image to be processed are obtained.

[0042] Among them, the image to be processed may include a two-dimensional face image, and the two-dimensional face image may refer to a two-dimensional image including a face. The image to be processed may be uploaded locally or obtained via a network. The adjustment instruction may be triggered by a user operation based on a user interaction interface, and the adjustment instruction may include a target object special effect parameter and a non-target object special effect parameter. For example, the target object may include an eyeball, then the target object feature parameter may refer to an eyeball special effect parameter, and the eyeball special effect parameter may refer to parameters related to display effects such as the shape, size, color (for example, including texture color and / or brightness), direction, and degree of special effect mixing of the eyeball area. At this time, the non-target object special effect parameter may refer to a special effect parameter of an area outside the eyeball area, for example, a facial special effect parameter, and the facial special effect parameter may refer to parameters related to the shape, size, color (for example, including texture color and / or brightness), direction, etc. of the non-eyeball area. Specifically, the target object special effect parameter and the non-target object special effect parameter may be set by the user based on the interaction interface. For example, if the target object includes an eyeball, one or more eyeball special effect parameters and one or more facial special effect parameters can be presented to the user in the form of a slider or options based on the interactive interface, as well as multiple values ​​corresponding to each eyeball special effect parameter and facial special effect parameter for the user to select, and the user can select the final parameter value from them based on the user operation (such as clicking). The parameter values ​​of each eyeball special effect parameter and facial special effect parameter can also be directly input based on the input box in the interactive interface.

[0043] In step S320, corresponding three-dimensional data is established based on the image to be processed.

[0044] Wherein, the image to be processed may include a plurality of feature points, and the corresponding three-dimensional data may be reconstructed according to these feature points, for example, a three-dimensional variability model (3DMM) may be used for reconstruction. Specifically, the image to be processed may be fitted based on a general 3DMM model to obtain three-dimensional data corresponding to the image to be processed, for example, the three-dimensional data may include a three-dimensional face image. In the process of reconstructing the three-dimensional data, the feature points of the image to be processed may form a mapping relationship with the vertices of the grid of the three-dimensional data, and the mapping relationship may be represented by a transformation matrix MVP. Based on the three-dimensional data, three-dimensional target object data and three-dimensional non-target object data may be determined, for example, the three-dimensional target object data and the three-dimensional non-target object data may respectively include the three-dimensional coordinates of the target object area and the non-target object area in the three-dimensional data in the three-dimensional coordinate system, and the three-dimensional coordinate system may include a first coordinate direction, a second coordinate direction and a third coordinate direction, and the first coordinate direction, the second coordinate direction and the third coordinate direction may be perpendicular to each other, for example, an xyz coordinate system.

[0045] In step S330, a target position, a target color, and a target position of a non-target object area of ​​a target object are determined based on the adjustment instruction and the three-dimensional data.

[0046] Among them, the shape and color of the target object (such as the eyeball) area can be adjusted by setting the adjustment instruction. Since the surface of the three-dimensional data can include multiple polygonal networks (such as triangular meshes or quadrilateral meshes), the target object special effect parameters can adjust the vertex positions of multiple meshes to change the shape of the meshes, thereby achieving changes in the shape, size, etc. of the target object. Specifically, the target object characteristic parameters may include eyeball special effect parameters, and the eyeball special effect parameters may also include parameters related to the iris and cornea to simulate the lighting effect of a real eyeball, thereby achieving a more realistic eyeball special effect.

[0047] In some embodiments, the eyeball special effect parameters include eyeball deformation parameters, eyeball material parameters and corneal thickness parameters. The adjustment instruction may include eyeball special effect parameters.

[0048] In some embodiments, determining a target position and a target color of a target object based on the adjustment instruction and the three-dimensional data further includes:

[0049] Determining a target position of the target object after the target object is deformed based on the target object deformation parameter and the initial position of the target object in the three-dimensional data, and obtaining a deformed image of the target object;

[0050] A target color of the target object is determined based on the target object material parameter, the corneal thickness parameter and the target object deformation image.

[0051] Among them, the target object deformation parameter can be used to indicate the deformation degree of the target object, and the target position of the target object after the target object is deformed can be determined accordingly. The target object material parameter can be used to indicate the fill color of the target object, and combined with the corneal thickness parameter, the target color of the target object under the lighting conditions of the eyeball can be determined.

[0052] In some embodiments, the surface of the three-dimensional data includes a plurality of meshes and vertices between the meshes; and the target object deformation parameters include a target object scaling parameter, an iris scaling parameter, and a target object displacement parameter.

[0053] The target object scaling parameter may indicate the scaling multiple of the target object. For example, if the target object scaling parameter is a, the deformed target object area is a times the original target object area in the three-dimensional data. When a is greater than 1, it indicates enlargement, and when a is less than 1, it indicates reduction. The iris scaling parameter may indicate the scaling multiple of the iris, and the target object displacement parameter may indicate the displacement of the target object.

[0054] In some embodiments, determining the target position of the target object after the target object is deformed based on the target object deformation parameter and the initial position of the target object in the three-dimensional data further includes:

[0055] Determine the shape of the target object based on the target object scaling parameter and the iris scaling parameter, and obtain a first position of the target object after deformation;

[0056] Determine a second position of the target object after displacement based on the target object displacement parameter and the first position;

[0057] Based on the mapping relationship between the image to be processed and the three-dimensional data, the second position is transformed to obtain a target position of the target object.

[0058] The mapping relationship between the image to be processed and the three-dimensional data may refer to the transformation matrix MVP in the process of establishing the three-dimensional data. The third coordinate and the fourth coordinate may be obtained based on the target object scaling parameter, the target object displacement parameter, the iris scaling parameter, and the first coordinate of the vertex of the target object area of ​​the three-dimensional data in the first coordinate direction and the second coordinate in the second coordinate direction; the sixth coordinate may be obtained based on the target object scaling parameter, the target object displacement parameter, the first coordinate, the second coordinate, and the fifth coordinate of the vertex of the target object area of ​​the three-dimensional data in the third coordinate direction; the target position of the target object may be obtained based on the mapping relationship between the image to be processed and the three-dimensional face image, as well as the third coordinate, the fourth coordinate, and the sixth coordinate.

[0059] In some embodiments, determining the shape of the target object based on the target object scaling parameter and the iris scaling parameter, and obtaining the first position of the target object after deformation, further comprises:

[0060] Determine the shape of the iris region in the target object based on the iris scaling parameter to obtain the target object after iris processing;

[0061] The target object after iris processing is scaled based on the target object scaling parameter to obtain the first position of the target object after deformation.

[0062] In some embodiments, determining the target position of the target object after deformation based on the target object deformation parameter and the initial position of the target object in the three-dimensional data to obtain a deformed image of the target object further includes:

[0063] Calculate a first difference between 1 and the preset weight of the iris scaling parameter, and a first product of the first difference and the first coordinate; calculate a second product of the first coordinate, the iris scaling parameter and the preset weight; calculate a first sum of the first product and the second product, and a third product of the first sum and the target object scaling parameter; calculate the sum of the third product and the first displacement parameter of the target object displacement parameter in the first coordinate direction to obtain the third coordinate;

[0064] Calculating a fourth product of the first difference and the second coordinate; calculating a fifth product of the second coordinate, the iris scaling parameter and the preset weight; calculating a second sum of the fourth product and the fifth product, and a sixth product of the second sum and the target object scaling parameter; calculating a sum of the sixth product and a second displacement parameter of the target object displacement parameter in the direction of the second coordinate to obtain the fourth coordinate;

[0065] Calculating a seventh product of the first difference and the fifth coordinate; calculating the sum of the squares of the first coordinate and the second coordinate, and the square root of the difference between the preset radius of the target object and the sum of the squares; calculating an eighth product of the square root and the preset weight, and a ninth product of the eighth product and the scaling parameter of the target object; obtaining the sixth coordinate based on the ninth product and the sum of the third displacement parameter of the displacement parameter of the target object in the third coordinate direction;

[0066] The target object deformed image is obtained based on the third coordinate, the fourth coordinate, the sixth coordinate and the tenth product of the transformation matrix, and the transformation matrix is ​​obtained when the three-dimensional data is constructed based on the image to be processed.

[0067] Specifically, the target object scaling parameter is s1, the iris scaling parameter is s2, the target object displacement parameter is △s (△x, △y, △z), the vertex position of the target object area in the three-dimensional data is Po (x, y, z), the center of the target object area is located at the origin of the coordinate system, and the direction of the three-dimensional data is along the positive direction of the z axis of the coordinate system, such as the direction opposite to the user. The preset weight w of the iris scaling parameter s2 can indicate the degree to which the vertex of the eyeball area is affected by the iris scaling parameter s2. The target position Pd (x, y, z) of the target object in the target object deformation image may include:

[0068] P1.x=(Po.x*(1-w)+Po.x*s2*w)*s1+△x;

[0069] P1.y=(Po.y*(1-w)+Po.y*s2*w)*s1+△y;

[0070] P1.z=(Po.z*(1-w)+sqrt(r-Po.x*Po.x-Po.y*Po.y)*w)*s1+△z;

[0071] Pd(x, y, z)=MVP*(P1.x, P1.y, P1.z).

[0072] It can be seen that the vertex position of the target object area in the three-dimensional data can be Pd(x, y, z) after deformation based on the target object deformation parameters, and a deformed image of the target object is obtained. Due to the mapping relationship between the image to be processed and the three-dimensional data, and the corresponding relationship between the vertex position Po(x, y, z) of the target object before deformation and the vertex position Pd(x, y, z) of the target object after deformation, the texture of the image to be processed can be mapped to the deformed three-dimensional data.

[0073] In some embodiments, the target object material parameters include target object texture direction parameters and target object rendering parameters, and the target object rendering parameters include at least one of the following: target object texture material parameters, light source color parameters, light source direction parameters, target object roughness, target object metalness or blending mode parameters.

[0074] Among them, the target object texture direction parameter can indicate the texture direction of the target object area, such as the normal map parameter n1. The target object texture material parameter can indicate the fill color or pattern of the target object area, such as the starry sky texture material, etc. The light source color parameter can indicate the color of the light source, such as white light, yellow light, etc. The light source direction parameter can indicate the direction of the light, such as the angle α between the direction of the incident light and the horizontal plane. The target object roughness can indicate the smoothness of the eyeball area, for example, the greater the eyeball roughness, the greater the roughness of the visual effect presented; the smaller the eyeball roughness, the smoother the visual effect presented. The target object metalness can indicate the degree of light concentration, for example, the greater the target object metalness, the brighter the visual effect presented; the smaller the target object metalness, the darker the visual effect presented. The mixed mode parameter can indicate the degree of fusion of the special effect and the original image. The larger the mixed mode parameter, the more obvious the special effect, and the smaller the mixed mode parameter, the less obvious the special effect.

[0075] In some embodiments, determining the target color of the target object based on the target object material parameter, the corneal thickness parameter and the target object deformation image further includes:

[0076] Obtaining a first target color based on the target object material parameter, the target object deformation image and a preset illumination function;

[0077] Obtaining a second target color based on a normalized corneal thickness parameter, a material parameter of the target object, a normal of the grid of the three-dimensional data, a deformation image of the target object, and the preset illumination function;

[0078] The target color is obtained based on the sum of the first target color and the second target color.

[0079] In some embodiments, the target object may include an eyeball. Further, in some embodiments, determining the target color of the target object based on the target object material parameter, the corneal thickness parameter, and the target object deformation image may include: determining the target color of the eyeball based on the eyeball material parameter, the corneal thickness parameter, and the eyeball deformation image.

[0080] In some embodiments, determining the target color of the eyeball based on the eyeball material parameter, the corneal thickness parameter and the eyeball deformation image further includes:

[0081] Calculating a normal function of eyeball texture direction parameters to obtain a first normal; and calculating an illumination function of the eyeball deformed face image, the eyeball rendering parameters and the first normal to obtain a first target color;

[0082] Calculating a second difference between 1 and the normalized corneal thickness parameter, and an eleventh product of the second difference and the first normal; calculating a twelfth product between the second normal of the grid of the three-dimensional face image and the normalized corneal thickness parameter; and obtaining a second target color based on an illumination function regarding the eyeball deformed face image, the eyeball rendering parameter, and the sum of the eleventh product and the twelfth product;

[0083] The eyeball target color is obtained based on the sum of the first target color and the second target color.

[0084] Specifically, the eyeball texture direction parameter (such as normal map) can be n1, the eyeball rendering parameter can be m, the normal of the grid in the three-dimensional data can be n2, the normalized corneal thickness can be d, d∈[0.0, 1.0], N is the texture direction function (i.e., normal function) converted from the eyeball texture direction parameter, and F is the preset illumination model. Then the color of the eyeball area can be calculated based on this to achieve a multi-layer illumination effect caused by corneal thickness. The eyeball target color C of each network in the eyeball area in the target object deformation image Se can include:

[0085] C1=F(Se,m,N(n1));

[0086] C2=F(Se,m,N(n1)*(1-d)+n2*d);

[0087] C=C1+C2.

[0088] In some embodiments, the adjustment instruction further includes: a first state parameter, a second state parameter, and a regional deformation parameter. The first state parameter may be a deformation parameter of a first part of the non-target object area, the second state parameter may be a deformation parameter of a second part of the non-target object area, and the regional deformation parameter may be a deformation parameter of a third part of the non-target object area. For example, the first part may refer to the left eye socket, the second part may refer to the right eye socket, and the third part may refer to other areas of the face.

[0089] In some embodiments, determining a target position of a non-target object area based on the adjustment instruction and the three-dimensional data further includes:

[0090] Determine a first target position of a first part and a second target position of a second part in the non-target object area based on the first state parameter and the second state parameter respectively;

[0091] Determine the shape of a third part in the non-target object area based on the regional deformation parameter, and obtain a target position of the third part;

[0092] The non-target object area target position is obtained based on the first part target position, the second part target position and the third part target position.

[0093] In some embodiments, determining a first target position of a first part and a second target position of a second part in a non-target object region based on the first state parameter and the second state parameter respectively further comprises:

[0094] Determining a first state angle of a first portion and a second state angle of a second portion in a non-target object region based on the three-dimensional data;

[0095] Determine the shape of the first part based on the first state angle and the first state parameter, and obtain the target position of the first part;

[0096] The shape of the second part is determined based on the second state angle and the second state parameter to obtain the target position of the second part.

[0097] Among them, since the target object area is deformed and color processed based on the target object special effect parameters, in order to avoid the mismatch between the target object and other areas of the image, the non-target object area can be adjusted accordingly based on the first state parameters, second state parameters and regional deformation parameters of the non-target object, thereby improving the overall coordination of the image.

[0098] Specifically, the target object may include an eyeball. Accordingly, the adjustment instruction includes non-target object special effect parameters, which may include left eye open deformation parameters, which may refer to deformation parameters of the left eye in an open state; right eye open deformation parameters may refer to deformation parameters of the right eye in an open state; eye deformation parameters may refer to deformation parameters of the entire eye. Among them, the angle may refer to the angle between the first line between the center position of the eyeball and the highest feature point of the eye (such as the highest feature point of the left eye or the highest feature point of the right eye), and the second line between the center position of the eyeball and the lowest feature point of the eye (such as the lowest feature point of the left eye or the lowest feature point of the right eye).

[0099] Furthermore, in some embodiments, determining a target position of a non-target object area based on the adjustment instruction and the three-dimensional data further includes:

[0100] Determine the left eye opening angle and the right eye opening angle based on the three-dimensional data;

[0101] Calculating a left eye opening position based on the left eye opening deformation parameter and the vertex position of the three-dimensional data, and obtaining a first target position based on the product of the left eye opening position and the left eye opening angle;

[0102] Calculating a right eye opening position based on the right eye opening deformation parameter and the vertex position of the three-dimensional data, and obtaining a second target position based on the product of the right eye opening position and the right eye opening angle;

[0103] Calculating a third target position based on the eye deformation parameter and the vertex position of the three-dimensional data;

[0104] The non-target object area target position is determined based on the first target position, the second target position, and the third target position.

[0105] Specifically, the adjustment instruction may include left eye opening deformation parameter BlendShapes1, right eye opening deformation parameter BlendShapes2 and eye deformation parameter BlendShapes3. The left eye opening angle d1 and the right eye opening angle d2 may be determined according to the three-dimensional data. The vertex position of the three-dimensional data (i.e. before deformation) is Pfo (x, y, z). Then, the vertex position Pfd (x, y, z) of the non-target object after deformation processing may include:

[0106] Pfd=BlendShapes3(Pfo)+BlendShapes1(Pfo)*d1+BlendShapes2(Pfo)*d2.

[0107] It can be seen that the vertex position Pfd(x, y, z) of the non-target object after deformation processing is the target position of the non-target object area.

[0108] In step S340, the three-dimensional data is processed based on the target position and the target color of the target object and the target position of the non-target object area to obtain a target image.

[0109] In some embodiments, the three-dimensional data is processed based on the target position, the target color of the target object and the target position of the non-target object area to obtain a target image, including:

[0110] Rendering to obtain a target object area of ​​the target image based on the target position and the target color;

[0111] The non-target object region of the target image is obtained based on the target position of the non-target object region and the texture rendering of the non-target region of the image to be processed.

[0112] Specifically, for example, if the target object includes an eyeball, the positions of the vertices of the eyeball area can be changed to the target position, and the eyeball area can be rendered based on the target color of the corresponding grid. For the non-eyeball area, the vertex position of the non-eyeball area can be changed to the non-eyeball area target position, and based on the correspondence between the vertex position and the feature point of the image to be processed, the texture pattern or color of the image to be processed is rendered in the non-eyeball area, thereby obtaining a three-dimensional target image that has been processed with eyeball special effects and deformed in the non-eyeball area. Furthermore, the three-dimensional target image can be converted into a two-dimensional image based on the transformation matrix MVP to obtain the target image.

[0113] It can be seen that the image processing method according to the embodiment of the present disclosure can not only enrich and improve the effect of the eyeball special effects, but also process the facial area outside the eyeball at the same time, which can make the eyeball and other areas of the face more coordinated, and further improve the effect of image processing. Users can flexibly set the eyeball special effects through eyeball special effects parameters and facial special effects parameters, which improves the efficiency of eyeball special effects processing and reduces production costs.

[0114] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0115] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0116] Based on the same technical concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure further provides an image processing device, the image processing device comprising:

[0117] A user interaction module, used to obtain an image to be processed and an adjustment instruction for the image to be processed;

[0118] A three-dimensional reconstruction module, used to establish corresponding three-dimensional data based on the image to be processed;

[0119] The special effects generation module is used to determine the target position, target color and target position of the non-target object area of ​​the target object based on the adjustment instruction and the three-dimensional data; and to process the three-dimensional data based on the target position, target color and target position of the non-target object area to obtain a target image.

[0120] In some embodiments, the user interaction module is further used to output the target image, for example, display the target image in an interactive interface.

[0121] Specifically, see Figure 4 , Figure 4 A schematic diagram of an image processing device according to an embodiment of the present disclosure is shown. Figure 4 In the process, the user can determine the two-dimensional image to be processed based on the user interaction module, and the three-dimensional reconstruction module can reconstruct the input image to be processed into a three-dimensional model to obtain the corresponding three-dimensional time data; the special effect generation module can perform eyeball deformation and color processing on the three-dimensional data based on the three-dimensional data and the adjustment information of the target object and non-target object deformation input by the user based on the user interaction module, as well as the eyeball material parameters and corneal thickness information set by the user, to generate the target object special effect. And adjust the overall area to generate a three-dimensional target image. Finally, the three-dimensional target image is converted into a two-dimensional target image output, and the two-dimensional target image can be a two-dimensional image after the target object special effect processing.

[0122] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0123] The device of the above embodiment is used to implement the corresponding image processing method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0124] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the image processing method described in any of the above embodiments.

[0125] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0126] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the image processing method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0127] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0128] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the known power / ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0129] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0130] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. An image processing method, the method comprising: Acquire an image to be processed and an adjustment instruction for the image to be processed; Establishing corresponding three-dimensional data based on the image to be processed; Determine a target position, a target color, and a non-target object region target position of a target object based on the adjustment instruction and the three-dimensional data; The three-dimensional data is processed based on the target position and the target color of the target object and the target position of the non-target object area to obtain a target image.

2. The method according to claim 1, wherein The adjustment instruction includes: a target object deformation parameter, a target object material parameter and a corneal thickness parameter of the target object; Then determining the target position and the target color of the target object based on the adjustment instruction and the three-dimensional data further includes: Determining a target position of the target object after the target object is deformed based on the target object deformation parameter and the initial position of the target object in the three-dimensional data, and obtaining a deformed image of the target object; A target color of the target object is determined based on the target object material parameter, the corneal thickness parameter and the target object deformation image.

3. The method according to claim 2, wherein: The surface of the three-dimensional data includes a plurality of grids and vertices between the grids; the target object deformation parameters include a target object scaling parameter, an iris scaling parameter and a target object displacement parameter; Then determining the target position of the target object after the target object is deformed based on the target object deformation parameter and the initial position of the target object in the three-dimensional data further includes: Determine the shape of the target object based on the target object scaling parameter and the iris scaling parameter, and obtain a first position of the target object after deformation; Determine a second position of the target object after displacement based on the target object displacement parameter and the first position; Based on the mapping relationship between the image to be processed and the three-dimensional data, the second position is transformed to obtain a target position of the target object.

4. The method according to claim 3, determining the shape of the target object based on the target object scaling parameter and the iris scaling parameter to obtain the first position of the target object after deformation, further comprising: Determine the shape of the iris region in the target object based on the iris scaling parameter to obtain the target object after iris processing; The target object after iris processing is scaled based on the target object scaling parameter to obtain the first position of the target object after deformation.

5. The method according to claim 2, wherein: Determining the target color of the target object based on the target object material parameter, the corneal thickness parameter and the target object deformation image further includes: Obtaining a first target color based on the target object material parameter, the target object deformation image and a preset illumination function; Obtaining a second target color based on a normalized corneal thickness parameter, a material parameter of the target object, a normal of the grid of the three-dimensional data, a deformation image of the target object, and the preset illumination function; The target color is obtained based on the sum of the first target color and the second target color.

6. The method according to claim 1, wherein: The adjustment instruction also includes: a first state parameter, a second state parameter and a regional deformation parameter; Then determining the target position of the non-target object area based on the adjustment instruction and the three-dimensional data further includes: Determine a first target position of a first part and a second target position of a second part in the non-target object area based on the first state parameter and the second state parameter respectively; Determine the shape of a third part in the non-target object area based on the regional deformation parameter, and obtain a target position of the third part; The non-target object area target position is obtained based on the first part target position, the second part target position and the third part target position.

7. The method according to claim 1, wherein: The three-dimensional data is processed based on the target position, the target color of the target object and the target position of the non-target object area to obtain a target image, including: Rendering to obtain a target object area of ​​the target image based on the target position and the target color; The non-target object region of the target image is obtained based on the target position of the non-target object region and the texture rendering of the non-target region of the image to be processed.

8. An image processing device, comprising: A user interaction module, used to obtain an image to be processed and an adjustment instruction for the image to be processed; A three-dimensional reconstruction module, used to establish corresponding three-dimensional data based on the image to be processed; A special effect generation module, used to determine a target position, a target color, and a target position of a non-target object area of ​​a target object based on the adjustment instruction and the three-dimensional data; And the three-dimensional data is processed based on the target position and target color of the target object and the target position of the non-target object area to obtain a target image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program. 10 . A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to claim 1 .

Images