Image processing method and related device

Abstract

Provided in the present disclosure are an image processing method and a related device. The method comprises: acquiring an image to be processed, and an adjustment instruction for said image; on the basis of said image, building corresponding three-dimensional data; on the basis of the adjustment instruction and the three-dimensional data, determining a target position and a target color of a target object and a target position of a non-target object region; and on the basis of the target position and the target color of the target object and the target position of the non-target object region, processing the three-dimensional data to obtain a target image.

Description

Image processing method and related equipment

[0001] This application claims priority to the Chinese invention patent application entitled “Image Processing Method and Related Equipment” and application number 202311120184.0 filed on August 31, 2023, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of computer technology, and in particular to an image processing method and related equipment. Background Art

[0003] The effect of target object special effects in existing image processing is single, and target object special effects often cause deformation of local areas, which does not conform to the characteristics of other areas, resulting in the target object area after the target object special effects in the overall image being inconsistent with the overall image, resulting in poor image processing effect.

[0004] Summary of the Invention

[0005] The present disclosure provides an image processing method, apparatus, device, storage medium, and program product.

[0006] In a first aspect of the present disclosure, an image processing method is provided, comprising: acquiring an image to be processed and an adjustment instruction for the image to be processed; establishing corresponding three-dimensional data based on the image to be processed; determining a target position, a target color, and a target position of a non-target object area of ​​a target object based on the adjustment instruction and the three-dimensional data; and processing the three-dimensional data based on the target position, the target color, and the target position of the non-target object area to obtain a target image.

[0007] In a second aspect of the present disclosure, an image processing device is provided, comprising: a user interaction module for acquiring an image to be processed and an adjustment instruction for the image to be processed; a three-dimensional reconstruction module for establishing corresponding three-dimensional data based on the image to be processed; a special effect generation module for determining a target position, a target color, and a target position of a non-target object area of ​​a target object based on the adjustment instruction and the three-dimensional data; and processing the three-dimensional data based on the target position, target color, and target position of the non-target object area to obtain a target image.

[0008] In a third aspect of the present disclosure, an electronic device is provided, comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method according to the first aspect or the second aspect.

[0009] In a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the processors are caused to execute the method described in the first aspect or the second aspect.

[0010] In a fifth aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the method according to the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] FIG1 is a schematic diagram of an image processing architecture according to an embodiment of the present disclosure;

[0013] FIG2 is a schematic diagram of the hardware structure of an exemplary electronic device according to an embodiment of the present disclosure;

[0014] FIG3 is a schematic flow chart of an image processing method according to an embodiment of the present disclosure; and

[0015] FIG4 is a schematic diagram of an image processing apparatus according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0017] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the described object changes, the relative position relationship may also change accordingly.

[0018] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0019] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0020] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0021] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0022] FIG1 shows a schematic diagram of an image processing architecture according to an embodiment of the present disclosure. Referring to FIG1 , the image processing architecture 100 may include a server 110, a client 120, and a network 130 providing a communication link. The server 110 and the client 120 may be connected via a wired or wireless network 130. The server 110 may be an independent physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, security services, and CDN.

[0023] The client 120 can be implemented in hardware or software. For example, if the client 120 is implemented in hardware, it can be any electronic device with a display screen that supports page display, including but not limited to smartphones, tablet computers, e-book readers, laptop computers, and desktop computers. If the client 120 device is implemented in software, it can be installed in the electronic devices listed above; it can be implemented as multiple software or software modules (such as software or software modules used to provide distributed services), or it can be implemented as a single software or software module, and no specific limitations are given here.

[0024] It should be noted that the image processing method provided in the embodiments of the present application can be executed by the client 120 or the server 110. It should be understood that the number of clients, networks, and servers in FIG1 is for illustration only and is not intended to be limiting. Any number of clients, networks, and servers may be used as required.

[0025] Figure 2 shows a schematic diagram of the hardware structure of an exemplary electronic device 200 provided by an embodiment of the present disclosure. As shown in Figure 2, electronic device 200 may include: a processor 202, a memory 204, a network module 206, a peripheral interface 208, and a bus 210. The processor 202, memory 204, network module 206, and peripheral interface 208 are communicatively connected to each other within electronic device 200 via bus 210.

[0026] The processor 202 may be a central processing unit (CPU), an image processor, a neural network processor (NPU), a microcontroller (MCU), a programmable logic device (PLD), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or one or more integrated circuits. The processor 202 may be used to perform functions related to the technology described in this disclosure. In some embodiments, the processor 202 may also include multiple processors integrated into a single logical component. For example, as shown in FIG2 , the processor 202 may include multiple processors 202 a, 202 b, and 202 c.

[0027] The memory 204 can be configured to store data (e.g., instructions, computer codes, etc.). As shown in Figure 2, the data stored in the memory 204 can include program instructions (e.g., program instructions for implementing the image processing method of the embodiment of the present disclosure) and data to be processed (e.g., the memory can store configuration files of other modules, etc.). The processor 202 can also access the program instructions and data stored in the memory 204, and execute the program instructions to operate on the data to be processed. The memory 204 can include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 204 can include a random access memory (RAM), a read-only memory (ROM), an optical disc, a magnetic disk, a hard disk, a solid-state drive (SSD), a flash memory, a memory stick, etc.

[0028] The network module 206 can be configured to provide the electronic device 200 with communication with other external devices via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC)), a cellular network, the Internet, or a combination thereof. It will be appreciated that the type of network is not limited to the specific examples above. In some embodiments, the network module 306 can include any number of network interface controllers (NICs), radio frequency modules, transceivers, modems, routers, gateways, adapters, cellular network chips, and the like.

[0029] The peripheral interface 208 can be configured to connect the electronic device 200 to one or more peripheral devices to implement information input and output. For example, the peripheral devices can include input devices such as a keyboard, a mouse, a touchpad, a touch screen, a microphone, and various sensors, and output devices such as a display, a speaker, a vibrator, and an indicator light.

[0030] The bus 210 can be configured to transmit information between the various components of the electronic device 200 (e.g., the processor 202, the memory 204, the network module 206, and the peripheral interface 208), such as an internal bus (e.g., a processor-memory bus), an external bus (USB port, PCI-E bus), etc.

[0031] It should be noted that although the architecture of the electronic device 200 shown above only includes the processor 202, the memory 204, the network module 206, the peripheral interface 208, and the bus 210, in a specific implementation, the architecture of the electronic device 200 may also include other components necessary for normal operation. In addition, those skilled in the art will understand that the architecture of the electronic device 200 may also include only the components necessary to implement the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0032] Existing object-specific effects are often implemented based on 2D keypoints. For example, for eye effects, users can define pupil texture materials and use algorithms to detect 2D pupil keypoints. The pupil texture is then mapped to the appropriate area to produce the output facial image. However, these eye effects fail to simulate the physically based reflections of the eyeball, resulting in relatively limited effects. Furthermore, deforming the eyeball and orbital components of the output image can lead to distortions that mismatch the overall characteristics, such as pupil distortion or squareness, resulting in poor image processing. While some technologies exist for object-specific effects based on 3D reconstruction, existing illumination simulation methods make it difficult to simulate corneal thickness and adjust the shape of the object. Another approach to object-specific effects relies on network training. This allows users to input an original image and output an object-specific effect of a specific shape. However, re-adjusting the texture, shape, and other aspects of the object-specific effect requires retraining the model, resulting in high production costs and inflexible adjustments without the ability to adjust the effect in real time. Therefore, how to improve the effectiveness and efficiency of special effects for target objects while reducing production costs has become a technical problem that needs to be solved urgently.

[0033] In view of this, embodiments of the present disclosure provide an image processing method and related apparatus. By converting the image to be processed into three-dimensional data and adjusting the shape and color of the target object based on adjustment instructions, the special effects of the target object can be enriched and enhanced. At the same time, non-target object areas are processed based on the adjustment instructions, which can make the target object more coordinated with other areas, further improving the image processing effect. Users can flexibly set the special effects of the target object through adjustment instructions, improving the efficiency of target object special effects processing and reducing production costs.

[0034] Referring to Figure 3 , which shows a schematic flow chart of an image processing method according to an embodiment of the present disclosure, the image processing method 300 in Figure 3 may further include the following steps.

[0035] In step S310, an image to be processed and an adjustment instruction for the image to be processed are obtained.

[0036] Among them, the image to be processed may include a two-dimensional face image, and the two-dimensional face image may refer to a two-dimensional image including a face. The image to be processed may be uploaded locally or obtained via a network. The adjustment instruction may be triggered by a user operation based on a user interaction interface, and the adjustment instruction may include target object special effect parameters and non-target object special effect parameters. For example, the target object may include an eyeball, then the target object feature parameter may refer to an eyeball special effect parameter, and the eyeball special effect parameter may refer to parameters related to the display effect such as the shape, size, color (for example, including texture color and / or brightness), direction, and degree of special effect mixing of the eyeball area. At this time, the non-target object special effect parameter may refer to the special effect parameter of an area outside the eyeball area, for example, the facial special effect parameter, and the facial special effect parameter may refer to parameters related to the shape, size, color (for example, including texture color and / or brightness), direction, etc. of the non-eyeball area. Specifically, the target object special effect parameter and the non-target object special effect parameter may be set by the user based on the interaction interface. For example, if the target object includes an eyeball, one or more eyeball special effect parameters and one or more facial special effect parameters can be presented to the user through a slider or options on the interactive interface, as well as multiple values ​​corresponding to each eyeball special effect parameter and facial special effect parameter for the user to select. The user can select the final parameter value from them based on a user operation (such as a click). Parameter values ​​for each eyeball special effect parameter and facial special effect parameter can also be directly input through input boxes in the interactive interface.

[0037] In step S320, corresponding three-dimensional data is created based on the image to be processed.

[0038] The image to be processed may include multiple feature points, and corresponding three-dimensional data can be reconstructed based on these feature points, for example, a three-dimensional variability model (3DMM) can be used for reconstruction. Specifically, a general 3DMM model can be used to fit the image to be processed to obtain three-dimensional data corresponding to the image to be processed, for example, the three-dimensional data may include a three-dimensional face image. In the process of reconstructing the three-dimensional data, the feature points of the image to be processed can form a mapping relationship with the vertices of the grid of the three-dimensional data, and the mapping relationship can be represented by a transformation matrix MVP. Based on the three-dimensional data, three-dimensional target object data and three-dimensional non-target object data can be determined, for example, the three-dimensional target object data and the three-dimensional non-target object data can respectively include the three-dimensional coordinates of the target object area and the non-target object area in the three-dimensional data in a three-dimensional coordinate system, and the three-dimensional coordinate system can include a first coordinate direction, a second coordinate direction and a third coordinate direction, and the first coordinate direction, the second coordinate direction and the third coordinate direction can be perpendicular to each other, for example, an xyz coordinate system.

[0039] In step S330 , a target position, a target color, and a target position of a non-target object area of ​​the target object are determined based on the adjustment instruction and the three-dimensional data.

[0040] Among them, the shape and color of the target object (such as the eyeball) area can be adjusted by setting adjustment instructions. Since the surface of the three-dimensional data can include multiple polygonal networks (such as triangular meshes or quadrilateral meshes), the target object special effect parameters can adjust the vertex positions of multiple meshes to change the shape of the meshes, thereby achieving changes in the shape, size, etc. of the target object. Specifically, the target object characteristic parameters may include eyeball special effect parameters, and the eyeball special effect parameters may also include parameters related to the iris and cornea to simulate the lighting effect of a real eyeball, thereby achieving a more realistic eyeball special effect.

[0041] In some embodiments, the eyeball special effect parameters include eyeball deformation parameters, eyeball material parameters, and corneal thickness parameters. The adjustment instruction may include the eyeball special effect parameters.

[0042] In some embodiments, determining a target position and a target color of a target object based on the adjustment instruction and the three-dimensional data further includes:

[0043] determining a target position of the target object after deformation based on the target object deformation parameter and the initial position of the target object in the three-dimensional data, and obtaining a deformed image of the target object;

[0044] A target color of the target object is determined based on the target object material parameter, the corneal thickness parameter, and the target object deformation image.

[0045] The target object deformation parameter can be used to indicate the degree of deformation of the target object, and the target position of the target object after deformation can be determined accordingly. The target object material parameter can be used to indicate the fill color of the target object, and combined with the corneal thickness parameter, the target color of the target object under the lighting conditions of the eyeball can be determined.

[0046] In some embodiments, the surface of the three-dimensional data includes a plurality of meshes and vertices between the meshes; and the target object deformation parameters include a target object scaling parameter, an iris scaling parameter, and a target object displacement parameter.

[0047] The target object scaling parameter may indicate the scaling factor of the target object. For example, if the target object scaling parameter is a, the deformed target object area is a times the original target object area in the 3D data. When a is greater than 1, it indicates enlargement, and when a is less than 1, it indicates reduction. The iris scaling parameter may indicate the iris scaling factor, and the target object displacement parameter may indicate the displacement of the target object.

[0048] In some embodiments, determining the target position of the target object after deformation based on the target object deformation parameter and the initial position of the target object in the three-dimensional data further includes:

[0049] determining a shape of the target object based on a target object scaling parameter and the iris scaling parameter, and obtaining a first position of the target object after deformation;

[0050] determining a second position of the target object after displacement based on the target object displacement parameter and the first position;

[0051] Based on a mapping relationship between the image to be processed and the three-dimensional data, the second position is transformed to obtain a target position of the target object.

[0052] The mapping relationship between the image to be processed and the three-dimensional data may refer to a transformation matrix MVP in the process of establishing the three-dimensional data. A third coordinate and a fourth coordinate may be obtained based on a target object scaling parameter, a target object displacement parameter, an iris scaling parameter, and a first coordinate of a vertex of the target object region of the three-dimensional data in a first coordinate direction and a second coordinate in a second coordinate direction; a sixth coordinate may be obtained based on the target object scaling parameter, the target object displacement parameter, the first coordinate, the second coordinate, and a fifth coordinate of a vertex of the target object region of the three-dimensional data in a third coordinate direction; and the target position of the target object may be obtained based on the mapping relationship between the image to be processed and the three-dimensional face image, as well as the third coordinate, the fourth coordinate, and the sixth coordinate.

[0053] In some embodiments, determining the shape of the target object based on the target object scaling parameter and the iris scaling parameter, and obtaining the first position of the target object after deformation, further includes:

[0054] determining a shape of an iris region in the target object based on the iris scaling parameter to obtain an iris-processed target object;

[0055] The iris-processed target object is scaled based on the target object scaling parameter to obtain the first position of the target object after deformation.

[0056] In some embodiments, determining the target position of the target object after deformation based on the target object deformation parameter and the initial position of the target object in the three-dimensional data to obtain a deformed image of the target object further includes:

[0057] Calculating a first difference between 1 and a preset weight of the iris scaling parameter, and a first product of the first difference and the first coordinate; calculating a second product of the first coordinate, the iris scaling parameter, and the preset weight; calculating a first sum of the first product and the second product, and a third product of the first sum and the target object scaling parameter; calculating the sum of the third product and a first displacement parameter of the target object displacement parameter in the first coordinate direction to obtain the third coordinate;

[0058] Calculating a fourth product of the first difference and the second coordinate; calculating a fifth product of the second coordinate, the iris scaling parameter, and the preset weight; calculating a second sum of the fourth product and the fifth product, and a sixth product of the second sum and the target object scaling parameter; calculating a sum of the sixth product and a second displacement parameter of the target object displacement parameter in the direction of the second coordinate to obtain the fourth coordinate;

[0059] Calculating a seventh product of the first difference and the fifth coordinate; calculating the sum of the squares of the first coordinate and the second coordinate, and the square root of the difference between a preset radius of the target object and the square sum; calculating an eighth product of the square root and the preset weight, and a ninth product of the eighth product and the target object scaling parameter; and obtaining the sixth coordinate based on the sum of the ninth product and a third displacement parameter of the target object displacement parameter in the third coordinate direction;

[0060] The deformed image of the target object is obtained based on the tenth product of the third coordinate, the fourth coordinate, the sixth coordinate and a transformation matrix, where the transformation matrix is ​​obtained when constructing the three-dimensional data based on the image to be processed.

[0061] Specifically, the target object scaling parameter is s1, the iris scaling parameter is s2, the target object displacement parameter is Δs(Δx, Δy, Δz), the vertex positions of the target object region in the 3D data are Po(x, y, z), the center of the target object region is located at the origin of the coordinate system, and the 3D data is oriented along the positive z-axis of the coordinate system, for example, toward the user. The preset weight w for the iris scaling parameter s2 can represent the degree to which the vertices of the eye region are affected by the iris scaling parameter s2. The target position Pd(x, y, z) of the target object in the target object deformation image may include: P1.x = (Po.x*(1-w)+Po.x*s2*w)*s1+△x; P1.y = (Po.y*(1-w)+Po.y*s2*w)*s1+△y; P1.z = (Po.z*(1-w)+sqrt(r-Po.x*Po.x-Po.y*Po.y)*w)*s1+△z; Pd(x, y, z) = MVP*(P1.x, P1.y, P1.z).

[0062] As can be seen, the vertex positions of the target object region in the three-dimensional data can be deformed based on the target object deformation parameters to obtain the target position Pd(x, y, z), thereby obtaining a deformed image of the target object. Due to the mapping relationship between the image to be processed and the three-dimensional data, and the corresponding relationship between the vertex positions Po(x, y, z) of the target object before deformation and the vertex positions Pd(x, y, z) of the target object after deformation, the texture of the image to be processed can be mapped to the deformed three-dimensional data.

[0063] In some embodiments, the target object material parameters include target object texture direction parameters and target object rendering parameters, and the target object rendering parameters include at least one of the following: target object texture material parameters, light source color parameters, light source direction parameters, target object roughness, target object metalness or blending mode parameters.

[0064] The target object texture direction parameter can indicate the texture direction of the target object area, such as the normal map parameter n1. The target object texture material parameter can indicate the fill color or pattern of the target object area, such as a starry sky texture material. The light source color parameter can indicate the color of the light source, such as white light or yellow light. The light source direction parameter can indicate the direction of the light, such as the angle α between the direction of the incident light and the horizontal plane. The target object roughness can indicate the smoothness of the eye area. For example, the greater the eye roughness, the rougher the visual effect; the smaller the eye roughness, the smoother the visual effect. The target object metalness can indicate the degree of light concentration. For example, the greater the target object metalness, the brighter the visual effect; the smaller the target object metalness, the darker the visual effect. The blending mode parameter can indicate the degree of integration of the special effect with the original image. A larger blending mode parameter indicates a more pronounced special effect, while a smaller blending mode parameter indicates a less pronounced special effect.

[0065] In some embodiments, determining the target color of the target object based on the target object material parameter, the corneal thickness parameter, and the target object deformation image further includes:

[0066] Obtaining a first target color based on the target object material parameters, the target object deformation image, and a preset illumination function;

[0067] Obtaining a second target color based on a normalized corneal thickness parameter, a material parameter of the target object, a normal of the mesh of the three-dimensional data, the target object deformation image, and the preset illumination function;

[0068] The target color is obtained based on the sum of the first target color and the second target color.

[0069] In some embodiments, the target object may include an eyeball. Further, in some embodiments, determining the target color of the target object based on the target object material parameters, the corneal thickness parameters, and the target object deformation image may include: determining the target color of the eyeball based on the eyeball material parameters, the corneal thickness parameters, and the eyeball deformation image.

[0070] In some embodiments, determining the target color of the eyeball based on the eyeball material parameter, the corneal thickness parameter, and the eyeball deformation image further includes:

[0071] Calculating a normal function with respect to an eyeball texture direction parameter to obtain a first normal; and calculating an illumination function with respect to the eyeball deformed face image, the eyeball rendering parameter, and the first normal to obtain a first target color;

[0072] Calculating a second difference between 1 and a normalized corneal thickness parameter, and an eleventh product of the second difference and the first normal; calculating a twelfth product between a second normal of the mesh of the three-dimensional facial image and the normalized corneal thickness parameter; and obtaining a second target color based on an illumination function related to the eyeball deformed facial image, the eyeball rendering parameter, and a sum of the eleventh product and the twelfth product;

[0073] The eyeball target color is obtained based on the sum of the first target color and the second target color.

[0074] Specifically, the eyeball texture direction parameter (e.g., normal map) can be n1, the eyeball rendering parameter can be m, the normal of the mesh in the three-dimensional data can be n2, the normalized corneal thickness can be d, d∈[0.0,1.0], N is the texture direction function (i.e., normal function) converted from the eyeball texture direction parameter, and F is the preset illumination model. The color of the eyeball area can then be calculated based on this to achieve a multi-layer illumination effect based on corneal thickness. The eyeball target color C of each network in the eyeball area in the target object deformation image Se can include: C1=F(Se,m,N(n1)); C2=F(Se,m,N(n1)*(1-d)+n2*d); C=C1+C2.

[0075] In some embodiments, the adjustment instruction further includes: a first state parameter, a second state parameter, and a region deformation parameter. The first state parameter may be a deformation parameter for a first portion of the non-target object area, the second state parameter may be a deformation parameter for a second portion of the non-target object area, and the region deformation parameter may be a deformation parameter for a third portion of the non-target object area. For example, the first portion may be the left eye socket, the second portion may be the right eye socket, and the third portion may be other facial areas.

[0076] In some embodiments, determining a target position of a non-target object area based on the adjustment instruction and the three-dimensional data further includes:

[0077] determining a first target position of a first part and a second target position of a second part in the non-target object area based on the first state parameter and the second state parameter respectively;

[0078] determining a shape of a third part in the non-target object area based on the regional deformation parameter, and obtaining a target position of the third part;

[0079] The non-target object area target position is obtained based on the first part target position, the second part target position and the third part target position.

[0080] In some embodiments, determining a first target position of a first part and a second target position of a second part in a non-target object area based on the first state parameter and the second state parameter, respectively, further includes:

[0081] determining a first state angle of a first portion and a second state angle of a second portion in a non-target object area based on the three-dimensional data;

[0082] determining the shape of the first part based on the first state angle and the first state parameter, and obtaining a target position of the first part;

[0083] The shape of the second part is determined based on the second state angle and the second state parameter to obtain the target position of the second part.

[0084] Among them, since the target object area is deformed and color processed based on the target object special effect parameters, in order to avoid the mismatch between the target object and other areas of the image, the non-target object area can be adjusted accordingly based on the first state parameters, second state parameters and area deformation parameters of the non-target object, thereby improving the overall coordination of the image.

[0085] Specifically, the target object may include an eyeball. Accordingly, the adjustment instruction includes non-target object special effect parameters, which may include left eye open deformation parameters, which may refer to deformation parameters when the left eye is open; right eye open deformation parameters may refer to deformation parameters when the right eye is open; and eye deformation parameters may refer to deformation parameters of the entire eye. Among them, the angle may refer to the angle between the first line connecting the center position of the eyeball and the highest feature point of the eye (such as the highest feature point of the left eye or the highest feature point of the right eye), and the second line connecting the center position of the eyeball and the lowest feature point of the eye (such as the lowest feature point of the left eye or the lowest feature point of the right eye).

[0086] Furthermore, in some embodiments, determining a target position of a non-target object area based on the adjustment instruction and the three-dimensional data further includes:

[0087] determining a left eye opening angle and a right eye opening angle based on the three-dimensional data;

[0088] Calculating a left eye opening position based on the left eye opening deformation parameter and a vertex position of the three-dimensional data, and obtaining a first target position based on a product of the left eye opening position and the left eye opening angle;

[0089] Calculating a right eye opening position based on the right eye opening deformation parameter and the vertex position of the three-dimensional data, and obtaining a second target position based on a product of the right eye opening position and the right eye opening angle;

[0090] Calculating a third target position based on the eye deformation parameter and the vertex position of the three-dimensional data;

[0091] The non-target object area target position is determined based on the first target position, the second target position, and the third target position.

[0092] Specifically, the adjustment instruction may include left eye opening deformation parameter BlendShapes1, right eye opening deformation parameter BlendShapes2, and eye deformation parameter BlendShapes3. The left eye opening angle d1 and the right eye opening angle d2 can be determined based on the three-dimensional data. The vertex position of the three-dimensional data (i.e., before deformation) is Pfo(x, y, z). Then, the vertex position Pfd(x, y, z) of the non-target object after deformation processing may include: Pfd=BlendShapes3(Pfo)+BlendShapes1(Pfo)*d1+BlendShapes2(Pfo)*d2.

[0093] It can be seen that the vertex position Pfd(x, y, z) of the non-target object after deformation processing is the target position of the non-target object area.

[0094] In step S340 , the three-dimensional data is processed based on the target position and target color of the target object and the target position of the non-target object area to obtain a target image.

[0095] In some embodiments, processing the three-dimensional data based on the target position and target color of the target object and the target position of the non-target object area to obtain a target image includes:

[0096] Rendering to obtain a target object area of ​​the target image based on the target position and the target color;

[0097] The non-target object region of the target image is obtained based on the target position of the non-target object region and the texture rendering of the non-target region of the image to be processed.

[0098] Specifically, for example, if the target object includes an eyeball, the positions of each vertex in the eyeball region can be converted to the target position, and the eyeball region can be rendered based on the target color of the corresponding grid. For non-eyeball regions, the vertex positions in the non-eyeball region can be converted to the non-eyeball region target position, and based on the correspondence between the vertex positions and the feature points of the image to be processed, the texture pattern or color of the image to be processed can be rendered in the non-eyeball region, thereby obtaining a three-dimensional target image that has been processed with eyeball special effects and deformed in the non-eyeball region. Furthermore, the three-dimensional target image can be converted into a two-dimensional image based on the transformation matrix MVP to obtain the target image.

[0099] As can be seen, the image processing method according to the disclosed embodiments not only enriches and enhances the effects of eye effects, but also simultaneously processes facial areas outside the eyeballs, making the eyeballs more coordinated with other areas of the face, further improving the image processing effect. Users can flexibly set eye effects through eye and facial effect parameters, improving the efficiency of eye effects processing and reducing production costs.

[0100] It should be noted that the method of the embodiments of the present disclosure can be performed by a single device, such as a computer or server. The method of the embodiments of the present disclosure can also be applied in a distributed scenario, where multiple devices cooperate to perform the method. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiments of the present disclosure, and the multiple devices will interact with each other to complete the method.

[0101] It should be noted that the above description is limited to some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0102] Based on the same technical concept, corresponding to any of the above embodiments and methods, the present disclosure further provides an image processing device, the image processing device comprising:

[0103] A user interaction module, configured to obtain an image to be processed and an adjustment instruction for the image to be processed;

[0104] A three-dimensional reconstruction module, configured to create corresponding three-dimensional data based on the image to be processed;

[0105] A special effects generation module is used to determine the target position, target color and target position of the non-target object area of ​​the target object based on the adjustment instruction and the three-dimensional data; and to process the three-dimensional data based on the target position, target color and target position of the non-target object area to obtain a target image.

[0106] In some embodiments, the user interaction module is further configured to output the target image, for example, by displaying the target image in an interactive interface.

[0107] Specifically, referring to FIG4 , FIG4 shows a schematic diagram of an image processing device according to an embodiment of the present disclosure. In FIG4 , a user can determine a two-dimensional image to be processed based on a user interaction module, and a three-dimensional reconstruction module can reconstruct a three-dimensional model of the input image to be processed to obtain corresponding three-dimensional time data; a special effect generation module can perform eyeball deformation and color processing on the three-dimensional data based on the three-dimensional data and the adjustment information of the target object and non-target object deformation input by the user based on the user interaction module, as well as the eyeball material parameters and corneal thickness information set by the user, to generate a target object special effect. The overall area is adjusted to generate a three-dimensional target image. Finally, the three-dimensional target image is converted into a two-dimensional target image output, and the two-dimensional target image can be a two-dimensional image after the target object special effect processing.

[0108] For the convenience of description, the above devices are described as being functionally divided into various modules. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0109] The apparatus of the above embodiment is used to implement the corresponding image processing method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0110] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the image processing method described in any of the above embodiments.

[0111] As can be seen from the above, the image processing method and related equipment provided by the present disclosure can enrich and improve the special effects of the target object by converting the image to be processed into three-dimensional data and adjusting the shape and color of the target object based on adjustment instructions. At the same time, non-target object areas are processed based on adjustment instructions, which can make the target object more coordinated with other areas, further improving the image processing effect. Users can flexibly set the special effects of the target object through adjustment instructions, improving the efficiency of target object special effects processing and reducing production costs.

[0112] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0113] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the image processing method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0114] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Within the scope of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0115] In addition, to simplify the description and discussion, and so as not to obscure the embodiments of the present disclosure, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, devices may be shown in the form of block diagrams to avoid obscuring the embodiments of the present disclosure, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure are to be implemented (i.e., these details should be fully within the purview of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0116] Although the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0117] The embodiments of the present disclosure are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. An image processing method, the method comprising: Acquire an image to be processed and an adjustment instruction for the image to be processed; Establishing corresponding three-dimensional data based on the image to be processed; Determine a target position, a target color, and a non-target object region target position of a target object based on the adjustment instruction and the three-dimensional data; The three-dimensional data is processed based on the target position and the target color of the target object and the target position of the non-target object area to obtain a target image.

2. The method according to claim 1, wherein: The adjustment instruction includes: a target object deformation parameter, a target object material parameter and a corneal thickness parameter of the target object; Then determining the target position and the target color of the target object based on the adjustment instruction and the three-dimensional data further includes: Determining a target position of the target object after the target object is deformed based on the target object deformation parameter and the initial position of the target object in the three-dimensional data, and obtaining a deformed image of the target object; A target color of the target object is determined based on the target object material parameter, the corneal thickness parameter and the target object deformation image.

3. The method according to claim 2, wherein: The surface of the three-dimensional data includes a plurality of grids and vertices between the grids; the target object deformation parameters include a target object scaling parameter, an iris scaling parameter and a target object displacement parameter; Then determining the target position of the target object after the target object is deformed based on the target object deformation parameter and the initial position of the target object in the three-dimensional data further includes: Determine the shape of the target object based on the target object scaling parameter and the iris scaling parameter, and obtain a first position of the target object after deformation; Determine a second position of the target object after displacement based on the target object displacement parameter and the first position; Based on the mapping relationship between the image to be processed and the three-dimensional data, the second position is transformed to obtain a target position of the target object.

4. The method according to claim 3, determining the shape of the target object based on the target object scaling parameter and the iris scaling parameter to obtain the first position of the target object after deformation, further comprising: Determine the shape of the iris region in the target object based on the iris scaling parameter to obtain the target object after iris processing; The target object after iris processing is scaled based on the target object scaling parameter to obtain the first position of the target object after deformation.

5. The method according to claim 2, wherein: Determining the target color of the target object based on the target object material parameter, the corneal thickness parameter and the target object deformation image further includes: Obtaining a first target color based on the target object material parameter, the target object deformation image and a preset illumination function; Obtaining a second target color based on a normalized corneal thickness parameter, a material parameter of the target object, a normal of a mesh of the three-dimensional data, a deformation image of the target object, and the preset illumination function; The target color is obtained based on the sum of the first target color and the second target color.

6. The method according to claim 1, wherein: The adjustment instruction also includes: a first state parameter, a second state parameter and a regional deformation parameter; Then determining the target position of the non-target object area based on the adjustment instruction and the three-dimensional data further includes: Determine a first target position of a first part and a second target position of a second part in the non-target object area based on the first state parameter and the second state parameter respectively; Determine the shape of a third part in the non-target object area based on the regional deformation parameter, and obtain a target position of the third part; The non-target object area target position is obtained based on the first part target position, the second part target position and the third part target position.

7. The method according to claim 1, wherein: The three-dimensional data is processed based on the target position, the target color of the target object and the target position of the non-target object area to obtain a target image, including: Rendering to obtain a target object area of ​​the target image based on the target position and the target color; The non-target object region of the target image is obtained based on the target position of the non-target object region and the texture rendering of the non-target region of the image to be processed.

8. An image processing device, comprising: A user interaction module, used to obtain an image to be processed and an adjustment instruction for the image to be processed; A three-dimensional reconstruction module, used to establish corresponding three-dimensional data based on the image to be processed; A special effect generation module, used to determine a target position, a target color, and a target position of a non-target object area of ​​a target object based on the adjustment instruction and the three-dimensional data; And the three-dimensional data is processed based on the target position and target color of the target object and the target position of the non-target object area to obtain a target image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 7.

Images