Image processing methods and related equipment

Abstract

This disclosure provides an image processing method and related equipment. The method includes obtaining an image to be processed and adjustment commands for the image to be processed; creating corresponding three-dimensional data based on the image to be processed; determining the target position of a target object, a target color, and the target position of a non-target object region based on the adjustment commands and the three-dimensional data; and processing the three-dimensional data based on the target position of the target object, a target color, and the target position of a non-target object region to obtain a target image.

Description

Technical Field

[0001] (Cross - reference to Related Applications) This application claims priority to a Chinese patent application for invention with application number 202311120184.0 and title "Image Processing Method and Related Equipment" filed on August 31, 2023. The entire content of the said application is incorporated herein by reference.

[0002] (Field of the Invention) This disclosure relates to the field of computer technology, and particularly to an image processing method and related equipment.

Background Art

[0003] The effect of existing image - processing target object effects is single. The target object effect tends to deform a local area and does not conform to the characteristics of other areas. As a result, the overall image of the target object effect after adjustment between the target object area and the entire image is not adjusted, and the image - processing result deteriorates.

Summary of the Invention

Problems to be Solved by the Invention

[0004] Embodiments of this disclosure provide an image - processing method, apparatus, device, storage medium, and program product.

Means for Solving the Problems

[0005] In a first aspect of this disclosure, an image - processing method is provided, which includes obtaining an image to be processed and an adjustment instruction for the image to be processed, creating corresponding three - dimensional data based on the image to be processed, determining the target position and target color of a target object and the target position of a non - target object area based on the adjustment instruction and the three - dimensional data, and processing the three - dimensional data based on the target position and target color of the target object and the target position of the non - target object area to obtain a target image.

[0006] A second aspect of the present disclosure provides an image processing apparatus, which includes a user interaction module for obtaining an image to be processed and adjustment commands for the image to be processed; a three-dimensional recreation module for creating corresponding three-dimensional data based on the image to be processed; and an effect generation module for determining the target position of a target object, the target color and the target position of a non-target object region based on the adjustment commands and the three-dimensional data, and for processing the three-dimensional data based on the target position of the target object, the target color and the target position of a non-target object region to obtain a target image.

[0007] A third aspect of the present disclosure provides electronic equipment including memory, one or more processors, and one or more computer programs stored in the memory and capable of running on one or more processors, the programs including instructions for performing the methods described in the first or second aspect.

[0008] In a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium is provided which, when the computer program is executed by one or more processors, causes the processors to perform the method described in the first or second aspect.

[0009] In a fifth aspect of this disclosure, a computer program product is provided which includes a computer program instruction that, when executed on a computer, causes the computer to perform the method described in the first aspect. [Brief explanation of the drawing]

[0010] To more clearly illustrate the technical solutions in this disclosure or related technologies, the accompanying drawings used in the descriptions of embodiments or related technologies are briefly introduced below. However, the accompanying drawings in the following descriptions are merely embodiments of this disclosure, and it will be apparent to those skilled in the art that other accompanying drawings can be obtained from these drawings without requiring any creative effort. [Figure 1] This is a schematic diagram of the image processing architecture of the embodiment disclosed herein. [Figure 2] This is a schematic diagram of the hardware structure of an exemplary electronic device according to an embodiment of the present disclosure. [Figure 3] This is a schematic flowchart illustrating the image processing method according to the embodiments of this disclosure. [Figure 4] This is a schematic diagram of the image processing apparatus of the present disclosure embodiment. [Modes for carrying out the invention]

[0011] The purpose, technical solutions, and benefits of this disclosure will be described in more detail below, along with specific embodiments, with reference to the accompanying drawings, in order to make them clearer and easier to understand.

[0012] Unless otherwise defined, technical or scientific terms used in the embodiments of this disclosure should have the ordinary meanings understood by those skilled in the art of the field to which this disclosure belongs. Terms such as “first,” “second,” etc., used in the embodiments of this disclosure do not imply any order, quantity, or importance, but are used simply to distinguish different components. “Includes” or “constitutes” and similar phrases mean that the components or objects appearing before the phrase encompass the components or objects and their equivalents listed after the phrase, without excluding other components or objects. Terms such as “connected” or “connected” may include electrical connections, whether direct or indirect, but are not limited to physical or mechanical connections. “Up,” “down,” “left,” “right” Terms like "..." are used only to indicate relative positional relationships, and if the absolute position of the described object changes, the relative positional relationship may also change accordingly. If the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0013] Before using the technical solutions disclosed in each embodiment of this disclosure, please understand that, in accordance with applicable laws and regulations, it is necessary to notify users of the types, scope, and scenarios of use of personal information related to this disclosure and to obtain their consent in an appropriate manner.

[0014] For example, in response to receiving a voluntary request from a user, prompt information is sent to the user, explicitly informing the user that the requested operation requires the acquisition and use of the user's personal information. This allows the user to independently choose, based on the prompt information, whether or not to provide personal information to software or hardware such as electronic devices, applications, servers, or storage media that perform the operation of the technical solution of this disclosure.

[0015] In an optional but non-limiting embodiment, in response to receiving a voluntary request from the user, prompt information is sent to the user, for example, in the form of a pop-up window in which the prompt information is presented in text form. Furthermore, the pop-up window may include option controls for the user to select whether to "agree" or "disagree" to providing personal information to the electronic device.

[0016] The above notice and user authorization process are general in nature and do not limit the ways in which this disclosure may be implemented. Please understand that other methods that comply with applicable laws and regulations may be applied in how this disclosure may be implemented.

[0017] Figure 1 shows a schematic diagram of an image processing architecture according to an embodiment of the present disclosure. Referring to Figure 1, this image processing architecture 100 may include a server 110, a client 120, and a network 130 that provides communication links. The server 110 and the client 120 can be connected by a wired or wireless network 130. Here, the server 110 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, security services, and CDNs.

[0018] Client 120 can be implemented in hardware or software. For example, if Client 120 is implemented in hardware, it may be a variety of electronic devices having a display and supporting page display, including, but not limited to, intelligent mobile phones, tablet PCs, e-book readers, laptop-type portable computers, and desktop computers. If Client 120 is implemented in software, it can be attached to the electronic devices listed above and may be implemented as multiple software or software modules (e.g., software or software modules for providing distributed services) or as a single software or software module, without specific limitations.

[0019] The image processing method according to the embodiment of this application may be executed by client 120 or by server 110. It should be understood that the number of clients, networks, and servers in Figure 1 is merely an example and not intended to limit them. Any number of clients, networks, and servers can be used depending on the implementation needs.

[0020] FIG. 2 shows a schematic hardware structure diagram of an exemplary electronic device 200 according to an embodiment of the present disclosure. As shown in FIG. 2, the electronic device 200 may include a processor 202, a memory 204, a network module 206, a peripheral interface 208, and a bus 210. Here, the processor 202, the memory 204, the network module 206, and the peripheral interface 208 realize communication connections inside the electronic device 200 with each other via the bus 210.

[0021] The processor 202 may be a central processing unit (CPU), an image processor, a neural network processor (NPU), a microcontroller (MCU), a programmable logic device, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or one or more integrated circuits. The processor 202 may be used to execute functions related to the technologies described in the present disclosure. In some embodiments, the processor 202 may further include a plurality of processors integrated into a single logical component. For example, as shown in FIG. 2, the processor 202 may include a plurality of processors 202a, 202b, and 202c.

[0022] Memory 204 may be configured to store data (e.g., instructions, computer code, etc.). As shown in FIG. 2, the data stored in memory 204 may include program instructions (e.g., program instructions for implementing the image processing method of the embodiments of the present disclosure) and processed data (e.g., the memory may store configuration files of other modules, etc.). Processor 202 can further access the program instructions and data stored in memory 204 and execute the program instructions to manipulate the processed data. Memory 204 may include a volatile storage device or a non-volatile storage device. In some embodiments, memory 204 may include a random access memory (RAM), a read-only memory (ROM), a disk, a magnetic disk, a hard disk, a solid state drive (SSD), a flash memory, a memory stick, etc.

[0023] Network module 206 may be configured to provide communication between electronic device 200 and other external devices via a network. This network may be any wired or wireless network capable of transmitting and receiving data. For example, this network may be a wired network, a local wireless network (e.g., Bluetooth (registered trademark), WiFi, near field communication (NFC), etc.), a cellular network, the Internet, or a combination of the above. As can be understood, the type of network is not limited to the above specific examples. In some embodiments, network module 306 may include any combination of any number of network interface controllers (NICs), radio frequency modules, transceivers, modems, routers, gateways, adapters, cellular network chips, etc.

[0024] The peripheral interface 208 may be configured to connect the electronic device 200 to one or more peripheral devices in order to enable information input and output. For example, the peripheral devices may include input devices such as keyboards, mice, touchpads, touchscreens, microphones, and various sensors, and output devices such as displays, speakers, vibrators, and indicators.

[0025] Bus 210 may be configured to transmit information such as internal buses (e.g., processor-memory bus) and external buses (USB port, PCI-E bus) between various components of the electronic device 200 (e.g., processor 202, memory 204, network module 206, and peripheral interface 208).

[0026] Although the architecture of the electronic device 200 shown above only includes the processor 202, memory 204, network module 206, peripheral interface 208, and bus 210, in a specific implementation, the architecture of the electronic device 200 may include other components necessary to achieve normal operation. A person skilled in the art will understand that the architecture of the electronic device 200 may include only the components necessary to implement the embodiments of this disclosure, and does not necessarily have to include all the components shown.

[0027] Conventional target object effects are often implemented based on 2D keys. For example, in an eye effect, the user can define a pupil texture material, and based on an algorithm, detect 2D pupil keys and draw the pupil texture to the appropriate area to output a face image. However, such eye effects cannot simulate the physical reflection effect of the eyeball, and the effect that can be achieved is relatively limited. Furthermore, if the output image deforms the entire eyeball and orbit, the deformed result will not match the overall features, causing problems such as pupil distortion and morphology, and the image processing effect will be poor. There are also some techniques that implement target object effects based on 3D recreation, but with conventional light illumination simulation methods, it is difficult for the user to simulate corneal thickness, and similarly, it is difficult to achieve shape adjustments to the target object. Another target object effect method involves training the target object effect based on a network, enabling it to take an original image as input and output a target object effect of a certain shape. However, if the user needs to readjust the texture, shape, or other effects of the target object effect, they must retrain the model, resulting in relatively high production costs. Furthermore, the target object effect cannot be adjusted in real time, making adjustments less flexible. Therefore, finding a way to improve the effectiveness and efficiency of target object effects while simultaneously lowering production costs is an urgent technical challenge that needs to be addressed.

[0028] In this regard, embodiments of this disclosure propose an image processing method and related equipment. By converting the image to be processed into three-dimensional data and performing adjustments to the shape and color of the target object based on adjustment commands, the effect of the target object effect can be enriched and improved. At the same time, by processing non-target object areas based on adjustment commands, the target object and other areas can be more harmonized, further improving the effect of image processing. Users can flexibly set the target object effect by adjusting the commands, improve the efficiency of target object effect processing, and reduce production costs.

[0029] Referring to Figure 3, Figure 3 shows an exemplary flowchart of an image processing method according to an embodiment of the present disclosure. In Figure 3, the image processing method 300 may further include the following steps.

[0030] In step S310, the image to be processed and adjustment commands for the processed image are obtained.

[0031] Here, the image to be processed may include a 2D face image, where a 2D face image refers to a 2D image containing a face. The image to be processed can be uploaded locally or obtained via a network. Adjustment commands can be operated and triggered by the user based on a user interaction interface, and these adjustment commands may include target object effect parameters and non-target object effect parameters. For example, if the target object may include an eyeball, the target object feature parameter may refer to the eyeball effect parameter, where the eyeball effect parameter may refer to parameters related to display effects such as the shape, size, color (e.g., including texture color and / or brightness), orientation, and effect hybridity of the eyeball region. In this case, the non-target object effect parameter may refer to effect parameters of regions other than the eyeball region, for example, the face effect parameter, where the face effect parameter may refer to parameters related to the shape, size, color (e.g., including texture color and / or brightness), orientation, etc., of the non-eyeball region. Specifically, the target object effect parameter and the target object effect parameter may be set by the user based on an interaction interface. For example, if the target object includes an eyeball, the interaction interface may present the user with one or more eyeball effect parameters and one or more face effect parameters, along with multiple numerical values ​​corresponding to each eyeball and face effect parameter, in the form of sliders or options, allowing the user to select from them. The user can then select the final parameter value based on user interaction (e.g., a click). Alternatively, the parameter values ​​for each eyeball and face effect parameter can be directly entered via input boxes in the interaction interface.

[0032] In step S320, corresponding three-dimensional data is created based on the image being processed.

[0033] Here, the image to be processed may include multiple feature points, and based on these feature points, corresponding three-dimensional data can be recreated, for example, by employing a three-dimensional variability model (3DMM). Specifically, by fitting the image to be processed with a general 3DMM model, three-dimensional data corresponding to the image to be processed can be obtained, for example, the three-dimensional data may include a three-dimensional face image. In the process of recreating the three-dimensional data, the feature points of the image to be processed can form a mapping relationship with the vertices of the mesh of the three-dimensional data, and this mapping relationship can be represented by a transformation matrix MVP. Based on the three-dimensional data, three-dimensional target object data and three-dimensional non-target object data can be determined, for example, the three-dimensional target object data and three-dimensional non-target object data may include three-dimensional coordinates in the three-dimensional coordinate system of the target object region and non-target object region in the three-dimensional data, respectively, and the three-dimensional coordinate system may include a first coordinate direction, a second coordinate direction and a third coordinate direction, and the first coordinate direction, the second coordinate direction and the third coordinate direction, for example, the xyz coordinate system, may be perpendicular to each other.

[0034] In step S330, the target position, target color, and target position of the non-target object region are determined based on the adjustment command and the three-dimensional data.

[0035] Here, the shape and color of the target object (e.g., eyeball) region can be adjusted by setting adjustment commands. Since the surface of the three-dimensional data may include multiple polygonal networks (e.g., triangular or quadrilateral meshes), the target object effect parameters can change the shape and size of the target object by adjusting the vertex positions of multiple meshes to change the shape of the mesh. Specifically, the target object characteristic parameters may include eyeball effect parameters, and the eyeball effect parameters may further include parameters related to the iris and cornea, thereby achieving a more realistic eyeball effect by simulating the light illumination effect of an actual eyeball.

[0036] In some embodiments, the eyeball effect parameters include eyeball deformation parameters, eyeball material parameters, and corneal thickness parameters. Here, the adjustment command may also include eyeball effect parameters.

[0037] In some embodiments, the target position and target color of the target object are determined based on the adjustment command and the three-dimensional data. Based on the deformation parameters of the target object and the initial position of the target object in the three-dimensional data, the target position of the target object after deformation is determined, and a deformed image of the target object is obtained. This further includes determining the target color of the target object based on the material parameters of the target object, the corneal thickness parameters, and a deformed image of the target object.

[0038] Here, the deformation parameter of the target object may be used to indicate the degree of deformation of the target object, thereby determining the target position of the target object after deformation. The material parameter of the target object may be used to indicate the color of the target object's coating, and in conjunction with the corneal thickness parameter, the target color of the target object under the light illumination conditions of the eyeball can be determined.

[0039] In some embodiments, the surface of the three-dimensional data includes a plurality of meshes and vertices between the meshes, and the deformation parameters of the target object include a scaling parameter for the target object, an iris scaling parameter, and a displacement parameter for the target object.

[0040] Here, the scaling parameter of the target object can indicate a multiple of the target object's scaling. For example, if the scaling parameter of the target object is 'a', the deformed target object region will be 'a' times the original target object region in the three-dimensional data. If 'a' is greater than 1, it represents expansion; if 'a' is less than 1, it represents contraction. The iris scaling parameter can indicate a multiple of the iris scaling, while the displacement parameter of the target object can indicate the displacement of the target object's movement.

[0041] In some embodiments, determining the target position of the target object after deformation based on the deformation parameters of the target object and the initial position of the target object in the three-dimensional data is possible. The shape of the target object is determined based on the scaling parameters of the target object and the iris scaling parameters, and a first position of the target object after deformation is obtained. The second position of the target object after displacement is determined based on the displacement parameters of the target object and the first position, The process further includes transforming the second position based on the mapping relationship between the processed image and the three-dimensional data to obtain the target position of the target object.

[0042] Here, the mapping relationship between the image to be processed and the three-dimensional data can refer to the transformation matrix MVP in the process of creating the three-dimensional data. Based on the scaling parameter of the target object, the displacement parameter of the target object, the iris scaling parameter, and the first coordinate in the first coordinate direction and the second coordinate in the second coordinate direction of the vertices of the target object region of the three-dimensional data, the third and fourth coordinates are obtained. Based on the scaling parameter of the target object, the displacement parameter of the target object, the first coordinate, the second coordinate and the fifth coordinate in the third coordinate direction of the vertices of the target object region of the three-dimensional data, the sixth coordinate is obtained. Based on the mapping relationship between the image to be processed and the three-dimensional face image, and the third, fourth, and sixth coordinates, the target position of the target object can be obtained.

[0043] In some embodiments, the shape of the target object is determined based on the scaling parameters of the target object and the iris scaling parameters, and a first position after deformation of the target object is obtained. Based on the iris scaling parameters, the shape of the iris region in the target object is determined, and an iris-processed target object is obtained. The process further includes scaling the iris-processed target object based on the scaling parameters of the target object to obtain the first position after deformation of the target object.

[0044] In some embodiments, the target position after deformation of the target object is determined based on the deformation parameters of the target object and the initial position of the target object in the three-dimensional data, and a deformed image of the target object is obtained. The first difference value between 1 and the preset weight of the iris scaling parameter, and the first product of the first difference value and the first coordinate are calculated, the second product of the first coordinate, the iris scaling parameter and the preset weight is calculated, the first sum of the first product and the second product is calculated, and the third product of the first sum and the scaling parameter of the target object is calculated, the sum of the third product and the first displacement parameter of the target object in the first coordinate direction is calculated, and the third coordinate is obtained. The fourth product of the first difference value and the second coordinate is calculated, the fifth product of the second coordinate, the iris scaling parameter and the preset weight is calculated, the second sum of the fourth product and the fifth product, and the sixth product of the second sum and the scaling parameter of the target object is calculated, the sum of the sixth product and the second displacement parameter of the target object in the second coordinate direction is calculated, and the fourth coordinate is obtained. The 7th product of the 1st difference value and the 5th coordinate is calculated, the square root of the difference between the sum of squares of the 1st coordinate and the 2nd coordinate, and the preset radius of the target object and the sum of squares is calculated, the 8th product of the square root and the preset weight is calculated, and the 9th product of the 8th product and the scaling parameter of the target object is calculated, and the 6th coordinate is obtained based on the sum of the 9th product and the 3rd displacement parameter of the target object's displacement parameter in the 3rd coordinate direction, A deformed image of the target object is obtained based on the product of the third coordinate, the fourth coordinate, the sixth coordinate and the transformation matrix, the transformation matrix being obtained when constructing the three-dimensional data based on the image being processed.

[0045] Specifically, the scaling parameter of the target object is s1, the iris scaling parameter is s2, the displacement parameter of the target object is △s(△x, △y, △z), the vertex position of the target object region in the three-dimensional data is Po(x, y, z), the center of this target object region is located at the origin of the coordinate system, and the direction of the three-dimensional data is the positive direction along the z axis of this coordinate system, for example, the opposite user direction. The preset weight w of the iris scaling parameter s2 may represent the degree of influence of the iris scaling parameter s2 at the vertices of the eyeball region. The target position Pd(x, y, z) of the target object in the deformed image of the target object is, P1.x=(Po.x*(1-w)+Po.x*s2*w)*s1+△x, P1.y=(Po.y*(1-w)+Po.y*s2*w)*s1+△y, P1.z=(Po.z*(1-w)+sqrt(r-Po.x*Po.x-Po.y*Po.y)*w)*s1+△z, Pd(x, y, z) = MVP*(P1.x, P1.y, P1.z) may also be included.

[0046] As can be seen, the target position obtained by deforming the vertex positions of the target object region in the three-dimensional data based on the deformation parameters of the target object may be Pd(x, y, z), and a deformed image of the target object is obtained. Due to the mapping relationship between the image being processed and the three-dimensional data, and the correspondence between the target object vertex positions Po(x, y, z) before deformation and the target object vertex positions Pd(x, y, z) after deformation, the texture of the image being processed can be mapped to the deformed three-dimensional data.

[0047] In some embodiments, the material parameters of the target object include a target object texture direction parameter and a target object rendering parameter, the target object rendering parameter includes at least one of the target object texture material parameter, light source color parameter, light source direction parameter, target object roughness, target object materiality, or hybrid mode parameter.

[0048] Here, the target object texture direction parameter can indicate the texture direction of the target object region, for example, the normal map parameter n1. The target object texture material parameter can indicate the color or pattern of the paint in the target object region, for example, a starry sky texture material. The light source color parameter can indicate the color derived from the light illumination, for example, white light, yellow light, etc. The light source direction parameter can indicate the direction of the light illumination, for example, the angle α between the direction of the incident light and the horizontal plane. The target object roughness can indicate the smoothness of the eyeball region; for example, the greater the roughness of the eyeball, the greater the roughness of the presented visual effect, and the smaller the roughness of the eyeball, the smoother the presented visual effect. The target object materiality can indicate the degree of light illumination concentration; for example, the greater the target object materiality, the brighter the presented visual effect, and the smaller the target object materiality, the darker the presented visual effect. The hybrid mode parameter can indicate the degree of fusion between the effect and the original image; a larger hybrid mode parameter indicates a more pronounced effect, and a smaller hybrid mode parameter indicates a less pronounced effect.

[0049] In some embodiments, determining the target color of the target object based on the material parameters of the target object, the corneal film thickness parameters, and the deformed image of the target object is possible. Based on the material parameters of the target object, the deformed image of the target object, and the preset light irradiation function, a first target color is obtained. A second target color is obtained based on the normalized corneal thickness parameter, the material parameter of the target object, the normal of the mesh in the three-dimensional data, the deformed image of the target object, and the preset light irradiation function. The method further includes obtaining the target color based on the sum of the first target color and the second target color.

[0050] In some embodiments, the target object may include an eyeball. Furthermore, in some embodiments, determining the target color of the target object based on the material parameters of the target object, the corneal thickness parameters, and the deformed image of the target object may include determining the target color of the eyeball based on the material parameters of the eyeball, the corneal thickness parameters, and the deformed image of the eyeball.

[0051] In some embodiments, the target color of the eyeball is determined based on the eyeball material parameter, the corneal thickness parameter, and the eyeball deformation image. The normal function related to the eyeball texture direction parameter is calculated to obtain a first normal, and the light irradiation function related to the eyeball deformation face image, the eyeball rendering parameter and the first normal is calculated to obtain a first target color. The process involves calculating a second difference value between 1 and the normalized corneal thickness parameter, and an eleventh product of the second difference value and the first normal, calculating a twelfth product between the second normal of the mesh of the three-dimensional face image and the normalized corneal thickness parameter, and obtaining a second target color based on the eyeball deformation face image, the eyeball rendering parameter, and a light irradiation function relating to the sum of the eleventh product and the twelfth product. The method further includes obtaining the eyeball target color based on the sum of the first target color and the second target color.

[0052] Specifically, the eyeball texture direction parameter (e.g., normal map) may be n1, the eyeball rendering parameter may be m, the mesh normal in the 3D data may be n2, the normalized corneal thickness may be d, d∈[0.0, 1.0], N is the texture direction function (i.e., normal function) obtained by transforming the eyeball texture direction parameter, and F is the preset light illumination model. This makes it possible to calculate the color of the eyeball region and realize a multi-layer light illumination effect due to the corneal thickness. The eyeball target color C of each network in the eyeball region in the deformed image Se of the target object is: C1 = F(Se, m, N(n1)), C2=F(Se, m, N(n1)*(1-d)+n2*d), C = C1 + C2 may also be included.

[0053] In some embodiments, the adjustment command further includes a first state parameter, a second state parameter, and a region deformation parameter, where the first state parameter may be a deformation parameter for a first part of a non-target object region, the second state parameter may be a deformation parameter for a second part of a non-target object region, and the region deformation parameter may be a deformation parameter for a third part of a non-target object region. For example, the first part may refer to the left orbit, the second part may refer to the right orbit, and the third part may refer to another part of the face.

[0054] In some embodiments, determining the target position of a non-target object region based on the adjustment command and the three-dimensional data is performed as follows: Based on the first state parameter and the second state parameter, the first target position of the first part and the second target position of the second part in the non-target object region are determined, Based on the region deformation parameters, the shape of the third part in the non-target object region is determined, and the target position of the third part is obtained. The method further includes obtaining the target position of the non-target object region based on the target position of the first part, the target position of the second part, and the target position of the third part.

[0055] In some embodiments, determining the first target position of the first part and the second target position of the second part in the non-target object region based on the first and second state parameters is as follows: Based on the aforementioned three-dimensional data, the first state angle of the first part and the second state angle of the second part in the non-target object region are determined, The shape of the first part is determined based on the first state angle and the first state parameter, and the target position of the first part is obtained. The method further includes determining the shape of the second part based on the second state angle and the second state parameter, and obtaining the target position of the second part.

[0056] Here, since the target object region is deformed and color-processed based on the target object effect parameters, the overall harmony of the image can be improved by making appropriate adjustments to the non-target object region based on the first state parameter, second state parameter, and region deformation parameter of the non-target object, in order to avoid inconsistencies between the target object and other areas of the image.

[0057] Specifically, the target object may include an eyeball. Accordingly, the adjustment command may include a non-target object effect parameter, which may include a left eye opening deformation parameter, which can refer to the deformation parameter in the left eye's open state; the right eye opening deformation parameter, which can refer to the deformation parameter in the right eye's open state; and the eye deformation parameter, which can refer to the deformation parameter of the entire eye. Here, the angle may refer to the angle between a first line connecting the center of the eyeball and the highest feature point of the eye (e.g., the highest feature point of the left eye or the highest feature point of the right eye), and a second line connecting the center of the eyeball and the lowest feature point of the eye (e.g., the lowest feature point of the left eye or the lowest feature point of the right eye).

[0058] Furthermore, in some embodiments, determining the target position of a non-target object region based on the adjustment command and the three-dimensional data is performed. Based on the aforementioned three-dimensional data, the opening angles of the left eye and the right eye are determined, The left eye opening position is calculated based on the left eye opening deformation parameter and the vertex position of the three-dimensional data, and a first target position is obtained based on the product of the left eye opening position and the left eye opening angle. The opening position of the right eye is calculated based on the right eye opening deformation parameter and the vertex position of the three-dimensional data, and a second target position is obtained based on the product of the right eye opening position and the right eye opening angle. The third target position is calculated based on the eye deformation parameters and the vertex positions of the three-dimensional data, The method further includes determining the target position of the non-target object region based on the first target position, the second target position, and the third target position.

[0059] Specifically, the adjustment command may include BlendShapes1 for the left eye opening deformation parameter, BlendShapes2 for the right eye opening deformation parameter, and BlendShapes3 for the eye deformation parameter. If the opening angle d1 of the left eye, the opening angle d2 of the right eye, and the vertex positions of the three-dimensional data (i.e., before deformation) can be determined as Pfo(x, y, z), then the deformed non-target object vertex positions Pfd(x, y, z) are: Pfd may include BlendShapes3(Pfo) + BlendShapes1(Pfo)*d1 + BlendShapes2(Pfo)*d2.

[0060] As you can see, the deformed non-target object vertex positions Pfd(x, y, z) are the target positions in the non-target object region.

[0061] In step S340, the three-dimensional data is processed based on the target position and target color of the target object and the target position of the non-target object region to obtain a target image.

[0062] In some embodiments, the three-dimensional data is processed based on the target position and target color of the target object and the target position of the non-target object region to obtain a target image. The process involves rendering based on the target position and target color to obtain the target object region of the target image, This includes rendering based on the target position of the non-target object region and the texture of the non-target region of the image to be processed to obtain the non-target object region of the target image.

[0063] Specifically, for example, the target object may include an eyeball. The position of each vertex in this eyeball region can be changed to a target position, and this eyeball region can be rendered based on the target color of the corresponding mesh. For non-eyeball regions, the vertex positions of this non-eyeball region can be changed to non-eyeball region target positions, and based on the correspondence between the vertex positions and the feature points of the image being processed, the texture pattern or color of the image being processed can be rendered onto the non-eyeball region, thereby obtaining a three-dimensional target image with eyeball effect processing and non-eyeball region deformation processing. Furthermore, the three-dimensional target image can be converted to a two-dimensional image based on the transformation matrix MVP, thereby obtaining the target image.

[0064] As can be seen, the image processing method according to the embodiments of this disclosure can not only enrich and improve the effect of the eyeball effect, but also, by processing the face region outside the eyeball at the same time, it can better harmonize the eyeball with other areas of the face and further improve the effect of the image processing. Users can achieve flexible settings for the eyeball effect using eyeball effect parameters and face effect parameters, thereby improving the efficiency of eyeball effect processing and reducing production costs.

[0065] The methods of the embodiments of this disclosure can be performed by a single device, such as a single computer or server. The methods of the embodiments can also be applied to distributed scenarios in which multiple devices can cooperate to complete the process. In such distributed scenarios, one of the multiple devices may perform only one or more steps of the methods of the embodiments of this disclosure, while the multiple devices may interact with each other to complete the described method.

[0066] The above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the operations or steps described in the claims may be performed in an order different from that in the embodiments above, and the desired results may still be achieved. Furthermore, the processes depicted in the accompanying drawings do not necessarily require that only a specific order or sequence shown be followed to achieve the desired results. In some embodiments, multitasking and parallel processing may be possible or advantageous.

[0067] Based on the same technical concept and corresponding to the methods of any of the above embodiments, the present disclosure further provides an image processing apparatus, the image processing apparatus is, A user interaction module for obtaining an image to be processed and adjustment commands for the image to be processed, A three-dimensional reconstruction module for creating corresponding three-dimensional data based on the processed image, The system includes an effect generation module for determining the target position, target color, and target position of a non-target object region based on the adjustment command and the three-dimensional data, and for processing the three-dimensional data based on the target position, target color, and target position of the non-target object region to obtain a target image.

[0068] In some embodiments, the user interaction module is further used to output the target image. For example, the target image is displayed on the interaction interface.

[0069] Specifically, referring to Figure 4, Figure 4 shows a schematic diagram of an image processing apparatus according to an embodiment of the present disclosure. In Figure 4, the user can determine a two-dimensional image to be processed based on the user interaction module, the three-dimensional reconstruction module can reconstruct a three-dimensional model from the input image to be processed and obtain corresponding three-dimensional data, and the effect generation module can perform eyeball deformation and color processing on the three-dimensional data based on this three-dimensional data, adjustment information for the deformation of target and non-target objects input by the user based on the user interaction module, and eyeball material parameters and corneal thickness information set by the user, thereby generating a target object effect. The entire area is adjusted to generate a three-dimensional target image. Finally, when the three-dimensional target image is converted to a two-dimensional target image and output, this two-dimensional target image may be a two-dimensional image that has undergone target object effect processing.

[0070] For the sake of clarity, the above-mentioned device will be described by dividing it into various modules based on its function. Of course, when implementing this disclosure, the functions of each module can be realized using the same or multiple software and / or hardware.

[0071] The apparatus of the above embodiment is used to implement the corresponding image processing method in any one of the above embodiments and has the beneficial effects of the embodiment of the corresponding method, and will not be described further here.

[0072] Based on the same technical concept and corresponding to the methods of any of the above embodiments, the present disclosure further provides a non-temporary computer-readable storage medium in which computer instructions are stored, the computer instructions being used to have the computer perform the image processing method described in any one of the above embodiments.

[0073] As can be seen from the above, the image processing method and related equipment disclosed herein can enrich and improve the effect of target object effects by converting the image to be processed into three-dimensional data and adjusting the shape and color of the target object based on adjustment commands. At the same time, by processing non-target object areas based on adjustment commands, the effect of image processing can be further improved by harmonizing the target object with other areas. Users can flexibly set the target object effect by adjusting the commands, improve the efficiency of target object effect processing, and reduce production costs.

[0074] The computer-readable media of this embodiment may include non-volatile and volatile media, movable and immovable media, and information storage may be realized by any method or technique. The information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only disk memory (CD-ROM), digital versatile disk (DVD) or other optical storage, and cartridge-type magnetic tape. Magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media may be used to store information accessible by a computing device.

[0075] The computer instructions stored in the storage medium of the above embodiment are used to have the computer execute the image processing method described in any one of the above embodiments, and have the beneficial effects of the embodiment of the applicable method, which will not be described further here.

[0076] Those skilled in the art will understand that the discussion of any of the above embodiments is merely illustrative and not intended to limit the scope of this disclosure (including the claims) to these examples. Technical features in the above embodiments or different embodiments can also be combined in the concepts of this disclosure. The steps may be carried out in any order, and there are many other modifications in various aspects of the above embodiments of this disclosure, which are not shown in detail.

[0077] Furthermore, for the sake of simplifying the explanation and discussion, and to avoid obscuring the embodiments of this disclosure, well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be illustrated. It should also be noted that, to avoid obscuring the embodiments of this disclosure, the apparatus may be shown in block diagram form, and the details of the embodiments relating to the apparatus in these block diagrams are highly dependent on the platform on which the embodiments of this disclosure are implemented (i.e., these details should be entirely within the comprehension of those skilled in the art). Where specific details (e.g., circuits) are described to illustrate the exemplary embodiments of this disclosure, it will be apparent to those skilled in the art that the embodiments of this disclosure can be implemented without these specific details, or with modifications to these details. Therefore, these descriptions should be considered explanatory rather than restrictive.

[0078] While this disclosure has been described in relation to specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) can be used with the embodiments discussed.

[0079] The embodiments of this disclosure are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Accordingly, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image processing method, To obtain the image to be processed and the adjustment commands for the image to be processed, The process involves creating corresponding three-dimensional data based on the processed image, Based on the adjustment command and the three-dimensional data, the target position, target color, and target position of the non-target object region of the target object are determined. An image processing method comprising processing the three-dimensional data based on the target position and target color of the target object and the target position of the non-target object region to obtain a target image.

2. The adjustment command includes the deformation parameter of the target object, the material parameter of the target object, and the corneal thickness parameter of the target object. Determining the target position and target color of the target object based on the adjustment command and the three-dimensional data is: Based on the deformation parameters of the target object and the initial position of the target object in the three-dimensional data, the target position of the target object after deformation is determined, and a deformed image of the target object is obtained. The method according to claim 1, further comprising determining the target color of the target object based on the material parameters of the target object, the corneal thickness parameters, and a deformed image of the target object.

3. The surface of the three-dimensional data includes a plurality of meshes and vertices between the meshes, and the deformation parameters of the target object include the scaling parameter of the target object, the iris scaling parameter, and the displacement parameter of the target object. Determining the target position of the target object after deformation based on the deformation parameters of the target object and the initial position of the target object in the three-dimensional data is, The shape of the target object is determined based on the scaling parameters of the target object and the iris scaling parameters, and a first position of the target object after deformation is obtained. The second position of the target object after displacement is determined based on the displacement parameters of the target object and the first position, The method of claim 2, further comprising transforming the second position based on the mapping relationship between the processed image and the three-dimensional data to obtain the target position of the target object.

4. Determining the shape of the target object based on the scaling parameters of the target object and the iris scaling parameters, and obtaining the first position of the target object after deformation, Based on the iris scaling parameters, the shape of the iris region in the target object is determined, and an iris-processed target object is obtained. The method according to claim 3, further comprising scaling the iris-processed target object based on the scaling parameters of the target object to obtain the first position after deformation of the target object.

5. The target color of the target object is determined based on the material parameters of the target object, the corneal film thickness parameters, and the deformed image of the target object. Based on the material parameters of the target object, the deformed image of the target object, and the preset light irradiation function, a first target color is obtained. A second target color is obtained based on the normalized corneal thickness parameter, the material parameter of the target object, the normal of the mesh of the three-dimensional data, the deformed image of the target object, and the preset light irradiation function. The method according to claim 2, further comprising obtaining the target color based on the sum of the first target color and the second target color.

6. The adjustment command further includes a first state parameter, a second state parameter, and a region deformation parameter. Determining the target position of a non-target object region based on the adjustment command and the three-dimensional data is: Based on the first state parameter and the second state parameter, the first target position of the first part and the second target position of the second part in the non-target object region are determined, Based on the region deformation parameters, the shape of the third part in the non-target object region is determined, and the target position of the third part is obtained. The method according to claim 1, further comprising obtaining the target position of the non-target object region based on the target position of the first portion, the target position of the second portion, and the target position of the third portion.

7. Processing the three-dimensional data based on the target position and target color of the target object and the target position of the non-target object region to obtain a target image is: The process involves rendering based on the target position and target color to obtain the target object region of the target image, The method according to claim 1, further comprising rendering based on the target position of the non-target object region and the texture of the non-target region of the image to be processed to obtain the non-target object region of the target image.

8. An image processing device, A user interaction module for obtaining an image to be processed and adjustment commands for the image to be processed, A three-dimensional reconstruction module for creating corresponding three-dimensional data based on the processed image, An image processing apparatus comprising an effect generation module for determining the target position, target color, and target position of a non-target object region of a target object based on the adjustment command and the three-dimensional data, and for processing the three-dimensional data based on the target position, target color, and target position of the non-target object region to obtain a target image.

9. An electronic device comprising memory, a processor, and a computer program stored in the memory and operable on the processor, wherein the processor, when executing the program, implements the method according to any one of claims 1 to 7.

10. A non-temporary computer-readable storage medium in which computer instructions are stored, wherein the computer instructions are used to cause a computer to perform the method according to any one of claims 1 to 7.

Images