Image processing method, device, equipment, and storage medium

Abstract

The embodiments of the present application disclose an image processing method and apparatus, a device and a storage medium, wherein the method comprises: processing an acquired image including a target object to obtain 2D key points of the target object; obtaining a general 3D model according to the type of the target object, wherein the general 3D model is applicable to general target objects; obtaining a 3D model of an interactive resource determined for the target object; and displaying an augmented reality (AR) effect after superimposing the target object and the interactive resource according to the 2D key points, the general 3D model and the 3D model of the interactive resource.

Description

Technical Field

[0001] The embodiments of the present application relate to electronic technology, and relate to but are not limited to an image processing method and apparatus, device, and storage medium. Background Art

[0002] In the retail industry, there is a huge demand for online try-on and trial-wearing. Among them, the trial-wearing technology needs to simulate the effect of wearing real accessories, so that the accessories move naturally with the head, such as glasses, stickers, hairpins, headbands, etc.

[0003] At present, face detection, face recognition, face depth information reconstruction and other face technologies are widely used, but due to existing hardware limitations, most mobile terminals are not equipped with depth cameras. Therefore, it is difficult for most mobile terminals to obtain accurate 3D (3-Dimension) information, and most face technologies in mobile terminals only implement 2D (2-Dimension) face key point methods. In this way, in the existing 2D face key point system, how to provide AR (Augmented Reality) trial wearing technology has become a problem that needs to be solved urgently by technicians in this field. Summary of the invention

[0004] In view of this, embodiments of the present application provide an image processing method, apparatus, device, and storage medium.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0007] Processing the acquired image including the target object to obtain 2D key points of the target object;

[0008] Acquire a general 3D model according to the type of the target object, where the general 3D model is applicable to a general target object;

[0009] Obtaining a 3D model of an interactive resource determined for the target object;

[0010] Based on the 2D key points, the general 3D model and the 3D model of the interactive resource, an augmented reality AR effect after the target object and the interactive resource are superimposed is displayed.

[0011] In a second aspect, an embodiment of the present application provides an image processing device, the device comprising:

[0012] An extraction unit, used to process the acquired image including the target object to obtain 2D key points of the target object;

[0013] A determination unit, configured to obtain a general 3D model according to the type of the target object, wherein the general 3D model is applicable to a general target object;

[0014] An acquisition unit, configured to acquire a 3D model of an interactive resource determined for the target object;

[0015] A processing unit is used to display an augmented reality (AR) effect after the target object is superimposed with the interactive resource based on the 2D key points, the general 3D model and the 3D model of the interactive resource.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps in the above-mentioned image processing method are implemented.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above-mentioned image processing method when executed by a processor.

[0018] The embodiments of the present application provide an image processing method and apparatus, a device, and a storage medium, which obtain 2D key points of the target object by processing an acquired image including a target object; obtain a general 3D model according to the type of the target object, wherein the general 3D model is applicable to a general target object; obtain a 3D model of an interactive resource determined for the target object; and display an augmented reality (AR) effect after superimposing the target object and the interactive resource based on the 2D key points, the general 3D model, and the 3D model of the interactive resource. In this way, the AR effect of superimposing the interactive resource can be achieved based on the 2D key points without relying on 3D information. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The following is a schematic diagram of the implementation process of the image processing method of the present application embodiment. Figure 1 ;

[0020] Figure 2 The following is a schematic diagram of the implementation process of the image processing method of the present application embodiment. Figure 2 ;

[0021] Figure 3 The following is a schematic diagram of the implementation process of the image processing method of the present application embodiment. Figure 3 ;

[0022] Figure 4 This is a schematic block diagram of an AR glasses trial-wearing system according to an embodiment of the present application;

[0023] Figure 5A schematic diagram of the width of a human face according to an embodiment of the present application;

[0024] Figure 6 A schematic diagram of 2D face key points and a general 3D face model according to an embodiment of the present application;

[0025] Figure 7 A schematic diagram of the PnP solution principle of an embodiment of the present application;

[0026] Figure 8 This is a schematic diagram of the structure of the image processing device according to an embodiment of the present application;

[0027] Fig. 9 A hardware entity schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solution of the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0029] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0030] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0031] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are merely to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0032] An embodiment of the present application provides an image processing method, and the functions implemented by the method can be implemented by calling program codes by a processor in an electronic device. Of course, the program codes can be stored in a storage medium of the electronic device. Figure 1 The following is a schematic diagram of the implementation process of the image processing method of the present application embodiment. Figure 1 ,like Figure 1 As shown, the method includes:

[0033] Step S101: Processing the acquired image including the target object to obtain 2D key points of the target object;

[0034] In an embodiment of the present application, if the target object is a face, processing the acquired image including the target object to obtain 2D key points of the target object includes: processing the acquired face image based on face detection and face alignment algorithms to obtain 2D face key points.

[0035] Here, the electronic device can be various types of devices with information processing capabilities, such as mobile phones, PDAs (Personal Digital Assistants), navigators, digital phones, video phones, smart watches, smart bracelets, wearable devices, tablet computers, all-in-one computers, etc.

[0036] For example, if the target object is a human face, the image is a human face image.

[0037] Step S102: acquiring a general 3D model according to the type of the target object, wherein the general 3D model is applicable to a general target object;

[0038] For example, if the target object is a user's face, the general 3D model may be an average face model, which is applicable to general faces, that is, average face models corresponding to different users are the same average face model.

[0039] Step S103, obtaining a 3D model of the interactive resource determined for the target object;

[0040] Here, when the target object is a human face, the interactive resource may be glasses, hats, earrings, headphones, etc. When the target object is a human body, the interactive resource may be clothes, pants, etc. When the target object is a car, the interactive resource may be car interior, etc. When the target object is a pet, the interactive resource may be a pet item, such as pet clothes, collars, etc. In the embodiments of the present application, the specific types of the target object and its corresponding interactive resource are not limited.

[0041] Here, the 3D model of the interactive resource can be a universal 3D model of the interactive resource, which does not vary with the size of the target object. The 3D model of the interactive resource can also be a 3D model suitable for the size of the target object, matching the size of the target object.

[0042] Step S104: displaying an augmented reality (AR) effect after the target object and the interactive resource are superimposed based on the 2D key points, the general 3D model and the 3D model of the interactive resource.

[0043] In some embodiments, the display of the augmented reality AR effect after the target object and the interactive resource are superimposed on each other according to the 2D key points, the universal 3D model and the 3D model of the interactive resource comprises: determining the 3D model of the target object according to the 2D key points and the universal 3D model; and displaying the augmented reality AR effect after the target object and the interactive resource are superimposed on each other according to the 3D model of the target object and the 3D model of the interactive resource.

[0044] In an embodiment of the present application, the 2D key points of the target object are obtained by processing the acquired image including the target object; a general 3D model is obtained according to the type of the target object, and the general 3D model is applicable to a general target object; a 3D model of an interactive resource determined for the target object is obtained; and based on the 2D key points, the general 3D model and the 3D model of the interactive resource, an augmented reality AR effect after superimposing the target object and the interactive resource is displayed. In this way, the AR effect of superimposing interactive resources can be achieved based on 2D key points without relying on 3D information.

[0045] Based on the above embodiments, the present application further provides an image processing method. Figure 2 The following is a schematic diagram of the implementation process of the image processing method of the present application embodiment. Figure 2 ,like Figure 2 As shown, the method includes:

[0046] Step S201: Processing the acquired image including the target object to obtain 2D key points of the target object;

[0047] Step S202: acquiring a general 3D model according to the type of the target object, wherein the general 3D model is applicable to a general target object;

[0048] Step S203: acquiring a 3D model of the interactive resource determined for the target object;

[0049] Here, the 3D model of the interactive resource is a universal 3D model of the interactive resource, and will not vary with the size of the target object. For example, when a user tries on 3D glasses online, the size of the model of the 3D glasses will not change with different users.

[0050] Step S204: determining an anchor point according to the 2D key point and the 3D model of the interactive resource;

[0051] Here, a first key point (i.e., a positioning point) can be determined in the 2D key points, and then a second key point can be determined in the 3D model of the interactive resource through the first key point, and the first key point and the second key point are associated, and the associated point is determined as an anchor point. In addition, the first key point and the second key point can be contact points.

[0052] Step S205: determining the size of the interactive resource according to the 2D key points;

[0053] Here, the size of the interactive resource refers to the size that matches the size of the target object in the image. If the target object is far away from the camera when the image is captured, the target object occupies a smaller proportion in the image, and the size of the interactive resource is correspondingly smaller. If the target object is close to the camera when the image is captured, the target object occupies a larger proportion in the image, and the size of the interactive resource is correspondingly larger.

[0054] Of course, the size of the interactive resource can also be a size suitable for the physical size of the target object. For example, when shooting an image, the user can be prompted to shoot at a distance of 0.5 meters from the camera. In this way, when the distance is determined, the size of the interactive resource suitable for the physical size of the target object can be determined according to the size of the target object in the image.

[0055] For example, when a user tries on glasses online, the size of the glasses can be determined based on the user's 2D facial key points. Thus, when the user selects the glasses he wants to try on, the glasses that fit the size of the face in the face image can be worn on the face in the face image based on the determined size.

[0056] Step S206, determining the deflection angle of the target object according to the 2D key points and the general 3D model;

[0057] Here, the deflection angle of the target object may be equated with the deflection angle of the interactive resource.

[0058] For example, when a user tries on glasses online, the glasses need to rotate with the rotation of the user's face, that is, the glasses are stationary relative to the user's face. Therefore, the deflection angle of the user's face represents the deflection angle of the glasses. When the user's face is facing the camera on the display screen, the deflection angle is 0 degrees, and when the user turns his head, the deflection angle is 0 to 90 degrees.

[0059] Step S207: displaying an augmented reality (AR) effect after the target object and the interactive resource are superimposed according to the anchor point, the size of the interactive resource, and the deflection angle of the target object.

[0060] For example, if the target object is a human face and the interactive resource is a pair of glasses, during rendering, the glasses can be placed on the human face according to the anchor point (the midpoint of the glasses' support and the bridge of the human face) (i.e., the midpoint of the glasses' neck support is made to coincide with the bridge of the human face), and then the size of the glasses is adjusted according to the obtained size of the glasses, and the deflection of the glasses is adjusted according to the obtained deflection of the human face, and finally the rendered image is displayed.

[0061] In an embodiment of the present application, the 2D key points of the target object are obtained by processing the acquired image including the target object; a general 3D model is obtained according to the type of the target object, and the general 3D model is applicable to the general target object; a 3D model of the interactive resource determined for the target object is obtained; an anchor point is determined according to the 2D key points and the 3D model of the interactive resource; a size of the interactive resource is determined according to the 2D key points; a deflection angle of the target object is determined according to the 2D key points and the general 3D model; and an augmented reality AR effect after the target object and the interactive resource are superimposed is displayed according to the anchor point, the size of the interactive resource and the deflection angle of the target object. In this way, the AR effect of superimposing the interactive resource can be achieved based on the 2D key points without relying on 3D information.

[0062] Based on the above embodiments, the present application further provides an image processing method, the method comprising:

[0063] Step S211: Process the acquired image including the target object to obtain 2D key points of the target object;

[0064] Step S212: acquiring a general 3D model according to the type of the target object, wherein the general 3D model is applicable to a general target object;

[0065] Step S213, obtaining a 3D model of the interactive resource determined for the target object;

[0066] Step S214: among the 2D key points, determine a key point located at a specific position of the target object as a first key point;

[0067] Step S215, among the key points corresponding to the 3D model of the interactive resource, determine the second key point by a point that coincides with the projection point of the first key point on any same plane;

[0068] Step S216: determining the first key point and the second key point as anchor points;

[0069] Here, the first key point can be used as a positioning point, and based on the positioning point, the second key point can be determined in the interactive resource, and the first key point can be associated with the second key point as an anchor point. In this way, in the subsequent rendering stage, the interactive resource can be drawn onto the target object based on the anchor point.

[0070] Step S217, determining the size of the interactive resource according to the 2D key points;

[0071] Step S218, determining the deflection angle of the target object according to the 2D key points and the general 3D model;

[0072] Step S219: displaying an augmented reality (AR) effect after the target object and the interactive resource are superimposed based on the anchor point, the size of the interactive resource, and the deflection angle of the target object.

[0073] Based on the above embodiments, the present application further provides an image processing method, the method comprising:

[0074] Step S221: Process the acquired image including the target object to obtain 2D key points of the target object;

[0075] Step S222: acquiring a general 3D model according to the type of the target object, wherein the general 3D model is applicable to a general target object;

[0076] Step S223, obtaining a 3D model of the interactive resource determined for the target object;

[0077] Step S224, determining an anchor point according to the 2D key point and the 3D model of the interactive resource;

[0078] Step S225, determining the size of the target object according to the 2D key points;

[0079] Here, the size of the target object determined according to the 2D key points refers to the size of the target object in the image.

[0080] Step S226, determining the ratio between the size of the target object and the size of the image;

[0081] In some embodiments, the target object is a human face. Accordingly, determining the size of the target object based on the 2D key points includes: determining the width of the face based on the acquired 2D face key points; determining the ratio between the size of the target object and the size of the image includes: determining the ratio between the width of the face and the width of the acquired face image.

[0082] Here, the method is applied to an electronic device with a display screen, so it can be considered that the ratio between the width of the face and the width of the face image is linearly related to the distance from the face to the display screen.

[0083] In some embodiments, determining the width of the face based on the acquired 2D face key points includes: determining N pairs of key points among the 2D face key points, the N pairs of key points being points in the face contour area, and N being a natural number greater than or equal to 2; determining the distance between each pair of key points in the N pairs of key points; sorting the N pairs of key points from large to small according to the distance to obtain a key point sequence; determining the average Euclidean distance of M pairs of key points in the key point sequence, where M is a natural number greater than or equal to 2 and M is less than or equal to N; and determining the average Euclidean distance as the width of the face.

[0084] Step S227: determining the distance from the target object to the display screen of the electronic device according to the ratio;

[0085] Step S228, determining the size of the interactive resource according to the distance;

[0086] Step S229, determining the deflection angle of the target object according to the 2D key points and the general 3D model;

[0087] Step S230: displaying an augmented reality (AR) effect after the target object and the interactive resource are superimposed according to the anchor point, the size of the interactive resource, and the deflection angle of the target object.

[0088] Based on the above embodiments, the present application further provides an image processing method, the method comprising:

[0089] Step S231: Process the acquired image including the target object to obtain 2D key points of the target object;

[0090] Step S232: acquiring a general 3D model according to the type of the target object, wherein the general 3D model is applicable to a general target object;

[0091] Step S233, obtaining a 3D model of the interactive resource determined for the target object;

[0092] Step S234, determining an anchor point according to the 2D key point and the 3D model of the interactive resource;

[0093] Step S235, determining the size of the interactive resource according to the 2D key point;

[0094] Step S236: aligning the 2D key points with the 3D points in the general 3D model to obtain a point pair relationship between the 2D space and the 3D space;

[0095] Step S237, determining the deflection angle of the target object according to the point pair relationship and the internal parameters of the image acquisition unit that acquires the image;

[0096] Here, if the user authorizes the APP (Application) to obtain the internal parameters of the image acquisition unit, the internal parameters of the image acquisition unit can be directly obtained. If the user does not authorize the APP to obtain the internal parameters of the image acquisition unit, the internal parameters of the image acquisition unit can be determined based on the parameters of the image.

[0097] Step S238: displaying an augmented reality (AR) effect after the target object and the interactive resource are superimposed based on the anchor point, the size of the interactive resource, and the deflection angle of the target object.

[0098] In some embodiments, the step S237, determining the deflection angle of the target object according to the point pair relationship and internal parameters of an image acquisition unit that acquires the image, includes:

[0099] Step S2371: determining the relative relationship between the world coordinate system corresponding to the universal 3D model and the camera coordinate system corresponding to the image acquisition unit according to the point pair relationship and the internal parameters of the image acquisition unit;

[0100] Step S2372: Determine the deflection angle of the target object according to the relative relationship.

[0101] In some embodiments, the step S2371, determining the relative relationship between the world coordinate system corresponding to the universal 3D model and the camera coordinate system corresponding to the image acquisition unit according to the point pair relationship and the internal parameters of the image acquisition unit, includes:

[0102] Step S2371a, based on the PnP algorithm, determine the relative relationship between the world coordinate system and the camera coordinate system according to the point pair relationship and the internal parameters of the image acquisition unit;

[0103] Here, the PnP algorithm, namely the Perspective-n-Point algorithm, is a method for solving the motion of 3D to 2D point pairs. The method describes how to estimate the pose of an image acquisition unit when multiple 3D space points and their projection positions are known.

[0104] Step S2371b, determining the Euler angle of the target object according to the relative relationship;

[0105] Step S2371c: determine the Euler angle as the deflection angle of the target object.

[0106] Based on the above-mentioned embodiments, the present application further provides an image processing method, which is applied to an electronic device with a display screen, and the method includes:

[0107] Step S241: Process the acquired image including the target object to obtain 2D key points of the target object;

[0108] Step S242: acquiring a general 3D model according to the type of the target object, wherein the general 3D model is applicable to a general target object;

[0109] Step S243, obtaining a 3D model of the interactive resource determined for the target object;

[0110] Step S244, among the 2D key points, determining a key point located at a specific position of the target object as a first key point;

[0111] In the embodiment of the present application, the first key point as a positioning point in the target object should be stable. For example, when the target object is a human face, the first key point can be the midpoint of the two eyes, that is, the bridge of the nose. When the face makes an expression, the position of the coordinate point of the bridge of the nose in the face will basically not change. However, key points such as the corners of the mouth will move up, down, left, and right in the face with the expression of the face. Therefore, key points such as the corners of the mouth cannot be used as positioning points.

[0112] Step S245, among the key points corresponding to the 3D model of the interactive resource, determine the second key point by a point that coincides with the projection point of the first key point on any same plane;

[0113] Step S246: determining the first key point and the second key point as anchor points;

[0114] Step S247, determining the size of the target object according to the 2D key points;

[0115] Step S248, determining the ratio between the size of the target object and the size of the image;

[0116] In some embodiments, the target object is a human face. Accordingly, determining the size of the target object based on the 2D key points includes: determining the width of the face based on the acquired 2D face key points; determining the ratio between the size of the target object and the size of the image includes: determining the ratio between the width of the face and the width of the acquired face image.

[0117] In some embodiments, determining the width of the face based on the acquired 2D face key points includes: determining N pairs of key points among the 2D face key points, the N pairs of key points being points in the face contour area, and N being a natural number greater than or equal to 2; determining the distance between each pair of key points in the N pairs of key points; sorting the N pairs of key points from large to small according to the distance to obtain a key point sequence; determining the average Euclidean distance of M pairs of key points in the key point sequence, where M is a natural number greater than or equal to 2 and M is less than or equal to N; and determining the average Euclidean distance as the width of the face.

[0118] Step S249: determining the distance from the target object to the display screen of the electronic device according to the ratio;

[0119] Step S250: determining the size of the interactive resource according to the distance;

[0120] Step S251, aligning the 2D key points with the 3D points in the general 3D model to obtain a point pair relationship between the 2D space and the 3D space;

[0121] Step S252, determining the deflection angle of the target object according to the point pair relationship and the internal parameters of the image acquisition unit that acquires the image;

[0122] In some embodiments, the method further includes: determining the focal length and optical center of the image acquisition unit based on the length and width of the image; setting the distortion coefficient of the image acquisition unit to a preset value; and determining internal parameters of the image acquisition unit based on the focal length, the optical center and the distortion coefficient.

[0123] Step S253: displaying an augmented reality (AR) effect after the target object and the interactive resource are superimposed according to the anchor point, the size of the interactive resource, and the deflection angle of the target object.

[0124] Based on the above embodiments, the present application further provides an image processing method. Figure 3 The following is a schematic diagram of the implementation process of the image processing method of the present application embodiment. Figure 3 ,like Figure 3 As shown, the method includes:

[0125] Step S301: Process the acquired image including the target object to obtain 2D key points of the target object;

[0126] Step S302: acquiring a general 3D model according to the type of the target object, wherein the general 3D model is applicable to a general target object;

[0127] Step S303: acquiring a 3D model of the interactive resource determined for the target object;

[0128] Step S304: determining an anchor point according to the 2D key point and the 3D model of the interactive resource;

[0129] Step S305: determining the deflection angle of the target object according to the 2D key points and the general 3D model;

[0130] Step S306: displaying an augmented reality (AR) effect after the target object is superimposed with the interactive resource according to the deflection angle between the anchor point and the target object.

[0131] Here, after displaying the augmented reality AR effect after the target object and the interactive resource are superimposed according to the deflection angle of the anchor point and the target object, the user can adjust the size of the interactive resource as needed. For example, when a user tries on clothes online, the user can manually adjust the size of the clothes on the body in the AR image after trying on, so as to determine the different effects of the size of the clothes after trying on the body.

[0132] In some embodiments, the step S304 of determining an anchor point according to the 2D key point and the 3D model of the interactive resource includes:

[0133] Step S3041: among the 2D key points, determine a key point located at a specific position of the target object as a first key point;

[0134] Step S3042: among the key points corresponding to the 3D model of the interactive resource, determine the second key point by a point that coincides with the projection point of the first key point on any same plane;

[0135] Step S3043, determining the first key point and the second key point as anchor points. In some embodiments, the step S305, determining the deflection angle of the target object according to the 2D key point and the general 3D model, includes:

[0136] Step S3051: align the 2D key points with the 3D points in the general 3D model to obtain a point pair relationship between the 2D space and the 3D space;

[0137] Step S3052: Determine the deflection angle of the target object according to the point pair relationship and internal parameters of an image acquisition unit that acquires the image.

[0138] In some embodiments, the step S3052 of determining the deflection angle of the target object according to the point pair relationship and internal parameters of an image acquisition unit that acquires the image includes:

[0139] Step S3052a, determining the relative relationship between the world coordinate system corresponding to the universal 3D model and the camera coordinate system corresponding to the image acquisition unit according to the point pair relationship and the internal parameters of the image acquisition unit;

[0140] Here, determining the relative relationship between the world coordinate system corresponding to the universal 3D model and the camera coordinate system corresponding to the image acquisition unit according to the point pair relationship and the internal parameters of the image acquisition unit includes: determining the relative relationship between the world coordinate system and the camera coordinate system according to the point pair relationship and the internal parameters of the image acquisition unit based on a PnP algorithm; determining the Euler angle of the target object according to the relative relationship; and determining the Euler angle as the deflection angle of the target object.

[0141] Step S3052b: determine the deflection angle of the target object according to the relative relationship.

[0142] In an embodiment of the present application, the 2D key points of the target object are obtained by processing the acquired image including the target object; a general 3D model is obtained according to the type of the target object, and the general 3D model is applicable to a general target object; a 3D model of an interactive resource determined for the target object is obtained; an anchor point is determined according to the 2D key points and the 3D model of the interactive resource; a deflection angle of the target object is determined according to the 2D key points and the general 3D model; and an augmented reality AR effect after superimposing the target object and the interactive resource is displayed according to the anchor point and the deflection angle of the target object. In this way, the AR effect of superimposing interactive resources can be achieved based on 2D key points without relying on 3D information.

[0143] Based on the above-mentioned embodiments, the present application further provides an image processing method, which is applied to an electronic device with a display screen, and the method includes:

[0144] Step S311: Process the acquired image including the target object to obtain 2D key points of the target object;

[0145] Step S312: acquiring a general 3D model according to the type of the target object, wherein the general 3D model is applicable to a general target object;

[0146] Step S313: acquiring a 3D model of an interactive resource determined for the target object; wherein the 3D model of the interactive resource is a general 3D model of the interactive resource;

[0147] Step S314: determining an anchor point according to the 2D key point and the 3D model of the interactive resource;

[0148] Step S315, determining the deflection angle of the target object according to the 2D key points and the general 3D model;

[0149] Step S316: determining the size of the interactive resource according to the 2D key points;

[0150] Step S317: determining a superimposed 3D model of the interactive resource according to the size of the interactive resource and the general 3D model of the interactive resource;

[0151] For example, when a user tries on glasses online, the width of the face in the face image can be determined based on the 2D face key points, and then the width of the glasses can be determined based on the width of the face. Finally, the superimposed 3D model of the glasses superimposed on the face in the image can be determined based on the width of the glasses and the general 3D model of the glasses.

[0152] Step S318: displaying an augmented reality (AR) effect after the target object is superimposed with the interactive resource according to the anchor point, the superimposed 3D model of the interactive resource, and the deflection angle of the target object.

[0153] In some embodiments, the step S316, determining the size of the interactive resource according to the 2D key point, includes:

[0154] Step S3161, determining the size of the target object according to the 2D key points;

[0155] Step S3162: Determine the ratio between the size of the target object and the size of the image;

[0156] In some embodiments, the target object is a human face. Accordingly, determining the size of the target object based on the 2D key points includes: determining the width of the face based on the acquired 2D face key points; determining the ratio between the size of the target object and the size of the image includes: determining the ratio between the width of the face and the width of the acquired face image.

[0157] In some embodiments, determining the width of the face based on the acquired 2D face key points includes: determining N pairs of key points among the 2D face key points, the N pairs of key points being points in the face contour area, and N being a natural number greater than or equal to 2; determining the distance between each pair of key points in the N pairs of key points; sorting the N pairs of key points from large to small according to the distance to obtain a key point sequence; determining the average Euclidean distance of M pairs of key points in the key point sequence, where M is a natural number greater than or equal to 2 and M is less than or equal to N; and determining the average Euclidean distance as the width of the face.

[0158] Step S3163: determining the distance from the target object to the display screen of the electronic device according to the ratio;

[0159] Step S3164: Determine the size of the interactive resource according to the distance.

[0160] Based on the above-mentioned embodiments, the present application further provides an image processing method, which is applied to an electronic device with a display screen, and the method includes:

[0161] Step S321: Based on face detection and face alignment algorithms, the acquired face image is processed to obtain 2D face key points;

[0162] Step S322: obtaining a general human face 3D model and a 3D model of glasses, wherein the general 3D model is applicable to a general human face;

[0163] Step S323, among the 2D face key points, determine the key point located at a specific position of the face as the first key point;

[0164] Step S324, among the key points corresponding to the 3D model of the glasses, determine the second key point by a point that coincides with the projection point of the first key point on any same plane;

[0165] Step S325: determining the first key point and the second key point as anchor points;

[0166] Here, the first key point in the 2D face key points and the second key point in the 3D model of the glasses may be associated as anchor points during rendering.

[0167] Step S326: Determine N pairs of key points among the 2D face key points, where the N pairs of key points are points in the face contour area, and N is a natural number greater than or equal to 2;

[0168] Step S327, determining the distance between each pair of key points in the N pairs of key points;

[0169] Step S328, sorting the N pairs of key points from large to small according to distance to obtain a key point sequence;

[0170] Step S329, determining the average Euclidean distance of M to the key points in the key point sequence, where M is a natural number greater than or equal to 2 and less than or equal to N;

[0171] Here, each pair of key points has actual physical meanings, for example, the points corresponding to the two temples on both sides of the face are a pair of key points. And each pair of key points are two symmetrical key points on both sides of the face contour area.

[0172] Step S330, determining the average Euclidean distance as the width of the face;

[0173] Step S331, determining the ratio between the width of the face and the width of the acquired face image;

[0174] Step S332: determining the distance from the face to the display screen of the electronic device according to the ratio;

[0175] Step S333, determining the size of the glasses according to the distance;

[0176] Step S321, aligning the 2D face key points with the 3D points in the universal face 3D model to obtain a point pair relationship between the 2D space and the 3D space;

[0177] Step S334: Based on the PnP algorithm, according to the point pair relationship and the internal parameters of the image acquisition unit that acquires the face image, determine the relative relationship between the world coordinate system corresponding to the universal face 3D model and the camera coordinate system corresponding to the image acquisition unit;

[0178] Step S335: determining a rotation vector and an offset vector of the face according to the relative relationship;

[0179] Step S336: determining the Euler angle of the face according to the rotation vector and the offset vector;

[0180] Step S337, determining the Euler angle of the face as the deflection angle of the glasses;

[0181] Step S338: Render the glasses on the face image according to the anchor point, the size of the glasses, and the deflection angle of the glasses to show an augmented reality (AR) effect after the face image and the glasses are superimposed.

[0182] Based on the aforementioned embodiments, the embodiments of the present application provide an AR glasses trial wearing method, which is applied to an electronic device with a display screen. The method expands the application capability of the 2D face key point system under AR trial wearing: the system inputs a single-purpose RGB (Red Green Blue) image, and after face detection and alignment, obtains 2D face key point information. Then, combining the coordinates of the 2D face key points with the model of the 3D glasses, a suitable associated point can be selected as an anchor point, and the width of the face can be approximately calculated based on the coordinates of the 2D face key points to obtain the size of the glasses. Then, based on the correspondence between the coordinates of the 2D face key points and the 3D average face model, PnP can be solved to obtain the deflection of the face, that is, the deflection when the glasses are worn. Finally, based on the three information of the anchor point, the size of the glasses, and the deflection when the glasses are worn, the 3D glasses can be accurately worn on the face.

[0183] Figure 4 Schematic diagram of the block diagram of the AR glasses fitting system according to the embodiment of the present application. Figure 4 As shown, the system input is a monocular RGB image, and face detection and face alignment are performed on the RGB image to obtain 2D facial key point information. At the same time, the model of the 3D glasses that need to be tried on and the 3D universal face model can be obtained. Then, according to the 2D facial key point information and the model of the 3D glasses, the anchor point is determined; the width of the face is calculated according to the 2D facial key point information, and the size of the 3D glasses is calculated according to the width of the face; the PYR (Pitch Yaw Roll, Euler angle) of the 3D glasses is calculated according to the 2D facial key point information and the 3D universal face model; finally, according to the anchor point, the size of the 3D glasses, and the PYR of the 3D glasses, the 3D glasses are worn on the face to obtain the glasses wearing effect.

[0184] In the embodiment of the present application, in order to wear the glasses on a person's face, the main steps are divided into three elements: accurate anchor points, stable dimensions, and PYR deflection angles.

[0185] Among them, the accurate anchor point is mainly obtained in the following way: the anchor point determines the relative movement of the glasses and the face when the head moves. If you want the glasses and the face to be relatively still, two factors are required: first, the key points of the face used as the positioning point must be stable, and second, select the appropriate 3D glasses model coordinates to associate with the positioning point.

[0186] In order to ensure the relative stillness of the glasses and the face during rendering, the anchor point on the glasses needs to be projected onto the 2D positioning point associated with it. In order to ensure that this operation really produces the effect of the glasses and the face being relatively still, the glasses anchor point and the 2D positioning point should be contact points in three-dimensional space, because only the contact point coincides with the 2D projection surface at any angle. The midpoint of the mirror holder contacts the bridge of the nose, and the 2D projections of the midpoint of the mirror holder and the bridge of the nose are the same point at any viewing angle, that is, two points with a Euclidean distance of 0. If it is not a contact point, then these two points will only coincide at certain viewing angles, and the distance between the two projection points will be a variable, which will change with the posture of the face and the different facial contours. At this time, performing the rendering operation will produce a relative motion effect between the glasses and the face. Therefore, in the face image, if the bridge of the nose is determined as the positioning point, the midpoint of the mirror holder of the glasses is determined as the anchor point associated with the positioning point.

[0187] Here, we need to associate the key points of the 2D face with the points in the 3D glasses model. The anchor point is essentially a 3D coordinate point on the glasses, which can only be associated after being projected onto 2D. These two points will overlap during the rendering process. The projection of the contact point between the center of the glasses neck support and the bridge of the nose is always the same point at any angle. If the distance is a variable, the glasses will change up and down, left and right on the face during the rendering process. Therefore, in the time series, if the Euclidean distance between two points in each projection direction is 0, this point will be determined as the anchor point.

[0188] The stable size is mainly obtained in the following way: when trying on glasses, the size of the glasses is only a visual effect, which can be converted to determine the distance between the face and the screen. The ratio of the face width to the face image width can reflect the distance from the face to the screen to a certain extent, that is, assuming that this ratio is linearly related to the distance from the face to the screen. The determined face width needs to not change with turning the face, looking up, or looking down when the distance from the face to the screen remains unchanged.

[0189] Figure 5 Schematic diagram of the face width of the embodiment of the present application, such as Figure 5 As shown, there are 106 2D facial key points in the face image. Among them, these points are in a corresponding relationship and can form a pair of points. For example, the point labeled 0 and the point labeled 32 are a pair of points, representing the position of the temples on the face. The point labeled 1 and the point labeled 31 are a pair of points, the point labeled 2 and the point labeled 30 are a pair of points, the point labeled 3 and the point labeled 29 are a pair of points, the point labeled 4 and the point labeled 28 are a pair of points, and the point labeled 5 and the point labeled 27 are a pair of points. These 6 pairs of points are the points with the most stable coordinate changes in the facial contour area of ​​the face, and each pair of points in these 6 pairs of points has a certain distance between them, which is shown in the figure with connecting lines. Among the 6 pairs of points, the average Euclidean distance of the 3 pairs of points with the largest distance is taken as the face width.

[0190] The PYR deflection angle is mainly obtained by the following method:

[0191] The PYR deflection angle is actually to solve the relative position relationship between the face and the image acquisition unit, which can be solved by PnP. The PnP solution requires the coordinate information of the 2D face key points and the general 3D face model. Figure 6 Schematic diagram of 2D face key points and a general 3D face model according to an embodiment of the present application, as shown in FIG. Figure 6 As shown, picture 61 is a schematic diagram of 2D face key points, including 106 2D face key points. Picture 62 is a schematic diagram of a general 3D face model, including 68 3D points, that is, the general 3D face model uses an average face 3D model of 68 points.

[0192] First, 106 2D facial key points in image 61 are aligned with 68 3D points in image 62. For example, 2D key points numbered 0, 2, 4, 6, and 8 in image 61 correspond to 3D points numbered 0, 1, 2, 3, and 4 in image 62, respectively, and 2D key points numbered 1, 3, 5, and 7 in image 61 are discarded. In this way, all points in image 61 and image 62 are aligned, thus forming 68 pairs of 2D-3D point pairs.

[0193] The image acquisition unit in the embodiment of the present application includes a camera.

[0194] Here, since the glasses fitting requires the user to keep the glasses stable when making expressions, not all key points are used, especially those that move a lot when making expressions. In the embodiment of the present application, key points of cheeks, eyebrows, nose and other parts are used through experiments, and key points of corners of mouth, chin, eyes and other parts are discarded.

[0195] Secondly, solving PnP also requires the internal parameters of the image acquisition unit. The internal parameters of the image acquisition unit are estimated as follows: the focal length of the image acquisition unit is determined using formula (1), the optical center of the image acquisition unit is determined using formula (2), and the distortion coefficients are set to 0.

[0196] f x =f y =max(img.Width, img.Height) (1);

[0197] c x =img.Width / 2, c y =img.Height / 2 (2);

[0198] Here, f x is the horizontal focal length, f y is the longitudinal focal length, c x is the transverse optical center, c y is the longitudinal optical center, img.Width is the width of the face image, and img.Height is the length of the face image.

[0199] Finally, the 2D-3D point pair relationship and the internal parameters of the image acquisition unit can be used to solve the relative relationship between the world coordinate system and the camera coordinate system through PnP to obtain the rotation vector and the offset vector, and then convert the rotation vector and the offset vector into Euler angles PYR.

[0200] Figure 7 Schematic diagram of the PnP solution principle of the embodiment of the present application, as shown in Figure 7As shown, the figure includes three coordinate systems, namely the world coordinate system, the camera coordinate system and the image coordinate system. In the image coordinate system, the center of the image plane 71 is point c, which is equivalent to the intersection of the Z axis of the camera coordinate system and the image plane 71, and the image plane 71 is the position where the image is formed. Point P is any point in the world coordinate system, and the intersection of the line connecting point P and the coordinate origin o of the camera coordinate system and the image plane 71 is point p. The distance from point o to point c is the focal length of the image acquisition unit. The dotted line represents the relationship between the world coordinate system and the camera coordinate system, where R is the rotation matrix and t is the offset matrix. The rotation matrix and the offset matrix represent the posture.

[0201] The embodiment of the present application proposes a glasses fitting solution based on 2D facial key points, which decomposes glasses fitting into three elements: accurate anchor points, stable size, and PYR deflection. This solution provides a convenient way to implement glasses fitting and other similar AR applications on the existing 2D facial key point system. The appropriate anchor point is determined by aligning the 2D facial key points with the glasses model, the face width is calculated by selecting appropriate facial key points to approximate the size of the glasses model, and the PnP is solved by the correspondence between the 2D facial key points and the general 3D facial model to obtain the PYR deflection, thereby realizing AR wearing of glasses.

[0202] The solution in the embodiment of the present application provides a convenient way to try on AR glasses for the existing 2D face key point system. In order to complete the glasses trial, there is also a trial technology using a depth camera, but it needs to overturn the existing 2D face key point system. Then in the 2D-based trial technology, the glasses model can be bound to the 3D face model, so the glasses trial problem is converted into a 2D-3D PnP solution problem.

[0203] In the embodiment of the present application, the anchor point is a 3D point associated with a 2D facial key point in the glasses model, and the point must satisfy the requirement that the projection point at each viewing angle coincides with the associated 2D point. The PnP algorithm is the Perspective-n-Point algorithm, which is a method for solving the motion of 3D to 2D point pairs. The method describes how to estimate the camera's pose when multiple 3D space points and their projection positions are known. PYR is the Euler angle, where P (Pitch), Y (Yaw), and R (Roll) are used to describe the rotational motion of a rigid body in three-dimensional space, decomposing the motion of a rigid body into three rotations around different axes.

[0204] Based on the foregoing embodiments, an embodiment of the present application provides an image processing device, which includes the units included, the modules included in the units, and the components included in the modules, which can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a CPU (Central Processing Unit), MPU (Microprocessor Unit), DSP (Digital Signal Processing) or FPGA (Field Programmable Gate Array), etc.

[0205] Figure 8 Schematic diagram of the composition structure of the image processing device according to the embodiment of the present application. Figure 8 As shown, the device 800 includes:

[0206] An extraction unit 801 is used to process the acquired image including the target object to obtain 2D key points of the target object;

[0207] A determining unit 802 is configured to obtain a general 3D model according to the type of the target object, wherein the general 3D model is applicable to a general target object;

[0208] An acquisition unit 803 is used to acquire a 3D model of the interactive resource determined for the target object;

[0209] The processing unit 804 is used to display the augmented reality (AR) effect after the target object is superimposed with the interactive resource according to the 2D key points, the general 3D model and the 3D model of the interactive resource.

[0210] In some embodiments, the processing unit 804 includes: an anchor point determination module, used to determine the anchor point according to the 2D key point and the 3D model of the interactive resource; a size determination module, used to determine the size of the interactive resource according to the 2D key point; a deflection angle determination module, used to determine the deflection angle of the target object according to the 2D key point and the general 3D model; a processing module, used to display the augmented reality AR effect after the target object and the interactive resource are superimposed based on the anchor point, the size of the interactive resource and the deflection angle of the target object.

[0211] In some embodiments, the processing unit 804 includes: the anchor point determination module, which is also used to determine the anchor point based on the 2D key point and the 3D model of the interactive resource; the deflection angle determination module, which is also used to determine the deflection angle of the target object based on the 2D key point and the general 3D model; the processing module is also used to display the augmented reality AR effect after the target object is superimposed on the interactive resource based on the deflection angle of the anchor point and the target object.

[0212] In some embodiments, the 3D model of the interactive resource is a general 3D model of the interactive resource, and the apparatus further comprises: a size determination unit, configured to determine the size of the interactive resource according to the 2D key points;

[0213] A superimposed 3D model determining unit, configured to determine the superimposed 3D model of the interactive resource according to the size of the interactive resource and the general 3D model of the interactive resource;

[0214] Correspondingly, the processing module includes: a processing component for displaying an augmented reality AR effect after the target object is superimposed with the interactive resource according to the anchor point, the superimposed 3D model of the interactive resource and the deflection angle of the target object.

[0215] In some embodiments, the anchor point determination module includes: a first key point determination component, used to determine a key point located at a specific position of the target object as a first key point among the 2D key points; a second key point determination component, used to determine a second key point among the key points corresponding to the 3D model of the interactive resource, which is a point that coincides with a projection point of the first key point on any same plane;

[0216] An anchor point determination component is used to determine the first key point and the second key point as anchor points.

[0217] In some embodiments, the size determination module includes: a target object size determination component, used to determine the size of the target object based on the 2D key points; a ratio determination component, used to determine the ratio between the size of the target object and the size of the image; a distance determination component, used to determine the distance from the target object to the display screen of the electronic device based on the ratio; a size determination component, used to determine the size of the interactive resource based on the distance.

[0218] In some embodiments, the deflection angle determination module includes: a point pair relationship determination component, used to align the 2D key points with the 3D points in the general 3D model to obtain a point pair relationship between the 2D space and the 3D space; a deflection angle determination component, used to determine the deflection angle of the target object based on the point pair relationship and internal parameters of the image acquisition unit that acquires the image.

[0219] In some embodiments, the deflection angle determination component includes: a deflection angle determination subcomponent, used to determine the relative relationship between the world coordinate system corresponding to the universal 3D model and the camera coordinate system corresponding to the image acquisition unit based on the point pair relationship and the internal parameters of the image acquisition unit; the deflection angle determination subcomponent is also used to determine the deflection angle of the target object based on the relative relationship.

[0220] In some embodiments, the deflection angle determination subcomponent includes: a relative relationship determination component, which is used to determine the relative relationship between the world coordinate system and the camera coordinate system based on the PnP algorithm, according to the point pair relationship and the internal parameters of the image acquisition unit; an Euler angle determination component, which is used to determine the Euler angle of the target object according to the relative relationship; the Euler angle determination component is also used to determine the Euler angle as the deflection angle of the target object.

[0221] In some embodiments, the device also includes: a focal length and optical center determination unit, used to determine the focal length and optical center of the image acquisition unit according to the length and width of the image; a distortion coefficient determination unit, used to set the distortion coefficient of the image acquisition unit to a preset value; and an internal parameter determination unit, used to determine the internal parameters of the image acquisition unit according to the focal length, the optical center and the distortion coefficient.

[0222] In some embodiments, the target object is a face. Correspondingly, the extraction unit 801 includes: an extraction module, which is used to process the acquired face image based on face detection and face alignment algorithms to obtain 2D face key points.

[0223] In some embodiments, the target object is a human face, and correspondingly, the target object size determination component includes: a face width determination component, used to determine the width of the face based on the acquired 2D face key points; the ratio determination component includes: a face ratio determination component, used to determine the ratio between the width of the face and the width of the acquired face image.

[0224] In some embodiments, the face width determination component includes: a key point determination component, which is used to determine N pairs of key points among the 2D face key points, wherein the N pairs of key points are points in the face contour area, and N is a natural number greater than or equal to 2; a key point distance determination component, which is used to determine the distance between each pair of key points in the N pairs of key points; a key point sequence determination component, which is used to sort the N pairs of key points from large to small according to the distance to obtain a key point sequence; a Euclidean distance determination component, which is used to determine the average Euclidean distance of M pairs of key points in the key point sequence, where M is a natural number greater than or equal to 2 and M is less than or equal to N; and a width determination component, which is used to determine the average Euclidean distance as the width of the face.

[0225] In some embodiments, the processing unit 804 includes: a target object 3D model determination unit, used to determine the 3D model of the target object based on the 2D key points and the general 3D model; a display unit, used to display the augmented reality AR effect after the target object and the interactive resource are superimposed based on the 3D model of the target object and the 3D model of the interactive resource.

[0226] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0227] It should be noted that in the embodiments of the present application, if the above-mentioned image processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a personal computer, server, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a ROM (Read Only Memory), a disk or an optical disk. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0228] Correspondingly, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps in the image processing method provided in the above embodiment are implemented.

[0229] Correspondingly, an embodiment of the present application provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned image processing method are implemented.

[0230] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0231] It should be noted that Fig. 9 A hardware entity diagram of an electronic device according to an embodiment of the present application is shown in FIG. Fig. 9 As shown, the hardware entity of the electronic device 900 includes: a processor 901, a communication interface 902 and a memory 903, wherein

[0232] The processor 901 generally controls the overall operation of the electronic device 900 .

[0233] The communication interface 902 may enable the electronic device 900 to communicate with other terminals or servers via a network.

[0234] The memory 903 is configured to store instructions and applications executable by the processor 901, and can also cache data to be processed or processed by the processor 901 and various modules in the electronic device 900 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through FLASH (flash memory) or RAM (Random Access Memory).

[0235] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0236] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0237] In addition, all functional units in the embodiments of the present application can be integrated into one processing module, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units. A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium, which, when executed, executes the steps of the above method embodiments; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, disks or optical disks.

[0238] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain a new method embodiment. The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain a new product embodiment. The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain a new method embodiment or device embodiment.

[0239] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Processing the acquired image including the target object to obtain 2D key points of the target object; Acquire a general 3D model according to the type of the target object, where the general 3D model is applicable to a general target object; Acquire a 3D model of an interactive resource determined for the target object; the 3D model of the interactive resource includes a 3D model of a general interactive resource; According to the 2D key points, the general 3D model and the 3D model of the interactive resource, displaying an augmented reality (AR) effect after the target object and the interactive resource are superimposed; The step of displaying the augmented reality (AR) effect after the target object and the interactive resource are superimposed based on the 2D key points, the general 3D model and the 3D model of the interactive resource comprises: Determining an anchor point according to the 2D key point and the 3D model of the interactive resource; Determining the size of the interactive resource according to the 2D key points; Determining the deflection angle of the target object according to the 2D key points and the general 3D model; According to the anchor point, the size of the interactive resource and the deflection angle of the target object, an augmented reality (AR) effect after the target object and the interactive resource are superimposed is displayed; or, Determining an anchor point according to the 2D key point and the 3D model of the interactive resource; Determining the deflection angle of the target object according to the 2D key points and the general 3D model; According to the deflection angle between the anchor point and the target object, an augmented reality (AR) effect after the target object and the interactive resource are superimposed is displayed.

2. The method according to claim 1, characterized in that The method further comprises: Determining the size of the interactive resource according to the 2D key points; Determining a superimposed 3D model of the interactive resource according to the size of the interactive resource and the general 3D model of the interactive resource; Correspondingly, the display of the augmented reality AR effect after the target object and the interactive resource are superimposed according to the deflection angle of the anchor point and the target object includes: displaying the augmented reality AR effect after the target object and the interactive resource are superimposed according to the anchor point, the superimposed 3D model of the interactive resource and the deflection angle of the target object.

3. The method according to claim 1, characterized in that Determining the anchor point according to the 2D key point and the 3D model of the interactive resource includes: Among the 2D key points, a key point located at a specific position of the target object is determined as a first key point; Among the key points corresponding to the 3D model of the interactive resource, a point that coincides with a projection point of the first key point on any same plane is used to determine a second key point; The first key point and the second key point are determined as anchor points.

4. The method according to claim 1, characterized in that: The method is applied to an electronic device with a display screen; correspondingly, determining the size of the interactive resource according to the 2D key point includes: Determine the size of the target object according to the 2D key points; determining a ratio between a size of the target object and a size of the image; Determining the distance from the target object to the display screen of the electronic device according to the ratio; According to the distance, a size of the interactive resource is determined.

5. The method according to claim 1, characterized in that The step of determining the deflection angle of the target object according to the 2D key points and the general 3D model comprises: Aligning the 2D key points with the 3D points in the general 3D model to obtain a point pair relationship between the 2D space and the 3D space; The deflection angle of the target object is determined according to the point pair relationship and internal parameters of an image acquisition unit that acquires the image.

6. The method according to claim 5, characterized in that The step of determining the deflection angle of the target object according to the point pair relationship and internal parameters of an image acquisition unit that acquires the image comprises: Determining a relative relationship between a world coordinate system corresponding to the universal 3D model and a camera coordinate system corresponding to the image acquisition unit according to the point pair relationship and internal parameters of the image acquisition unit; According to the relative relationship, the deflection angle of the target object is determined.

7. The method according to claim 6, characterized in that Determining the relative relationship between the world coordinate system corresponding to the universal 3D model and the camera coordinate system corresponding to the image acquisition unit according to the point pair relationship and the internal parameters of the image acquisition unit includes: Based on the PnP algorithm, determining the relative relationship between the world coordinate system and the camera coordinate system according to the point pair relationship and the internal parameters of the image acquisition unit; Determining the Euler angle of the target object according to the relative relationship; The Euler angle is determined as the deflection angle of the target object.

8. The method according to claim 5, characterized in that The method further comprises: Determining the focal length and optical center of the image acquisition unit according to the length and width of the image; Setting the distortion coefficient of the image acquisition unit to a preset value; The internal parameters of the image acquisition unit are determined according to the focal length, the optical center and the distortion coefficient.

9. The method according to any one of claims 1 to 8, characterized in that: The target object is a human face; correspondingly, the acquired image including the target object is processed to obtain 2D key points of the target object, including: Based on face detection and face alignment algorithms, the acquired face images are processed to obtain 2D face key points.

10. The method according to claim 4, characterized in that The target object is a human face; correspondingly, determining the size of the target object according to the 2D key points includes: determining the width of the human face according to the acquired 2D human face key points; Determining the ratio between the size of the target object and the size of the image includes: determining the ratio between the width of the human face and the width of the acquired human face image.

11. The method according to claim 10, characterized in that Determining the width of the face according to the acquired 2D face key points includes: Determine N pairs of key points among the 2D face key points, where the N pairs of key points are points in the face contour area, and N is a natural number greater than or equal to 2; Determine the distance between each pair of key points in the N pairs of key points; Sort the N pairs of key points according to distance from large to small to obtain a key point sequence; Determine an average Euclidean distance of M pairs of key points in the key point sequence, where M is a natural number greater than or equal to 2 and M is less than or equal to N; The average Euclidean distance is determined as the width of the face.

12. An image processing device, characterized in that: The device comprises: An extraction unit, used to process the acquired image including the target object to obtain 2D key points of the target object; A determination unit, configured to obtain a general 3D model according to the type of the target object, wherein the general 3D model is applicable to a general target object; An acquisition unit, configured to acquire a 3D model of an interactive resource determined for the target object; the 3D model of the interactive resource includes a 3D model of a general interactive resource; A processing unit, configured to display an augmented reality (AR) effect after the target object and the interactive resource are superimposed based on the 2D key points, the general 3D model, and the 3D model of the interactive resource; The processing unit is specifically used to determine an anchor point according to the 2D key point and the 3D model of the interactive resource; determine the size of the interactive resource according to the 2D key point; determine the deflection angle of the target object according to the 2D key point and the general 3D model; and display the augmented reality AR effect after the target object and the interactive resource are superimposed according to the anchor point, the size of the interactive resource and the deflection angle of the target object; or, determine the anchor point according to the 2D key point and the 3D model of the interactive resource; determine the deflection angle of the target object according to the 2D key point and the general 3D model; and display the augmented reality AR effect after the target object and the interactive resource are superimposed according to the anchor point and the deflection angle of the target object.

13. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps of the image processing method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 11 are implemented.

Images