A semantic point determination method, device, electronic device and computer-readable storage medium

Abstract

The embodiments of the present invention provide a semantic point determination method, device, electronic device and computer-readable storage medium, the method comprising: obtaining at least one image of the user according to the user's 3D face model, and determining the key points in the image; traversing all position points in the 3D face model, and when the target position point meets the position relationship condition with the key point, determining the target position point as a semantic point with a corresponding relationship with the key point. There is no need to manually mark the key points of the image in the 3D face model, which reduces the cost of manual calibration and greatly improves the efficiency of binding key points and semantic points.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a semantic point determination method, device, electronic device and computer-readable storage medium. Background Art

[0002] In the monocular 3D face reconstruction task, it is generally necessary to locate the 3D semantic points corresponding to the 2D face key points.

[0003] In the prior art, it is necessary to observe 2D facial key points and manually determine 3D semantic points using 3D software.

[0004] Since 3D semantic points are manually calibrated using 3D software and the number of semantic points is huge, this method is very time-consuming and manual calibration is prone to errors. Summary of the invention

[0005] The objectives of the present invention include, for example, providing a semantic point determination method, device, electronic device and computer-readable storage medium, which can quickly realize the calibration of key points on a two-dimensional image on a 3D model.

[0006] In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for determining a semantic point, the method comprising:

[0008] Acquire at least one image of the user according to the 3D face model of the user;

[0009] For the image, determining key points in the image;

[0010] All position points in the 3D face model are traversed, and when a target position point satisfies a position relationship condition with the key point, the target position point is determined to be a semantic point having a corresponding relationship with the key point.

[0011] In an optional implementation, the step of traversing all position points in the 3D face model and, when a target position point satisfies a position relationship condition with the key point, determining that the target position point is a semantic point having a corresponding relationship with the key point comprises:

[0012] Determine the first coordinates of each of the key points;

[0013] Obtaining second coordinates of all position points in the 3D face model, wherein the second coordinates are two-dimensional coordinates;

[0014] Determine, according to each of the first coordinates and each of the second coordinates, a target position point among all the position points that satisfies a position relationship condition with the key point;

[0015] The target position point is determined to be a semantic point having a corresponding relationship with the key point.

[0016] In an optional implementation, the step of determining, according to each of the first coordinates and each of the second coordinates, a target position point among all the position points that satisfies a position relationship condition with the key point comprises:

[0017] For each of the key points, respectively calculating the distance from the key point to all the position points;

[0018] Get the location point corresponding to the minimum distance as the target location point.

[0019] In an optional implementation, the image includes a front view and / or a left view and / or a right view and / or a bottom view of the 3D face model, and the step of determining key points in the image includes:

[0020] When the image is a main view, determining that the key points in the main view are frontal key points of a human face;

[0021] and / or

[0022] When the image is a left view, determining the key points in the left view to be key points on the left side of the face;

[0023] and / or

[0024] When the image is a right view, determining the key points in the right view to be key points on the right side of the face;

[0025] and / or

[0026] When the image is a bottom view, determining the key points in the bottom view to be key points on the lower side of the face;

[0027] The step of traversing all position points in the 3D face model and determining that the target position point is a semantic point having a corresponding relationship with the key point when the target position point satisfies the position relationship condition with the key point comprises:

[0028] Traversing all position points in the 3D face model, when a target position point satisfies a position relationship condition with the front face key point, determining the target position point as a front face semantic point having a corresponding relationship with the front face key point;

[0029] and / or

[0030] Traversing all position points in the 3D face model, when the target position point satisfies the position relationship condition with the left key point of the face, determining the target position point as a left semantic point of the face having a corresponding relationship with the left key point of the face;

[0031] and / or

[0032] Traversing all position points in the 3D face model, when the target position point satisfies the position relationship condition with the key point on the right side of the face, determining the target position point as a semantic point on the right side of the face having a corresponding relationship with the key point on the right side of the face;

[0033] and / or

[0034] All position points in the 3D face model are traversed, and when a target position point satisfies a position relationship condition with the key point on the lower side of the face, the target position point is determined to be a semantic point on the lower side of the face having a corresponding relationship with the key point on the lower side of the face.

[0035] In an optional embodiment, the method further comprises:

[0036] The front semantic point of the face, the right semantic point of the face, the left semantic point of the face and the bottom semantic point of the face are used as a semantic point set. In an optional implementation, the method further includes:

[0037] Reconstructing a 3D face model based on the correspondence between the key points and the semantic points;

[0038] Based on the reconstructed 3D face model, AR face 3D special effects and virtual live broadcast are established.

[0039] In a second aspect, an embodiment of the present application provides a semantic point determination device, the device comprising:

[0040] An acquisition module, used to acquire at least one image of the user based on the 3D face model of the user;

[0041] A first determination module is used to determine key points in the image with respect to the image;

[0042] The second determination module is used to traverse all position points in the 3D face model, and when the target position point meets the position relationship condition with the key point, determine that the target position point is a semantic point having a corresponding relationship with the key point.

[0043] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the semantic point determination method when executing the computer program.

[0044] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the semantic point determination method when executed by a processor.

[0045] This application has the following beneficial effects:

[0046] This application obtains at least one image of the user based on the user's 3D face model, determines the key points in the image; traverses all the position points in the 3D face model, and when the target position point meets the position relationship conditions with the key point, determines that the target position point is a semantic point with a corresponding relationship with the key point. There is no need to manually mark the key points of the image in the 3D face model, which reduces the cost of manual calibration and greatly improves the efficiency of binding key points and semantic points. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 A block diagram of an electronic device provided by an embodiment of the present invention;

[0049] Figure 2 One of the flowcharts of a semantic point determination method provided by an embodiment of the present invention;

[0050] Figure 3 A schematic diagram of a 3D face model provided by an embodiment of the present invention;

[0051] Figure 4 A second flow chart of a method for determining a semantic point provided by an embodiment of the present invention;

[0052] Figure 5 A third flowchart of a method for determining a semantic point provided by an embodiment of the present invention;

[0053] Figure 6 A front view of a 3D face model provided by an embodiment of the present invention;

[0054] Figure 7 A left view of a 3D face model provided by an embodiment of the present invention;

[0055] Figure 8 A right view of the 3D face model provided by an embodiment of the present invention;

[0056] Fig. 9A bottom view of a 3D face model provided by an embodiment of the present invention;

[0057] Fig.10 A structural block diagram of a semantic point determination device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0059] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0061] In the description of the present invention, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. appear to indicate an orientation or position relationship, they are based on the orientation or position relationship shown in the accompanying drawings, or are the orientation or position relationship in which the product of the invention is usually placed when used. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0062] In addition, the terms “first”, “second”, etc., if used, are merely used to distinguish between the descriptions and should not be understood as indicating or implying relative importance.

[0063] In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0064] In some solutions for determining semantic points, it is usually necessary to observe 2D facial key points and manually determine 3D semantic points using 3D software.

[0065] However, after extensive research, the inventors found that this determination method is very time-consuming due to the large number of semantic points, and manual calibration is prone to errors.

[0066] In view of the discovery of the above problems, the present embodiment provides a semantic point determination method, device, electronic device and computer-readable storage medium, which can improve the problem of low efficiency of manual semantic point calibration. The solution provided by this embodiment is described in detail below.

[0067] This embodiment provides an electronic device that can determine semantic points. In a possible implementation, the electronic device can be a user terminal, for example, the electronic device can be, but is not limited to, a server, a smart phone, a personal computer (PC), a tablet computer, a personal digital assistant (PDA), a mobile Internet device (MID), and an image acquisition device.

[0068] The electronic device may have an image acquisition component (e.g., a camera); or a data interface capable of communicating with the image acquisition component, thereby acquiring an image acquired by the image acquisition component. The electronic device may also have a component capable of processing semantic point determination, such as a central processing unit (CPU), a graphics processing unit (GPU), etc., so as to execute the semantic point determination method provided in this embodiment.

[0069] In another possible implementation, the electronic device may also be a server capable of communicating with the user terminal. The server may obtain the user's 3D face model and the user's image, determine key points in the image, and establish a correspondence between the semantic points of the 3D face model and the key points in the image.

[0070] Please refer to Figure 1 The electronic device 100 is a schematic diagram of the structure of the electronic device 100. The electronic device 100 includes a semantic point determination device 110, a memory 120, a processor 130 and a communication unit 140.

[0071] The memory 120, the processor 130 and the communication unit 140 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The semantic point determination device 110 includes at least one software function module that can be stored in the memory 120 in the form of software or firmware or solidified in the operating system (OS) of the electronic device 100. The processor 130 is used to execute the executable modules stored in the memory 120, such as the software function modules and computer programs included in the semantic point determination device 110. When the computer executable instructions in the semantic point determination device 110 are executed by the processor, the semantic point determination method is implemented.

[0072] The memory 120 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electric erasable programmable read-only memory (EEPROM), etc. The memory 120 is used to store a program, and the processor 130 executes the program after receiving an execution instruction. The communication unit 140 is used to send and receive data through a network.

[0073] The processor 130 may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0074] Please refer to Figure 2 , Figure 2 For application Figure 1A flowchart of a semantic point determination method of the electronic device 100 is shown, and the various steps of the method will be described in detail below.

[0075] Step 201: Obtain at least one image of the user based on the user's 3D face model.

[0076] Step 202: For an image, determine key points in the image.

[0077] Step 203: traverse all position points in the 3D face model, and when the target position point meets the position relationship condition with the key point, determine the target position point as a semantic point having a corresponding relationship with the key point.

[0078] In this embodiment, the 3D face model may be a pre-established 3D face model, or a 3D face model of the user may be established in real time.

[0079] It should be noted that the user's 3D face model can be a 3D face model with texture, such as Figure 3 As shown, it is a schematic diagram of a 3D face model with texture of a user.

[0080] Acquire at least one user image of the 3D face model. The method of acquiring at least one user image of the 3D face model may be: capture or take photos of the 3D face model at different angles to obtain the image of the user corresponding to the 3D face model. Multiple user images of the 3D face model may be acquired, and the acquired images may be acquired at different angles. If one user image can acquire all the key points of the user, then only one image is required. If one user image does not cover all the key points of the user's face, then multiple images are acquired to ensure that all the key points of the user's face can be acquired.

[0081] Images at different angles correspond to different key points. For example, the left view of a 3D face model corresponds to the key points of the left view, and the right view of a 3D face model corresponds to the key points of the right view.

[0082] The key points in the image can be determined in the following ways:

[0083] Method 1: Images at different angles correspond to different key points. The corresponding relationship between images at different angles and key points is established, and the key points in the image can be determined based on the image.

[0084] Method 2: Determine the target object in the face image and determine the key points corresponding to the target object.

[0085] For example, the target object may be the user's eyes, mouth, eyebrows, ears, chin, nose, etc. The embodiment of the present invention does not impose any specific limitation on this. If the acquired image contains a mouth, nose, and eyebrows, then acquiring the key points of the mouth, nose, and eyebrows in the image is to determine the key points in the image.

[0086] When the image only contains a portion of the mouth or nose, the key points of the portion of the image containing the mouth or nose are determined. To determine the key points of the mouth or nose of another part of the user's face, it is necessary to obtain an image from another angle to obtain the key points corresponding to all the facial features of the user.

[0087] When determining the semantic points in the 3D face model based on the key points, it is necessary to obtain all the position points in the 3D face model. It can be understood that the 3D face model is composed of a large number of position points. Determine the positional relationship between each key point and all the positions in the 3D face model, obtain the position point that meets the positional relationship conditions as the target position point, establish the corresponding relationship between the target position point and the key point, and thus realize the binding of the key point in the 2D image with the target key point in the 3D face model.

[0088] In the process of determining the target position points, different images may contain the same key points. For example, the first image contains 50 key points, the second image contains 50 key points, and 20 key points of the first image and the second image overlap. After the first image determines 50 key points, the second image has 20 overlapping key points. To determine the key points of the second image, the 20 overlapping key points need to be removed. That is, the second image determines 30 key points, and the key points in the first image and the key points in the second image are different.

[0089] This application obtains at least one image of the user based on the user's 3D face model, determines the key points in the image; traverses all the position points in the 3D face model, and when the target position point meets the position relationship conditions with the key point, determines that the target position point is a semantic point with a corresponding relationship with the key point. There is no need to manually mark the key points of the image in the 3D face model, which reduces the cost of manual calibration and greatly improves the efficiency of binding key points and semantic points.

[0090] In order to improve the efficiency of determining the target location point, with respect to the above step 203, in another embodiment of the present application, as Figure 4 As shown, a semantic point binding method is provided, which specifically includes the following steps:

[0091] Step 203 - 1 : Determine the first coordinates of each key point.

[0092] Step 203 - 2: Obtain the second coordinates of all position points in the 3D face model.

[0093] Step 203 - 3 : Determine the target position point that satisfies the position relationship condition with the key point among all the position points according to the first coordinates and the second coordinates.

[0094] Step 203 - 4 : Determine that the target location point is a semantic point having a corresponding relationship with the key point.

[0095] It should be noted that, in order to facilitate the calculation of the distance between the key points and the position points, the coordinates of the key points in the image and all the position points in the 3D face model are two-dimensional coordinate points.

[0096] For example, the electronic device 100 determines the key points in the image, and the electronic device 100 determines the first coordinates corresponding to the key points. The second coordinates of all the position points in the 3D face model are determined. Since the coordinate points of the 3D face model are generally three-dimensional coordinates, and since the target position points need to be determined from all the position points based on the coordinate points of the key points, the second coordinates of all the position points in the 3D face model are two-dimensional coordinates, so as to facilitate the calculation of the first coordinates and the second coordinates.

[0097] Specifically, for each key point, the distance from the key point to all position points is calculated respectively; and the position point corresponding to the minimum distance is obtained as the target position point.

[0098] Specifically, the electronic device 100 determines the first coordinate of each key point, determines the second coordinates of all position points in the 3D face model, and calculates, for each key point, the distance between the key point and all position points in the 3D face model.

[0099] Based on the distance formula between two points, the distance between the key point and the position point is calculated based on the first coordinate and the second coordinate. For example, if the coordinates of a key point are (x1, y1) and the coordinates of a position point are (x2, y2), then the distance from the key point to the position point is:

[0100] The above calculation is performed on the first coordinate of each key point and each position point respectively to obtain the distance between each key point and all the position points, and the position point corresponding to the minimum distance is obtained, which is the target position point.

[0101] In addition to traversing all position points in the 3D face model and determining the target position point that satisfies the position relationship conditions with the key points, it is also possible to: determine the key points in the image and determine the angle of the current image, based on the angle of the image, determine the position points in the 3D face model at the same angle, traverse all position points at the angle, determine the target position point that satisfies the position relationship conditions with the key points, and determine the target position point as a semantic point that has a corresponding relationship with the key point.

[0102] The above method can reduce the amount of calculation and make the key points and the target position points in the 3D face model more efficiently bound.

[0103] For example: taking the facial features in an image as an example, first extract the 2D key points of the facial features of a certain image in the 3D face model, extract 80 key points, each key point is an x, y coordinate, determine all the position points that constitute the 3D face model, all the position points are 40,000 position points, determine the two-dimensional coordinates of the position points, calculate the distance from the 80 key points to the 40,000 position points, and get the 80 closest points from the 40,000 position points, which are the target position points. These 80 target position points are the 3D semantic points of the facial features.

[0104] In another optional method, taking the facial features in the image as an example, first extract the 2D key points of the facial features in the front view of the 3D face model, extract 80 key points, each of which is an x, y coordinate, determine the angle of the front view, and determine that all the position points of the 3D face model at the same angle are 2000 position points based on the angle. Determine the two-dimensional coordinates of the position points, calculate the distance from the 80 key points to the 2000 position points, and obtain the 80 closest points from the 2000 position points as the target position points. These 80 target position points are the 3D semantic points of the facial features.

[0105] The above process is adopted for different images until all semantic points in the 3D face model are determined.

[0106] This application obtains at least one image of the user based on the user's 3D face model, determines the key points in the image and the first coordinates of the key points; determines the second coordinates of all position points in the 3D face model, traverses all position points in the 3D face model, calculates the distance between the first coordinate and the second coordinate, and obtains the point with the closest distance as the target position point, and determines the target position point as a semantic point with a corresponding relationship with the key point. There is no need to manually mark the key points of the image in the 3D face model, which reduces the cost of manual calibration and greatly improves the efficiency of binding key points and semantic points. And directly calculate the distance between the position point and the key point to determine that the position point with the smallest distance is the target position point. The calculation method is simple and easy to implement.

[0107] In order to avoid repeated calculation of key points in the image and determine semantic points based on the key points in the image, with respect to the above steps 202 and 203, in another embodiment of the present application, as shown in FIG. Figure 5 As shown, a semantic point binding method is provided, which specifically includes the following steps:

[0108] Step 202-1: When the image is a main view, determine that the key points in the main view are front face key points.

[0109] When the image is a left view, the key points in the left view are determined to be the key points on the left side of the face. When the image is a right view, the key points in the right view are determined to be the key points on the right side of the face. When the image is an upward view, the key points in the upward view are determined to be the key points on the lower side of the face.

[0110] like Figure 6 This is the main view of the 3D face model. Figure 7 This is the left view of the 3D face model. Figure 8 This is the right view of the 3D face model. Fig. 9 This is a bottom view of the 3D face model.

[0111] Figure 6 In the figure, the points at the eyebrows, nose, eyes, and mouth are the key points of the front face.

[0112] In the present application, the key points in the main view, left view, right view and upward view of the 3D face are pre-set. The key points of the front face can be determined based on the main view, the key points of the left side of the face can be determined based on the left view, the key points of the right side of the face can be determined based on the right view, and the key points of the lower side of the face can be determined based on the upward view. For example: the number of all the key points of the face is 400. When the main view in the 3D face model is obtained, the key points numbered 1-100, i.e., the key points of the front face, can be obtained. When the left view in the 3D face model is obtained, the key points numbered 101-200, i.e., the key points on the left side of the face, can be obtained. When the right view in the 3D face model is obtained, the key points numbered 201-300, i.e., the key points on the right side of the face, are obtained. When the upward view in the 3D face model is obtained, the key points 301-400, i.e., the key points on the lower side of the face, are obtained.

[0113] As a result, when determining the key points in the face, there will be no duplication, and the calculation of the key points and the position points in the 3D face model will not be repeated, thereby improving the calibration efficiency of the semantic points.

[0114] Step 203-5: traverse all position points in the 3D face model, and when the target position point meets the position relationship condition with the front face key point, determine the target position point as a front face semantic point having a corresponding relationship with the front face key point.

[0115] When the image is the main view, determine the key points in the main view as the front face key points, determine all the position points in the 3D face model, and determine the first coordinates (x1, y1) of the front face key points and the second coordinates (x2, y2) of all the position points in the 3D face model. Based on the distance calculation formula between two points, Calculate the distance between each face front key point and each position point. Obtain the position point corresponding to the minimum distance as the target position point that meets the position relationship condition with the key point. Determine the target position point as the face front semantic point that has a corresponding relationship with the face key point.

[0116] Traverse all the position points in the 3D face model, and when the target position point meets the position relationship condition with the key point on the left side of the face, determine the target position point as the semantic point on the left side of the face that has a corresponding relationship with the key point on the left side of the face. Traverse all the position points in the 3D face model, and when the target position point meets the position relationship condition with the key point on the right side of the face, determine the target position point as the semantic point on the right side of the face that has a corresponding relationship with the key point on the right side of the face.

[0117] All position points in the 3D face model are traversed, and when the target position point meets the position relationship condition with the key point on the lower side of the face, the target position point is determined to be a semantic point on the lower side of the face that has a corresponding relationship with the key point on the lower side of the face.

[0118] When the image is a left view, a right view, and a bottom view, the semantic points in the 3D face model are determined by the above method for the main view, which will not be repeated here.

[0119] When there is at least one of the main view, the left view, the right view and the upward view, the key points in the main view and / or the left view and / or the right view and / or the upward view are determined, and the target position points in the 3D face model are determined based on the key points, and the face semantic points are determined based on the target position points. In order to reduce the amount of calculation, the front face key points in the main view, the left face key points in the left view, the right face key points in the right view, and the lower face key points in the upward view are all different.

[0120] The finally determined front face semantic points and / or left face semantic points and / or right face semantic points and / or lower face semantic points are taken as a semantic point set.

[0121] This application obtains at least one image of the user based on the user's 3D face model, determines the key points in the image; traverses all the position points in the 3D face model, and when the target position point meets the position relationship conditions with the key point, determines that the target position point is a semantic point with a corresponding relationship with the key point. There is no need to manually mark the key points of the image in the 3D face model, which reduces the cost of manual calibration and greatly improves the efficiency of binding key points and semantic points.

[0122] The semantic point determination method provided in the present application reconstructs a 3D face model based on the correspondence between key points and semantic points; and establishes AR face 3D special effects and virtual live broadcast based on the reconstructed 3D face model.

[0123] The 3D face model is reconstructed through the semantic point determination method described above, that is, based on the correspondence between the semantic points of the 3D face model and the key points of the face image, the 3D face model is reconstructed. The reconstructed 3D face model can be used for: creating 3D special effects for the host's face during live broadcast, building a virtual character based on the host's face and performing virtual live broadcast, etc. When the host makes an expression, the created special effects or the virtual character's expression are synchronized with the host's expression based on the correspondence between the key points of the face image and the semantic points of the 3D face model. Please refer to Fig.10 This embodiment also provides a method for applying Figure 1 The semantic point determination device 110 of the electronic device 100 includes:

[0124] The acquisition module 111 is used to acquire at least one image of the user according to the 3D face model of the user.

[0125] In this embodiment, the acquisition module 111 can be used to execute Figure 2 In step 201, for a detailed description of the acquisition module 111, reference may be made to the description of step 201.

[0126] The first determination module 112 is configured to determine key points in the image.

[0127] In this embodiment, the first determining module 112 may be used to execute Figure 2 As shown in step 202 , for a detailed description of the first determining module 112 , please refer to the description of step 202 .

[0128] The second determination module 113 is used to traverse all the position points in the 3D face model, and when the target position point meets the position relationship condition with the key point, determine that the target position point is a semantic point having a corresponding relationship with the key point. Figure 2As shown in step 203 , for a detailed description of the second determining module 113 , please refer to the description of step 203 .

[0129] Optionally, in some possible implementations, the second determining module 113 is specifically configured to:

[0130] Determine the first coordinates of each of the key points;

[0131] Obtaining second coordinates of all position points in the 3D face model, wherein the second coordinates are two-dimensional coordinates;

[0132] Determine, according to each of the first coordinates and each of the second coordinates, a target position point among all the position points that satisfies a position relationship condition with the key point;

[0133] The target position point is determined to be a semantic point having a corresponding relationship with the key point.

[0134] Optionally, in some possible implementations, the second determining module 113 is specifically configured to:

[0135] For each of the key points, respectively calculating the distance from the key point to all the position points;

[0136] Get the location point corresponding to the minimum distance as the target location point.

[0137] Optionally, in some possible implementations, the first determining module 112 is specifically configured to:

[0138] When the image is a main view, determining that the key points in the main view are frontal key points of a human face;

[0139] and / or

[0140] When the image is a left view, determining the key points in the left view to be key points on the left side of the face;

[0141] and / or

[0142] When the image is a right view, determining the key points in the right view to be key points on the right side of the face;

[0143] and / or

[0144] When the image is a bottom view, determining the key points in the bottom view to be key points on the lower side of the face;

[0145] The second determining module 113 is specifically used for:

[0146] Traversing all position points in the 3D face model, when a target position point satisfies a position relationship condition with the front face key point, determining the target position point as a front face semantic point having a corresponding relationship with the front face key point;

[0147] and / or

[0148] Traversing all position points in the 3D face model, when the target position point satisfies the position relationship condition with the left key point of the face, determining the target position point as a left semantic point of the face having a corresponding relationship with the left key point of the face;

[0149] and / or

[0150] Traversing all position points in the 3D face model, when the target position point satisfies the position relationship condition with the key point on the right side of the face, determining the target position point as a semantic point on the right side of the face having a corresponding relationship with the key point on the right side of the face;

[0151] and / or

[0152] All position points in the 3D face model are traversed, and when a target position point satisfies a position relationship condition with the lower face key point, the target position point is determined to be a lower face semantic point having a corresponding relationship with the lower face key point.

[0153] Optionally, in some possible implementations, the semantic point determining device 110 further includes:

[0154] The third determining module 114 is used to take the semantic points of the front face, the semantic points of the right face, the semantic points of the left face and the semantic points of the bottom face as a semantic point set.

[0155] Optionally, in some possible implementations, the semantic point determining device 110 further includes:

[0156] A first establishing module 115, configured to reconstruct a 3D face model based on the correspondence between the key points and the semantic points;

[0157] The second establishing module 116 is used to establish AR face 3D special effects and virtual live broadcast based on the reconstructed 3D face model.

[0158] In summary, the present application provides a semantic point determination method, device, electronic device and computer-readable storage medium, which obtains at least one image of the user according to the user's 3D face model to determine the key points in the image; traverses all position points in the 3D face model, and when the target position point meets the position relationship conditions with the key point, determines the target position point as a semantic point with a corresponding relationship with the key point. There is no need to manually mark the key points of the image in the 3D face model, which reduces the cost of manual calibration and greatly improves the efficiency of binding key points and semantic points.

[0159] The present application also provides an electronic device 100, which includes a processor 130 and a memory 120. The memory 120 stores computer executable instructions, and when the computer executable instructions are executed by the processor 130, the semantic point determination method is implemented.

[0160] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by the processor 130, the semantic point determination method is implemented.

[0161] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0162] In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. If the function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk.

[0163] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0164] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A semantic point determination method, characterized in that: The method comprises: Acquire at least one image of the user according to the 3D face model of the user; For the image, determining key points in the image; Determine the first coordinates of each of the key points; Obtaining second coordinates of all position points in the 3D face model, wherein the second coordinates are two-dimensional coordinates; For each of the key points, respectively calculating the distance from the key point to all the position points; Get the position point corresponding to the minimum distance as the target position point; The target position point is determined to be a semantic point having a corresponding relationship with the key point.

2. The method according to claim 1, characterized in that: The image includes a front view and / or a left view and / or a right view and / or a bottom view of the 3D face model, and the step of determining key points in the image includes: When the image is a main view, determining that the key points in the main view are frontal key points of a human face; and / or When the image is a left view, determining the key points in the left view to be key points on the left side of the face; and / or When the image is a right view, determining the key points in the right view to be key points on the right side of the face; and / or When the image is a bottom view, determining the key points in the bottom view to be key points on the lower side of the face; The steps of traversing all position points in the 3D face model, determining a target position point that satisfies a position relationship condition with a key point, and determining the target position point as a semantic point having a corresponding relationship with the key point include: Traversing all position points in the 3D face model, determining a target position point that satisfies a position relationship condition with a front face key point, and determining that the target position point is a front face semantic point that has a corresponding relationship with the front face key point; and / or Traversing all position points in the 3D face model, determining a target position point that satisfies a position relationship condition with a key point on the left side of the face, and determining that the target position point is a semantic point on the left side of the face that has a corresponding relationship with the key point on the left side of the face; and / or Traversing all position points in the 3D face model, determining a target position point that satisfies a position relationship condition with a key point on the right side of the face, and determining that the target position point is a semantic point on the right side of the face that has a corresponding relationship with the key point on the right side of the face; and / or All position points in the 3D face model are traversed to determine a target position point that satisfies a position relationship condition with a key point on the lower side of the face, and the target position point is determined to be a semantic point on the lower side of the face that has a corresponding relationship with the key point on the lower side of the face.

3. The method according to claim 2, characterized in that The method also includes: The front semantic point of the human face, the right semantic point of the human face, the left semantic point of the human face and the lower semantic point of the human face are taken as a semantic point set.

4. The method according to claim 1, characterized in that: The method further comprises: Reconstructing a 3D face model based on the correspondence between the key points and the semantic points; Based on the reconstructed 3D face model, AR face 3D special effects and virtual live broadcast are established.

5. A semantic point determination device, characterized in that: The device comprises: An acquisition module, used to acquire at least one image of the user based on the 3D face model of the user; A first determination module is used to determine key points in the image with respect to the image; The second determination module is used to determine the first coordinates of each of the key points; obtain the second coordinates of all the position points in the 3D face model, wherein the second coordinates are two-dimensional coordinates; for each of the key points, respectively calculate the distance from the key point to all the position points; obtain the position point corresponding to the minimum distance as the target position point; determine the target position point as a semantic point that has a corresponding relationship with the key point.

6. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the steps of the method according to any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images