Head size measurement method and device, head grid generation method and device, medium and equipment

Abstract

The invention provides a head size measurement method and a 3D head grid generation method, and relates to the technical field of artificial intelligence. According to the method, the facial local area is determined through the facial feature points, the circular reference object is extracted from the facial local area, and the head size of the target object is determined based on the size of the circular reference object, so that the 3D head grid is generated, and the accuracy is high.

Description

Background Art

[0002] Currently, commercial 3D scanners are the most widely used method for obtaining 3D physical head models. However, they are expensive and inconvenient, which limits their popularity and accessibility. Therefore, reconstructing 3D models from 2D images is both cheap and convenient and more suitable for the public.

[0003] There are many mature 3D head / face reconstruction methods, which mainly focus on applications that do not require high physical size accuracy. On the contrary, size is very important in ergonomic design. Current methods cannot derive 3D head / face models with sufficient size accuracy.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide a head size measurement method, a 3D head mesh generation method and a device thereof, which at least to some extent overcome the problem of inaccurate head size determination in the related art.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a head size measurement method is provided, comprising: acquiring a facial image of a target object, the facial image having a circular reference object at the forehead position of the target object; determining facial feature points of the target object; determining a facial local area including the circular reference object based on the facial feature points; extracting the circular reference object from the facial local area; and determining the head size of the target object based on the size of the circular reference object.

[0008] In one embodiment, determining the head size of the target object based on the size of the circular reference object includes: determining the diameter of the circular reference object; and determining the head size of the target object based on the diameter of the circular reference object.

[0009] In one embodiment, the facial feature points include eye feature points, and the facial partial image including the circular reference object is a rectangular area including the circular reference object.

[0010] In one embodiment, extracting the circular reference object from the local facial area includes: using an image segmentation artificial neural network to segment the local facial area to obtain the circular reference object; using an edge detector to detect the edge of the circular reference object; using an ellipse fitting method to perform ellipse measurement on the edge of the circular reference object, thereby extracting the circular reference object.

[0011] In one embodiment, the image segmentation artificial neural network is U-Net or a fully convolutional network FCN, and / or the edge detector is a canny edge detector.

[0012] In one embodiment, the method further includes: dedistorting the facial image of the target object based on camera parameters of an image acquisition device, wherein the camera parameters include distortion coefficients and an intrinsic matrix, and the distortion coefficients include radial distortion coefficients and tangential distortion coefficients.

[0013] In one embodiment, the method further comprises: acquiring multi-view images of a preset chessboard at different angles and distances by the image acquisition device, thereby determining camera parameters of the image acquisition device.

[0014] According to another aspect of the present disclosure, a 3D head mesh generation method is provided, comprising: acquiring multiple 2D face images of a target object, the 2D face images comprising a front face image, a right face side image, and a left face side image, the front face image having a circular reference object at a forehead position of the target object; reconstructing a 3D head shape of the target object based on the multiple 2D face images; determining a head size of the target object based on the front face image; and generating a 3D head mesh of the target object having the head size.

[0015] In one embodiment, determining the head size of the target object based on the frontal face image includes: determining the head size of the target object based on the frontal face image according to the above-mentioned head size measurement method.

[0016] In one embodiment, reconstructing the 3D head shape of the target object based on the multiple 2D face images includes:

[0017] The encoder of the 3DMM based on the head shape and texture is used to reconstruct the 3D head shape of the target object based on the multiple 2D face images to obtain the reconstructed 3D head shape.

[0018] In one embodiment, the method further comprises: pre-training the head shape and texture based 3DMM based on training data.

[0019] In one embodiment, the pre-training of the 3DMM based on head shape and texture based on training data includes: generating a parameterized head based on the training data using a non-rigid iterative closest point (NICP) algorithm; aligning the parameterized head with a uniform size using a generalized Procter & Gamble analysis (GPA); and extracting principal components (PCs) from all parameterized head meshes using PCA to obtain the 3DMM based on head shape and texture.

[0020] In one embodiment, a 3D head shape of the target object is reconstructed based on the multiple 2D face images using an encoder based on a 3DMM of head shape and texture, and obtaining the reconstructed 3D head shape includes: inputting the multiple 2D face images into an artificial neural network encoder to predict 3D head mesh parameters, the 3D head mesh parameters including 3DMM coefficients, lighting parameters and camera parameters of the 3D head mesh, to minimize a network loss function, the network loss function including: pixel-level loss L img (ξ), the perceptual identity loss L of the reconstructed face image ide (ξ), face edge loss L edge (ξ) and the projection feature point loss L lan (ξ).

[0021] In one embodiment, the method further comprises: using a pre-trained face recognition network to extract perceptual features used in perceptual identity loss calculation.

[0022] According to another aspect of the present disclosure, a 3D head mesh use method is provided, which is applied to a 3D head mesh generated according to the above-mentioned 3D head mesh generation method, and includes: using the 3D head mesh for headwear design.

[0023] In one embodiment, the headgear includes headphones, glasses, a helmet, a mask, or a hat.

[0024] According to another aspect of the present disclosure, a head size measurement device is provided, comprising: a facial image acquisition module, used to acquire a facial image of a target object, wherein the facial image has a circular reference object at the forehead position of the target object; a facial feature point determination module, used to determine the facial feature points of the target object; a local area determination module, used to determine a facial local area including the circular reference object based on the facial feature points; a reference object advance module, used to extract the circular reference object from the facial local area; and a head size determination module, used to determine a diameter of the circular reference object, and determine the head size of the target object based on the diameter of the circular reference object.

[0025] According to another aspect of the present disclosure, a 3D head mesh generation device is provided, including: a face image acquisition module, used to acquire multiple 2D face images of a target object, the 2D face images including a front face image, a right face side image, and a left face side image, the front face image having a circular reference object at the forehead position of the target object; a head shape reconstruction module, used to perform 3D head shape reconstruction on the target object based on the multiple 2D face images; a head size determination module, used to determine the head size of the target object based on the front face image; and a head mesh generation module, used to generate a 3D head mesh with the head size of the target object.

[0026] According to another aspect of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-mentioned head size measurement method, or 3D head mesh generation method by executing the executable instructions.

[0027] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the head size measurement method or the 3D head mesh generation method described above is implemented.

[0028] The head size measurement method and 3D head mesh generation method provided by the embodiments of the present disclosure determine the head size of the target object based on the size of a circular reference object, and have a high accuracy rate in size determination.

[0029] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0031] Figure 1 A schematic diagram showing the structure of a computer system in one embodiment of the present disclosure is shown;

[0032] Figure 2 A flow chart of a method for measuring head size in one embodiment of the present disclosure is shown;

[0033] Figure 3 A schematic diagram of a square mark in one embodiment of the present disclosure is shown;

[0034] Figure 4 A schematic diagram of a circular mark in one embodiment of the present disclosure is shown;

[0035] Figure 5A A flow chart of a circular reference object extraction method according to an embodiment of the present disclosure is shown;

[0036] Figure 5B A schematic diagram of a circular reference object measurement channel in one embodiment of the present disclosure is shown;

[0037] Fig. 6A A flowchart of a method for generating a D head mesh in one embodiment of the present disclosure is shown;

[0038] Figure 6B A main workflow diagram of a method for generating a 3D head mesh according to an embodiment of the present disclosure is shown;

[0039] Figure 7 A schematic diagram showing a 3DMM of a human head in one embodiment of the present disclosure is shown;

[0040] Figure 8 A schematic diagram showing a 3D head shape reconstruction framework in one embodiment of the present disclosure;

[0041] Fig. 9 shows measurement evaluation of different circular reference objects in one embodiment;

[0042] Fig.10 A comparison of the qualitative results of the two methods using circle or square markings in one embodiment is shown;

[0043] Fig.11 Shows the application scenarios of head-related products;

[0044] Fig.12 A schematic diagram of a head size measuring device in one embodiment of the present disclosure is shown;

[0045] Fig.13 A schematic diagram of a 3D head mesh generation device in one embodiment of the present disclosure is shown; and

[0046] Fig.14 A structural block diagram of a computer device in one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0047] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0048] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0049] The solution provided by the present application uses a circular object as a measurement reference, and performs 3D head mesh reconstruction based on the head size determined by the circular reference object, and applies it to ergonomic design. For ease of understanding, several terms involved in the present application are first explained below.

[0050] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0051] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0052] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further process images so that the computer processes them into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D (3-Dimension) technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0053] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0054] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0055] The solution provided in the embodiments of the present application involves artificial intelligence computer vision, machine learning and other technologies, which are specifically described by the following embodiments:

[0056] Figure 1 The schematic diagram of the structure of a computer system provided in one embodiment of the present disclosure is shown. The system includes: a plurality of terminals 120 and a server (or server cluster) 140.

[0057] The terminal 120 may be a mobile terminal such as a mobile phone, a game console, a tablet computer, an e-book reader, smart glasses, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a smart home device, an AR (Augmented Reality) device, a VR (Virtual Reality) device, or the terminal 120 may also be a personal computer (PC), such as a laptop computer and a desktop computer.

[0058] The terminal 120 may be installed with an application program for providing and implementing the method disclosed herein.

[0059] The terminal 120 is connected to the server 140 via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0060] The server 140 is a server, or is composed of several servers, or is a virtualization platform, or is a cloud computing service center. The server 140 is used to provide background services for applications that provide ... Optionally, the server 140 undertakes the main computing work, and the terminal 120 undertakes the secondary computing work; or, the server 140 undertakes the secondary computing work, and the terminal 120 undertakes the main computing work; or, the terminal 120 and the server 140 adopt a distributed computing architecture for collaborative computing.

[0061] In some optional embodiments, the server 140 is used to store information such as the 3D head model established through training.

[0062] Optionally, the client of the application installed in different terminals 120 is the same, or the client of the application installed on the two terminals 120 is the client of the same type of application of different control system platforms. Based on the different terminal platforms, the specific form of the client of the application may also be different, for example, the application client may be a mobile client, a PC client or a World Wide Web (Web) client.

[0063] Those skilled in the art will appreciate that the number of the terminals 120 may be more or less. For example, there may be only one terminal, or there may be dozens or hundreds of terminals, or a greater number. The embodiment of the present application does not limit the number and device type of the terminals.

[0064] Optionally, the system may also include a management device ( Figure 1(not shown), the management device is connected to the server 140 via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0065] Optionally, the above-mentioned wireless network or wired network uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0066] The various steps of the head size measurement and reconstruction method in this example implementation will be described in more detail below in conjunction with the accompanying drawings and embodiments.

[0067] Figure 2 The method provided by the embodiment of the present disclosure can be executed by any electronic device with computing and processing capabilities, such as Figure 1 The terminal 120 and / or the server 140 in the embodiment. In the following example description, the server 140 is used as the execution subject for example description.

[0068] like Figure 2As shown, S202, a facial image of a target object is obtained, and the facial image has a circular reference object at the forehead position of the target object. The target object refers to a user. In one embodiment, the facial image uses a frontal facial image of the user. In one embodiment, during the facial image capture process, the user maintains a neutral expression (eyes open, mouth closed) to avoid any expression effects in the 3D head reconstruction. The forehead position may include the middle position of the forehead. In one embodiment, the circular reference object is a printed circular mark, for example, a black circular mark printed on white paper. The circular reference object can also be a reference object similar to a coin, and the color of the circular reference object is not limited to black, and other colors that are easily distinguished from the skin color can also be used.

[0069] S204, determining facial feature points of the target object from the facial image. In one embodiment, the facial feature points of the target object include eye feature points, eyebrow feature points, or both. Eye feature points may include eye corner feature points and eyeball feature points; eyebrow feature points may be the center of the eyebrow. In one embodiment, facial feature points may include nose feature points,

[0070] S206, determining a facial local area including a circular reference object based on facial feature points. Based on the determined facial feature points, the position range of the circular reference object on the target object's face can be determined, thereby determining a facial local area including the circular reference object, which can be a rectangle or a square.

[0071] S208, extracting a circular reference object from the local facial region. The circular reference object can be extracted from the local facial region by an image segmentation network based on an artificial neural network, for example, a U-Net network or an FCN network based on a CNN network. The local facial region can be determined more accurately by facial feature points, and extracting a circular reference object from the local facial region can improve the accuracy and speed of processing.

[0072] In one embodiment, the circular reference object segmentation based on U-Net includes the following steps: Conv+Pooling downsampling the local facial area image; then Deconv deconvolution upsampling, cropping the previous low-level feature map, and performing feature fusion; then upsampling again. Repeat the above process until a feature map of a predetermined output size is obtained, and finally the output segment map is obtained through softmax. The advantages of the U-Net network include that only a small sample training set is required for training, it is a relatively small segmentation network, and the structure is simple. Through the previous local facial area segmentation, it can better adapt to the situation of U-net, and the segmentation accuracy is high.

[0073] S210, determining the head size of the target object based on the size of the circular reference object. The head size of the target object can be determined based on the extracted size of the circular reference object. The size of the circular reference object can be the diameter, circumference, or area of ​​the circular reference object.

[0074] In the above embodiment, the local facial area is determined by facial feature points, a circular reference object is extracted from the local facial area, and the head size of the target object is determined based on the size of the circular reference object, with high accuracy. Compared with reference objects of other shapes, the circular reference object is more suitable for the head area, has less distortion caused by deformation, and improves the measurement accuracy. Placing the circular reference object in the forehead area can reduce the deformation of the circular reference object.

[0075] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0076] Before introducing face image capture, a method for measuring camera parameters is first introduced.

[0077] Print a checkerboard marking template before facial image capture (see e.g. Figure 3 For example, a black chessboard mark can be printed on A4 paper.

[0078] Through these chessboard images, OpenCV is used to calibrate the camera, calculate the camera parameters (i.e., distortion coefficients and intrinsic matrices), and undistort the captured images.

[0079] Camera parameter measurement

[0080] Some cameras introduce significant distortion to the captured images, including radial distortion and tangential distortion. Camera calibration can be performed to determine the distortion coefficients (e.g., five distortion parameters), and then the distortion of the captured images is removed. In addition, the camera's intrinsic parameter matrix K is measured, which is used for image distortion removal and perspective transformation steps of 3D head reconstruction. The coefficients in this matrix include focal length (fx, fy) and optical center (cx, cy).

[0081]

[0082] In one embodiment, a mobile phone is used as a representative tool to capture multi-view images (e.g., width and height: 3024×4025 pixels). A pre-designed chessboard (10×7 squares and 9×6 inner corners) with a total size of 210×297 mm is printed and then placed on a flat surface (e.g., a table). The size of all squares in the chessboard is the same (each square is 26.7 mm), and the size of the square is an important parameter for determining the actual distance in camera calibration.

[0083] Multiple images of the chessboard are taken at different angles and distances, for example more than 10 images, to provide various square angles with different viewing angles for camera calibration. Using these chessboard images, camera calibration is performed using, for example, OpenCV to calculate camera parameters (i.e., distortion coefficients and intrinsic matrices), and the captured images can be dedistorted according to the obtained camera parameters.

[0084] An example of preparation of a circular reference object is described below.

[0085] Use a black and white printer with A4 paper to print the pre-designed circular markings for head size measurements. Print the template as Figure 4 As shown, 54 circular markers can be cut from a piece of printed paper, for example. The circular markers can be attached to the forehead of the participant, for example, where the area has a high degree of flatness.

[0086] Here is a brief description of how to use the circular reference object.

[0087] 3D head dimensions can be measured from 2D images using a black circle marker with a constant diameter (e.g., 15 mm) as a reference object. The target subject (participant) is asked to place this circle at a location on their forehead, such as the center of the forehead, which is relatively flatter than other facial areas. First, a frontal facial image with the marker attached to the participant's forehead is captured. Then, two face images without the marker are captured from the left and right (e.g., from 10°–50°). During the facial image capture process, the participants are instructed to maintain a neutral expression (eyes open, mouth closed) to avoid any expression effects in the 3D head reconstruction. After the facial image is captured, an image dedistortion method can be applied to it based on the camera parameters determined previously.

[0088] Figure 5A A flow chart of a circular reference object extraction method in an embodiment of the present disclosure is shown. Figure 5B A schematic diagram of the corresponding circular reference object measurement channel is shown.

[0089] like Figure 5A As shown, S502, a rectangular area is cropped on the facial image of the target object based on the detected facial feature points.

[0090] In face image preprocessing, facial feature points (e.g., Figure 5B (a) shows the white dots. Determine the facial rectangular area based on the facial feature points (e.g., Figure 5B The rectangular area shown in (a) is used to extract the local facial area for cropping. The cropped image is shown in Figure 5B (b), for example, the size is 256×256 pixels.

[0091] S504, using the pre-trained U-Net to perform accurate circular reference region segmentation to obtain a circular reference object, such as Figure 5B The black circular object shown in (c).

[0092] S506, using an edge detector, such as a canny edge detector, to filter the edge of the circular reference object to obtain the edge of the circular reference object, such as Figure 5B (d) as shown.

[0093] S508, performing ellipse fitting based on the edge of the circular reference object to obtain the diameter of the fitted ellipse, see Figure 5B The circular reference object shown in (e).

[0094] In the above embodiment, the facial local area can be determined relatively accurately by using facial feature points, and the circular reference object is extracted from the facial local area, which can improve the accuracy and speed of processing.

[0095] Fig. 6A A flowchart of a method for generating a 3D head mesh in one embodiment of the present disclosure is shown.

[0096] like Fig. 6A As shown, S602, multiple 2D face images of the target object are obtained, the 2D face images include a front face image, a right face side image, and a left face side image, and the front face image has a circular reference object at the forehead position of the target object.

[0097] S604: reconstructing a 3D head shape of the target object based on multiple 2D face images.

[0098] S606, determining the head size of the target object based on the front face image. Determine the head size of the target object based on the size of the circular reference object in the face proof image.

[0099] S608: Generate a 3D head mesh with head size of the target object.

[0100] In the above embodiment, the head size of the target object is determined based on the circular reference object in the frontal face image, thereby generating a 3D head mesh with accurate head size, which can meet high-precision size requirements and is applied to applications such as headwear design that have high requirements on head model size.

[0101] Figure 6B The main workflow diagram of the method for generating a 3D head mesh according to one embodiment of the present disclosure is shown. Figure 6B As shown, the workflow consists of 4 parts:

[0102] (1) 2D face image capture;

[0103] (2) 3D head shape reconstruction (using the encoder and 3DMM of head shape and texture);

[0104] (3) 3D head size measurement (by measuring the diameter of a circular reference object using U-Net); and

[0105] (4) Application scenarios of reconstructing head mesh.

[0106] The 3D head shape reconstruction is described in detail below.

[0107] In one embodiment, 3D head shape reconstruction is implemented based on 3DMM. 3DMM is the basis for subsequent 3D head shape reconstruction. A large number of head shape 3DMMs are created based on the existing training data set. In addition, a head texture 3DMM model is also created. A specific example of creating a 3DMM model includes the following steps:

[0108] (1) Generate a parameterized head from a training dataset using a non-rigid iterative closest point (NICP) algorithm, which can be, for example, Amberg B, Romdhani S, Vetter T. Optimal step-size non-rigid ICP algorithm for surface registration. The training dataset can include a head shape training dataset and a head texture training dataset.

[0109] (2) aligning the parameterized heads to a uniform size using Generalized Platts Analysis (GPA);

[0110] (3) Principal components (PCs) were extracted from all parameterized head meshes using principal component analysis (PCA) to build the 3DMM model.

[0111] The schematic diagram of the generated 3DMM model of the human head is as follows Figure 7 As shown, the period includes head shape and head texture models.

[0112] Figure 8 FIG. 2 is a schematic diagram showing a 3D head shape reconstruction framework in one embodiment of the present disclosure. Figure 8As shown, when multiple face images are input, a CNN-based encoder (such as ResNet50) is used to predict parameters, including a 3DMM coefficient of a head mesh (including head shape 801 and head texture 802 parameters), and a corresponding number of lighting and camera parameters 803, 804, and a differential generator (Differentiable Renderer) 805 is used to generate a fitting face image (Fitting face image) 806, a fitting face feature (Fitting face landmark) 807, 808, and a loss is determined in the ground-truth face image (Ground-truth Face Images) 809, the ground-truth face edge heatmap (Ground-truth Face EdgeHeatmap) 810, and the ground-truth face feature (Ground-truth Face landmark) 811. The parameters ξ of the 3DMM-based shape reconstruction method include α shp ,α tex ,γ light,k ,γ view,k , where k = 1, 2, ..., N I , N I is the number of input images. The main network loss functions that need to be minimized include the pixel-level loss L img (ξ), the perceptual identity loss L for rendered face images ide (ξ), face edge loss L edge (ξ) and the projected landmark loss L lan (ξ); Use a pre-trained face recognition network (Arcface) to extract perceptual features in the perceptual identity loss calculation.

[0113] Fig. 9 The measurement evaluation of different circular reference objects in one embodiment is shown to evaluate the measurement of different circular reference objects. The measurement method can accurately detect coins of different sizes and colors, for example, 1 Hong Kong dollar (diameter: 25.5 mm); 0.5 Hong Kong dollar (diameter: 22.5 mm); 0.1 Hong Kong dollar (diameter: 17.5 mm). Experiments show that the measurement method of the coarse to fine circular reference object disclosed in the present invention can accurately and reliably detect reference objects with different sizes and colors, such as coins, and the measurement method is accurate, robust and convenient. As a result, the head reconstruction method disclosed in the present invention has a more consistent facial contour and better reconstruction accuracy.

[0114] Fig.10 The following is a comparison of the qualitative results of the two methods using circle or square marks in one embodiment. Fig.10 It can be seen that the method of the present disclosure using circular marks has higher accuracy than using square marks.

[0115] Fig.11 The application scenarios of head-related products are shown, where the top represents a solid color reconstructed head and the bottom represents a reconstructed head with texture. The head model generated by the 3D head mesh reconstruction method disclosed in the present invention has higher dimensional accuracy and is more suitable for ergonomic design, such as headwear design, and can be used for personalized customization (personalization), virtual try-on, and size selection of face and head-related products. Application scenarios of head-related products include, for example: (a) headphones; (b) sunglasses; (c) VR glasses; (d) masks; (e) helmets.

[0116] Fig.12 FIG. 2 is a schematic diagram of a head size measuring device according to an embodiment of the present disclosure. Fig.12 As shown, the head size measuring device comprises:

[0117] A facial image acquisition module 1201 is used to acquire a facial image of a target object, wherein the facial image has a circular reference object at the forehead of the target object;

[0118] A facial feature point determination module 1202 is used to determine the facial feature points of the target object;

[0119] A local area determination module, configured to determine a facial local area including the circular reference object based on the facial feature points;

[0120] A reference object advance module 1203, configured to extract the circular reference object from the facial local area; and

[0121] The head size determination module 1204 is used to determine the diameter of the circular reference object, and determine the head size of the target object based on the diameter of the circular reference object.

[0122] Fig.13 FIG. 2 is a schematic diagram of a 3D head mesh generation device in one embodiment of the present disclosure. Fig.13 As shown, the 3D head mesh generation device includes:

[0123] A face image acquisition module 1301 is used to acquire multiple 2D face images of a target object, wherein the 2D face images include a front face image, a right face side image, and a left face side image, and the front face image has a circular reference object at the forehead position of the target object;

[0124] A head shape reconstruction module 1302 is used to reconstruct a 3D head shape of the target object based on the multiple 2D face images;

[0125] A head size determination module 1303, configured to determine the head size of the target object based on the front face image;

[0126] The head mesh generation module 1304 is used to generate a 3D head mesh having the head size of the target object.

[0127] In the solution of the disclosed embodiment, two high-resolution head shape and texture 3DMMs are created. The 3DMM-based 3D head shape reconstruction method uses a CNN-based encoder with higher accuracy. In the CNN-based encoder, a new feature point to edge loss is introduced to ensure the overall reconstructed shape accuracy. The experimental results determine the size and cost-effective number of input face images, which is very useful for practical applications.

[0128] In the solution of the embodiment of the present disclosure, a circular reference object is used instead of a square mark. The circular reference object is less sensitive to deformation when attached to the face, and is more stable and easy to measure the head size. In addition, the use of coin-shaped objects is also common in daily life. The accurate head size measurement method based on U-Net is more stable and convenient. The technical solution method of the present disclosure is more suitable for ergonomic design.

[0129] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0130] Refer to the following Fig.14 14 to 15. The electronic device 1400 according to this embodiment of the present invention is described. Fig.14 The electronic device 1400 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0131] like Fig.14 As shown, the electronic device 1400 is in the form of a general computing device. The components of the electronic device 1400 may include but are not limited to: at least one processing unit 1410, at least one storage unit 1420, and a bus 1430 connecting different system components (including the storage unit 1420 and the processing unit 1410).

[0132] The storage unit stores program codes, which can be executed by the processing unit 1410, so that the processing unit 1410 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 1410 can perform the following steps: Figure 2 and Fig. 6A Follow the steps shown in .

[0133] The storage unit 1420 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 14201 and / or a cache storage unit 14202 , and may further include a read-only storage unit (ROM) 14203 .

[0134] The storage unit 1420 may also include a program / utility 14204 having a set (at least one) of program modules 14205, such program modules 14205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0135] Bus 1430 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0136] The electronic device 1400 may also communicate with one or more external devices 1500 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1500, and / or communicate with any device that enables the electronic device 1400 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1450. Furthermore, the electronic device 1400 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1460. As shown, the network adapter 1460 communicates with other modules of the electronic device 1400 via a bus 1430. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0137] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0138] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0139] A program product for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0140] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0141] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0142] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0143] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0144] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0145] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0146] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0147] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A method for measuring head size, characterized in that: include: Acquire a facial image of a target object, wherein the facial image has a circular reference object at the forehead position of the target object; Determining facial feature points of the target object; Determine a facial local area including the circular reference object based on the facial feature points; Extracting the circular reference object from the local facial area; The size of the target subject's head is determined based on the size of the circular reference object.

2. The method according to claim 1, characterized in that Determining the head size of the target object based on the size of the circular reference object comprises: determining a diameter of the circular reference object; The target subject's head size is determined based on the diameter of the circular reference object.

3. The method according to claim 1, characterized in that The facial feature points include eye feature points, and the facial partial image including the circular reference object is a rectangular area including the circular reference object.

4. The method according to claim 1, 2 or 3, characterized in that: The extracting the circular reference object from the local facial area comprises: Using an image segmentation artificial neural network to segment the local facial area to obtain the circular reference object; detecting an edge of the circular reference object using an edge detector; The ellipse fitting method is used to perform ellipse measurement on the edge of the circular reference object, thereby extracting the circular reference object.

5. The method according to claim 4, characterized in that The image segmentation artificial neural network is U-Net or a fully convolutional network FCN, and / or the edge detector is a canny edge detector.

6. The method according to claim 1, characterized in that Also includes: The facial image of the target object is dedistorted based on camera parameters of an image acquisition device, wherein the camera parameters include distortion coefficients and an intrinsic matrix, and the distortion coefficients include radial distortion coefficients and tangential distortion coefficients.

7. The method according to claim 6, characterized in that Also includes: The image acquisition device is used to acquire multi-view images of a preset chessboard at different angles and distances, thereby determining the camera parameters of the image acquisition device.

8. A 3D head mesh generation method, characterized in that: include: Acquire multiple 2D face images of the target object, wherein the 2D face images include a front face image, a right face side image, and a left face side image, and the front face image has a circular reference object at the forehead position of the target object; Reconstructing a 3D head shape of the target object based on the multiple 2D face images; Determining the head size of the target object based on the front face image; A 3D head mesh with head dimensions of the target object is generated.

9. The method according to claim 8, characterized in that Determining the head size of the target object based on the front face image includes: The head size measurement method according to any one of claims 1 to 7 determines the head size of the target object based on the frontal face image.

10. The method according to claim 8, characterized in that The reconstructing the 3D head shape of the target object based on the multiple 2D face images comprises: The encoder of the 3DMM based on the head shape and texture is used to reconstruct the 3D head shape of the target object based on the multiple 2D face images to obtain the reconstructed 3D head shape.

11. The method according to claim 10, characterized in that Also includes: The head shape and texture based 3DMM is pre-trained based on training data.

12. The method according to claim 11 is characterized in that The pre-training of the 3DMM based on head shape and texture based on training data comprises: generating a parameterized head based on the training data using a non-rigid iterative closest point (NICP) algorithm; aligning the parameterized heads to a uniform size using Generic Platts Analysis (GPA); Principal components (PCs) are extracted from all parameterized head meshes using principal component analysis (PCA) to obtain the 3DMM based on head shape and texture.

13. The method according to any one of claims 10 to 12, characterized in that Reconstructing the 3D head shape of the target object based on the multiple 2D face images using an encoder of a 3DMM based on head shape and texture, and obtaining the reconstructed 3D head shape includes: The plurality of 2D face images are input into an artificial neural network encoder to predict 3D head mesh parameters, wherein the 3D head mesh parameters include 3DMM coefficients, lighting parameters, and camera parameters of the 3D head mesh, so as to minimize a network loss function, wherein the network loss function includes: a pixel-level loss L img (ξ), the perceptual identity loss L of the reconstructed face image ide (ξ), face edge loss L edge (ξ) and the projection feature point loss L lan (ξ).

14. The method according to claim 13, characterized in that Also includes: A pre-trained face recognition network is used to extract perceptual features used in the perceptual identity loss computation.

15. A method for using a 3D head mesh, applied to a 3D head mesh generated by the 3D head mesh generation method according to any one of claims 8 to 14, comprising: The 3D head mesh is used for headwear design.

16. The method according to claim 15, characterized in that The headwear includes headphones, glasses, a helmet, a mask, or a hat.

17. A head size measuring device, characterized in that: include: A facial image acquisition module, used to acquire a facial image of a target object, wherein the facial image has a circular reference object at the forehead position of the target object; A facial feature point determination module, used to determine the facial feature points of the target object; A local area determination module, configured to determine a facial local area including the circular reference object based on the facial feature points; A reference object advance module, used for extracting the circular reference object from the local facial area; and The head size determination module is used to determine the diameter of the circular reference object, and determine the head size of the target object based on the diameter of the circular reference object.

18. A 3D head mesh generation device, characterized in that: include: A face image acquisition module, used to acquire multiple 2D face images of a target object, wherein the 2D face images include a front face image, a right face side image, and a left face side image, and the front face image has a circular reference object at the forehead position of the target object; A head shape reconstruction module, used for reconstructing a 3D head shape of the target object based on the multiple 2D face images; A head size determination module, used to determine the head size of the target object based on the front face image; The head mesh generation module is used to generate a 3D head mesh with head size of the target object.

19. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the head size measurement method described in any one of claims 1 to 7, or the 3D head mesh generation method described in any one of claims 8 to 14 by executing the executable instructions.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the head size measurement method described in any one of 1 to 7 or the 3D head mesh generation method described in any one of claims 8 to 14 is implemented.

Images