Keypoint detection method, training method, device, electronic device, and computer program

Abstract

The present application provides a keypoint detection method, a training method, a device, an apparatus, a medium, and a product, which are applicable to the field of artificial intelligence technology. The keypoint detection method includes the steps of obtaining a 3D mesh representing an object to be detected, determining the vertices of the 3D mesh and the connection relationships between the vertices, performing feature extraction on the vertices of the 3D mesh to obtain vertex features, performing global feature extraction on the object to obtain global features based on the vertex features, performing local feature extraction on the object to obtain local features based on the vertex features and the connection relationships between the vertices, and combining the vertex features, global features, and local features to detect keypoints of the object to obtain positions of the keypoints of the object on the object.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to a Chinese patent application bearing application number 202211576832.9, filed with the China Patent Office on December 9, 2022, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the field of artificial intelligence technology, and in particular to a keypoint detection method, a training method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. [Background technology]

[0003] In related technologies, keypoint detection for 3D facial characters can generally be divided into two main types. The first type is a method based on traditional geometric analysis, and the second type is a method based on deep learning. In the first type of method, keypoint location methods based on geometric analysis rely heavily on manually set rules and are difficult to apply to various head model shapes, resulting in low robustness. On the other hand, in the second type of method, the 3D head model is basically rendered into a 2D image, and then features are extracted using a 2D convolutional neural network to detect corresponding keypoints. This inevitably results in a loss of 3D geometric information. As a result, the accuracy of keypoint detection for 3D facial characters in related technologies is low. Summary of the Invention [Means for solving the problem]

[0004] Embodiments of the present application provide a keypoint detection method, a method for training a three-dimensional network model, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product that can improve the accuracy of keypoint detection using a three-dimensional network model.

[0005] The technical solutions of the embodiments of the present application are realized as follows:

[0006] An embodiment of the present application provides a keypoint detection method, the method comprising: Obtaining a three-dimensional mesh to represent the detected object, and determining the vertices and the connection relationships between the vertices of the three-dimensional mesh; performing feature extraction on the vertices of the three-dimensional mesh to obtain vertex features of the three-dimensional mesh; performing global feature extraction on the detection object based on the vertex features to obtain global features of the detection object; and performing local feature extraction on the detection object based on the vertex features and the connection relationships between the vertices to obtain local features of the detection object; performing feature splicing based on the vertex features, the global features, and the local features, detecting key points of the detected object, and obtaining positions of the key points of the detected object on the detected object.

[0007] An embodiment of the present application provides a keypoint detection device, the device comprising: an acquisition module; a first feature extraction module; a second feature extraction module; and an output module; The acquisition module is configured to acquire a three-dimensional mesh representing the detected object and determine vertices and connectivity relationships between the vertices of the three-dimensional mesh; the first feature extraction module is configured to perform feature extraction on vertices of the 3D mesh to obtain vertex features of the 3D mesh; the second feature extraction module is configured to perform global feature extraction on the detection object based on the vertex features to obtain global features of the detection object, and to perform local feature extraction on the detection object based on the vertex features and the connection relationships between the vertices to obtain local features of the detection object; The output module is configured to perform detection on key points of the detected object based on the vertex features, the global features, and the local features, and obtain positions of the key points of the detected object on the detected object.

[0008] An embodiment of the present application provides a method for training a three-dimensional network model, the three-dimensional network model comprising at least a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, and an output layer, the method comprising: obtaining object training samples with labels, the labels being for indicating actual locations of keypoints of the object training samples; obtaining a training 3D mesh to represent the object training sample, and determining vertices and connectivity relationships between vertices of the training 3D mesh; performing feature extraction on the vertices of the object training samples via the first feature extraction layer to obtain vertex features of the training 3D mesh; performing global feature extraction on the object training samples based on vertex features of the training 3D mesh through the second feature extraction layer to obtain global features of the object training samples; and performing local feature extraction on the object training samples based on vertex features and inter-vertex connection relationships of the training 3D mesh through the third feature extraction layer to obtain local features of the object training samples; via the output layer, performing detection on keypoints of the object training samples based on vertex features of the training 3D mesh, global features of the object training samples, and local features of the object training samples, and determining positions of the keypoints of the object training samples on the object training samples; A step of obtaining the difference between the positions of the keypoints of the object training sample and the label, training the three-dimensional network model based on the difference, and obtaining a target three-dimensional network model, wherein the target three-dimensional network model is configured to perform keypoint detection on the detected object and obtain the positions of the keypoints of the detected object on the detected object.

[0009] An embodiment of the present application provides a training device for a three-dimensional network model, the three-dimensional network model including at least a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, and an output layer, the device including an acquisition module, an acquisition module, a first feature extraction module, a second feature extraction module, an output module, and an update module; The acquisition module is configured to acquire object training samples having labels, the labels being for indicating actual locations of keypoints of the object training samples; the acquisition module is configured to acquire a training 3D mesh to represent the object training sample, and determine vertices and connectivity relationships between vertices of the training 3D mesh; the first feature extraction module is configured to perform feature extraction on vertices of the object training samples via the first feature extraction layer to obtain vertex features of the training 3D mesh; the second feature extraction module is configured to perform global feature extraction on the object training sample based on vertex features of the training 3D mesh through the second feature extraction layer, to obtain global features of the object training sample; and to perform local feature extraction on the object training sample based on vertices and inter-vertex connection relationships of the training 3D mesh through the third feature extraction layer, to obtain local features of the object training sample; the output module is configured to perform detection on keypoints of the object training samples based on vertex features of the training 3D mesh, global features of the object training samples, and local features of the object training samples via the output layer, and determine positions of the keypoints of the object training samples on the object training samples; The update module is configured to obtain differences between the positions of keypoints of the object training sample and the label, train the 3D network model based on the differences, and obtain a target 3D network model, and the target 3D network model is configured to perform keypoint detection on the detected object and obtain the positions of keypoints of the detected object on the detected object.

[0010] An embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory configured to store executable instructions; The processor, when executing the computer-executable instructions stored in the memory, performs the keypoint detection method provided in the embodiments of the present application.

[0011] An embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory configured to store executable instructions; The processor, when executing the computer-executable instructions stored in the memory, performs the method for training a three-dimensional network model provided in the embodiments of the present application.

[0012] An embodiment of the present application provides a computer-readable storage medium having stored thereon computer-executable instructions that, when executed by a processor, cause the processor to perform a keypoint detection method provided in the embodiment of the present application.

[0013] An embodiment of the present application provides a computer-readable storage medium having stored thereon computer-executable instructions that, when executed by a processor, cause the processor to perform a method for training a three-dimensional network model provided in the embodiment of the present application.

[0014] An embodiment of the present application provides a computer program product, the computer program product including a computer program or computer-executable instructions, the computer program or computer-executable instructions being stored in a computer-readable storage medium, a processor of an electronic device reading the computer program or computer-executable instructions from the computer-readable storage medium and executing the computer program or computer-executable instructions, thereby causing the electronic device to perform the keypoint detection method provided in the embodiment of the present application.

[0015] An embodiment of the present application provides a computer program product, the computer program product including a computer program or computer-executable instructions, the computer program or computer-executable instructions being stored in a computer-readable storage medium, and a processor of an electronic device reading and executing the computer program or computer-executable instructions from the computer-readable storage medium, thereby causing the electronic device to perform the method for training a 3D network model provided in the embodiment of the present application.

[0016] The embodiments of the present application have the following beneficial effects:

[0017] After obtaining a 3D mesh corresponding to the object to be detected, a dual-path feature extraction layer is constructed to extract global and local features of the object to be detected based on the vertex features and the connection relationships between the vertices obtained from the 3D mesh, respectively, and thereby obtain the locations of keypoints on the object to be detected based on the vertex features obtained from the 3D mesh and the extracted global and local features. In this way, by extracting more feature information of the object to be detected through the multi-layer feature extraction layer, and then detecting keypoints on the object to be detected based on the rich feature information, the accuracy of 3D keypoint detection can be greatly improved. [Brief explanation of the drawings]

[0018] [Figure 1]1 is a schematic diagram illustrating the architecture of a keypoint detection system 100 provided in an embodiment of the present application. [Figure 2] 1 is a schematic diagram illustrating a configuration of an electronic device provided in an embodiment of the present application. [Figure 3] 1 is a flowchart of a keypoint detection method provided in an embodiment of the present application; [Figure 4] FIG. 1 is a schematic diagram of a three-dimensional mesh of a human head provided in an embodiment of the present application. [Figure 5] 1 is a flowchart for determining local features of each vertex provided in an embodiment of the present application; [Figure 6] FIG. 1 is a schematic diagram of employing the attention mechanism provided in an embodiment of the present application to determine the correlation degree between a reference vertex and other vertices; [Figure 7] FIG. 2 is a schematic diagram illustrating the locations of key points on a detection object provided in an embodiment of the present application. [Figure 8] FIG. 1 is a schematic diagram illustrating the configuration of a three-dimensional network model provided in an embodiment of the present application. [Figure 9] FIG. 2 is a schematic diagram showing the configuration of a third feature extraction layer provided in an embodiment of the present application. [Figure 10] FIG. 1 is a schematic diagram illustrating the configuration of a three-dimensional network model provided in an embodiment of the present application. [Figure 11] 1 is a flowchart of a training process for a three-dimensional network model provided in an embodiment of the present application. [Figure 12] FIG. 2 is a schematic diagram illustrating the simplification of a surface of a three-dimensional mesh provided in an embodiment of the present application. [Figure 13] FIG. 2 is a schematic diagram illustrating face densification of a three-dimensional mesh provided in an embodiment of the present application. [Figure 14] 1 is a flowchart of a keypoint detection method provided in an embodiment of the present application; [Figure 15] FIG. 1 is a schematic diagram illustrating a graph convolutional neural network configuration provided in an embodiment of the present application. [Figure 16]FIG. 1 is a comparison diagram of geodesic distance and Euclidean distance provided in an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0019] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. The described embodiments do not limit the present application, and all other embodiments that can be obtained by those skilled in the art without any creative efforts are within the protection scope of the present application.

[0020] Below, references to "some embodiments" describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without contradiction.

[0021] The terms "first / second / third" and the like used herein do not limit a particular order but distinguish between similar objects. It is to be understood that "first / second / third" can be used to interchange a particular order or order when appropriate, and that the embodiments of the present application described herein can be performed in an order other than that shown or described herein.

[0022] Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art. The terms used herein are used only to describe the embodiments of the present application and are not intended to limit the present application.

[0023] Before describing the embodiments of the present application in detail, the nouns and terms used in the embodiments of the present application will be explained. The explanation of the nouns and terms used in the embodiments of the present application will be as follows.

[0024] 1) 3D Mesh: A 3D mesh refers to a manifold surface with a topological structure, such as a sphere divided into a combination of multiple vertices and multiple edges. In this application, it may be a 3D face mesh, where the 3D mesh is a graph structure.

[0025] 2) Client: A client, also known as the user side, refers to a program that provides local services to users in response to a server. Except for some applications that only run locally, it is generally installed on a regular client and must run in cooperation with a server. That is, the network requires a corresponding server and service program to provide the corresponding service, and therefore, the client and server sides must establish a specific communication connection to ensure the normal operation of the application.

[0026] 3) 3D facial keypoint detection: 3D facial keypoint detection refers to detecting the 3D coordinates of a set of facial keypoints with a given semantic given a 3D facial mesh model. There are no restrictions on the number of vertices or faces in a 3D facial model. Keypoints with a given semantic include the corners of the eyes, the corners of the mouth, the tip of the nose, the contours of the face, etc. The semantics and number of keypoints are determined by the specific task.

[0027] 4) Graph Neural Networks (GNNs): GNNs are a type of artificial neural network used to process data that can be represented as graphs. Compared to traditional 2D convolutional neural networks that operate on 2D pictures, graph neural networks extend their scope to graph data that represent 3D mesh forms. A key design element of graph neural networks is the use of pairwise message passing, allowing graph nodes to be iteratively updated by exchanging information with neighboring nodes.

[0028] 5) Loss: Loss is used to measure the gap between the actual and target results of a model in order to train and optimize the model.

[0029] 6) 3D heatmap regression (Heatmap): 3D heatmap regression refers to a graph neural network that uses a heatmap as the output layer, forms a regression loss with a standard heatmap, trains the neural network through forward propagation and gradient propagation, matches the output of the neural network with the labels, and finally calculates the coordinates of the keypoints from the heatmap.

[0030] 7) 3D scanner: A 3D scanner is a scientific instrument that detects and analyzes the shape (geometry) and appearance data (e.g., color, surface reflectance, and other properties) of real-world objects and environments. The collected data is typically used in 3D reconstruction calculations to create digital models of real objects in a virtual world. These models have a wide range of applications, including industrial design, defect detection, reverse engineering, robotic guidance, topographical surveying, medical informatics, bioinformatics, and timekeeping authentication.

[0031] 8) Multi-Layer Perceptron (MLP): A multi-layer perceptron is an artificial neural network with a forward-directed structure that maps a set of input vectors to a set of output vectors. An MLP can be viewed as a directed graph consisting of multiple layers of nodes, each fully connected to the next layer. In addition to input nodes, each node is a neuron (also called a processing unit) with a nonlinear activation function.

[0032] 9) Convolutional Neural Network (CNN): A CNN is a feedforward neural network that typically consists of one or more convolutional layers (network layers that use convolutional mathematical operations) and a terminal fully connected layer. Neurons within the network can respond to certain regions of the input image, and generally perform better in the field of visual image processing.

[0033] 10) Machine Learning (ML) i Machine learning (ML) is a multidisciplinary field related to various fields such as probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. ML focuses on the study of how computers can simulate or realize human learning behavior to acquire new knowledge and skills and reorganize existing knowledge structures to continuously improve their performance. ML is the core of artificial intelligence and a fundamental means of endowing computers with intelligence, and is applied to various fields of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, augmented learning, transfer learning, inductive learning, and learning from demonstration.

[0034] 11) Point cloud data: Point cloud data refers to a large collection of points on the target surface features, and can generally be obtained by laser measurement or photography. Point cloud data obtained by laser measurement includes three-dimensional coordinates and laser reflection intensity. This type of point cloud data usually determines the state of an object based on echo characteristics and reflection intensity. Point cloud data obtained by photography usually includes three-dimensional coordinates and color information.

[0035] 12) Graph Attention Network (GAT) (also known as Graph Attention Network): GAT is a new neural network architecture based on graph structure data.

[0036] With the research and progress of artificial intelligence technology, it has been researched and applied in various fields, such as general smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, and smart customer service. With the development of technology, it is believed that artificial intelligence technology will be applied in more fields and will play an increasingly important role.

[0037] The solutions provided in the embodiments of the present application relate to technologies such as three-dimensional network models of artificial intelligence, and can also be applied to fields such as cloud technology and the Internet of Vehicles, and are specifically described in the following embodiments.

[0038] Referring to FIG. 1, FIG. 1 is a schematic diagram illustrating the architecture of a keypoint detection system 100 provided in an embodiment of the present application. To realize an application scenario of keypoint detection (for example, when performing keypoint detection on a face, the face may first be three-dimensionally scanned with a three-dimensional scanner and then the positions of facial keypoints may be detected based on the three-dimensional scanned data), a terminal (terminal 400 is shown as an example) is connected to a server 200 via a network 300. The network 300 may be a wide area network, a local area network, or a combination of both. The terminal 400 is configured so that a user can use a client 401 displayed on a display interface (display interface 401-1 is shown as an example). The terminal 400 and the server 200 are connected via a wired or wireless network.

[0039] Here, the terminal 400 is configured to acquire three-dimensional scan data corresponding to the detection target object, and transmit the three-dimensional scan data to the server 200; the server 200 is configured to perform the steps of receiving the 3D scan data; obtaining a 3D mesh representing the detected object based on the 3D scan data, and determining the vertices of the 3D mesh and the connection relationships between the vertices; performing feature extraction on the vertices of the 3D mesh to obtain vertex features of the 3D mesh; performing global feature extraction on the detected object based on the vertex features to obtain global features of the detected object; performing local feature extraction on the detected object based on the vertex features and the connection relationships between the vertices to obtain local features of the detected object; detecting key points of the detected object based on the vertex features, the global features, and the local features to obtain positions of the key points of the detected object on the detected object; and transmitting the positions of the key points on the detected object to the terminal 400; The terminal 400 is further configured to display the positions of the key points on the detected object based on the display interface.

[0040] In some embodiments, server 200 may be an independent physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), big data, and artificial intelligence platforms. Terminal 400 may be, but is not limited to, a smartphone, tablet, laptop, desktop computer, set-top box, intelligent voice interaction device, smart home appliance, in-vehicle terminal, aircraft, mobile device (e.g., mobile phone, portable music player, personal digital assistant, dedicated messaging device, portable game device), etc. The terminal device and the server may be directly or indirectly connected via wired or wireless communication, and the embodiments of the present application are not limited thereto.

[0041] Referring to FIG. 2, FIG. 2 is a schematic diagram showing the configuration of an electronic device provided in an embodiment of the present application. In practical application, the electronic device may be the server 200 or the terminal 400 shown in FIG. 1. Referring to FIG. 2, the electronic device shown in FIG. 2 includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. Each component in the terminal 400 is spliced ​​through a bus system 440. It is understood that the bus system 440 realizes connection communication between these components. In addition to a data bus, the bus system 440 includes a power bus, a control bus, and a status signal bus. However, for clarity of explanation, various buses are denoted as the bus system 440 in FIG. 2.

[0042] The processor 410 may be an integrated circuit chip with signal processing capabilities, such as a general purpose processor, a digital signal processor (DSP), a programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc., where the general purpose processor may be a microprocessor or any conventional processor, etc.

[0043] The user interface 430 includes one or more output devices 431 that enable media content to be presented, including one or more speakers and / or one or more visual displays. The user interface 430 further includes one or more input devices 432 that facilitate user input, such as a keyboard, mouse, microphone, touchscreen display, camera, or other input buttons and controls.

[0044] Memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. Memory 450 includes one or more storage devices located physically separate from processor 410.

[0045] The memory 450 may include volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The nonvolatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in the embodiments of the present application may include any suitable type of memory.

[0046] In some embodiments, memory 450 may store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as illustrated below.

[0047] The operating system 451 includes system programs, such as a frame layer, a core library layer, a drive layer, etc., for processing various basic system services and executing hardware-related tasks, and is configured to realize various basic services and process hardware-based tasks.

[0048] The network communication module 452 is configured to reach other computing devices via one or more (wired or wireless) network interfaces 420, example network interfaces 420 including Bluetooth, Wireless Firmware (Wi-Fi), and Universal Serial Bus (USB), etc.

[0049] The display module 453 is configured to enable the display of information via one or more output devices 431 (e.g., display, speakers) associated with the user interface 430 (e.g., configured to operate peripheral devices and the user interface to display content and information).

[0050] The input processing module 454 is configured to detect one or more user inputs or interactions from the input device(s) 432 and to translate the detected inputs or interactions.

[0051] In some embodiments, the apparatus provided in the embodiments of the present application may be realized in software. Figure 2 shows an apparatus for detecting keypoints in a virtual scene 455 stored in memory 450, which may be software in the form of a program or plug-in, and includes the following software modules: an acquisition module 4551, a first feature extraction module 4552, a second feature extraction module 4553, and an output module 4554. These modules are logical and can be arbitrarily combined or further divided according to the functions to be realized. The functions of each module will be described later.

[0052] In some other embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. For example, the keypoint detection apparatus provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the keypoint detection method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may employ one or more Application Specific Integrated Circuits (ASICs), DSPs, Digital Signal Processors (PLDs), Complex Programmable Logic Devices (CPLDs), Field-Programmable Gate Arrays (FPGAs) or other electronic elements.

[0053] In some embodiments, a terminal or a server can implement the keypoint detection method provided in the embodiments of the present application by executing a computer program. For example, the computer program may be a native program or software module in an operating system, a native application (APP) that needs to be installed in an operating system to run, an applet that can be run simply by downloading it to a browser environment, or an applet that can be embedded in any APP. In short, the computer program may be any type of application, module, or plug-in.

[0054] Based on the above description of the keypoint detection system and electronic device provided in the embodiment of the present application, the following describes the keypoint detection method provided in the embodiment of the present application. In actual implementation, the keypoint detection method provided in the embodiment of the present application can be performed independently by a terminal or a server, or in cooperation with a terminal and a server. The keypoint detection method provided in the embodiment of the present application will be described below using the case where it is performed independently by the server 200 of FIG. 1. Referring to FIG. 3, FIG. 3 is a flowchart of the keypoint detection method provided in the embodiment of the present application. The following description will be made with reference to the steps shown in FIG. 3.

[0055] In step 101, the server obtains a 3D mesh to represent the detected object, and determines the vertices of the 3D mesh and the connection relationships between the vertices.

[0056] In actual implementation, obtaining a 3D mesh representing the detected object may be achieved by directly receiving a 3D mesh of the detected object transmitted from another device, or by using point cloud data (i.e., 3D scan data) corresponding to the detected object. Here, point cloud data is configured as a collection of a large number of points representing the surface features of the detected object and can generally be obtained by laser measurement or photographic measurement. Specifically, first, point cloud data corresponding to the detected object is obtained, and then a 3D mesh representing the detected object is obtained based on the point cloud data. That is, a 3D mesh corresponding to the detected object is constructed. Here, there are various methods for obtaining the point cloud data corresponding to the detected object. The point cloud data may be pre-stored locally on the terminal, obtained from an external source (e.g., the Internet), or collected in real time, for example, by a 3D scanning device such as a 3D scanner.

[0057] In some embodiments, when point cloud data is collected in real time by a three-dimensional scanning device such as a three-dimensional scanner, the process of constructing a three-dimensional mesh corresponding to the detected object specifically includes: scanning the detected object with a three-dimensional scanning device to obtain point cloud data of the geometric surface of the detected object; and constructing a three-dimensional mesh corresponding to the detected object based on the point cloud data. For example, referring to FIG. 4, FIG. 4 is a schematic diagram of a three-dimensional mesh of a human head provided in an embodiment of the present application. Based on FIG. 4, when the detected object is a face, a three-dimensional scan is performed on the human head with a three-dimensional scanner to obtain point cloud data corresponding to the head, and a three-dimensional mesh corresponding to the head is constructed based on the point cloud data.

[0058] It should be noted that the process of constructing a 3D mesh corresponding to the detected object based on point cloud data can be as follows: First, the point cloud data is preprocessed to obtain target point cloud data, where preprocessing includes operations such as filtering, noise removal, and point cloud registration, where filtering can remove noise points, noise removal can further reduce noise points and invalid points, and point cloud registration can align the point cloud data to the same coordinate system. Next, the target point cloud data is subjected to mesh reconstruction to obtain a 3D mesh, where mesh reconstruction is a process of converting discrete target point cloud data into a 3D mesh. Commonly used mesh reconstruction algorithms include mesh-based methods, voxel-based methods, implicit function-based methods, etc. Here, the mesh-based method converts the target point cloud data into a triangular mesh, the voxel-based method converts the target point cloud data into a voxel mesh, and the implicit function-based method represents the 3D mesh using a data function.

[0059] It should be noted that embodiments of the present application involve related data, such as real-time scanning, and where the embodiments of the present application are applicable to a particular product or technology, user permission or consent is required, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0060] In practical implementation, the connection relationships between vertices of a 3D mesh may be expressed as a connection relationship matrix between vertices to indicate whether or not the vertices are related to each other. The size is N×N, and the value is either 0 or 1. Here, N is the number of vertices. If vertex i is connected to vertex j, then the connection relationship A between the two vertices is ij is 1, otherwise it is 0.

[0061] For example, there is a connectivity relationship between the vertices of a three-dimensional mesh indicating the position of the eyes on the face, but there is no connectivity relationship between the vertices of a three-dimensional mesh indicating the position of the eyes and the vertices of a three-dimensional mesh indicating the position of the chin.

[0062] In step 102, feature extraction is performed on the vertices of the 3D mesh to obtain vertex features of the 3D mesh.

[0063] In practical implementation, feature extraction is performed on the vertices of the 3D mesh to obtain vertex features of the 3D mesh, where the vertex features include information on the positions of the corresponding vertices and the corresponding positions on the face indicated by the corresponding vertices. For example, the vertex features may be N×(6+X), where N represents the number of vertices corresponding to the 3D mesh, 6 represents the dimensions occupied by the vertex coordinates and normal vectors, i.e., six directional dimensions corresponding to the three coordinate dimensions of the vertex coordinates (x, y, z), and X includes other characteristics of the vertices of the 3D mesh, i.e., information on the corresponding positions on the face indicated by the corresponding vertices, such as curvature and texture information. It should be noted that these other characteristics can be adjusted according to the data and task. Thus, when the present application is applied to a model, adding these other characteristics during the model training phase can improve the learning efficiency of the model.

[0064] In step 103, global feature extraction is performed on the object to be detected based on the vertex features to obtain global features of the object to be detected, and local feature extraction is performed on the object to be detected based on the vertex features and the connection relationships between the vertices to obtain local features of the object to be detected.

[0065] It should be noted that after determining the connection relationship between the vertices of the 3D mesh and the vertex features of the 3D mesh, global feature extraction and local feature extraction are performed on the detected object, respectively, to obtain the global features and local features of the detected object.

[0066] In some embodiments, the process of performing global feature extraction on the detection object based on vertex features to obtain the global features of the detection object may be as follows: first, feature extraction is performed on the detection object based on vertex features, and then max-pooling processing is performed on the extracted features to obtain max-pooling features, so that all vertices share the max-pooling features, and the max-pooling features are used as the global features of the detection object.

[0067] In some embodiments, the process of performing local feature extraction on the detected object based on vertex features and the connectivity relationships between the vertices to obtain local features of the detected object may include a step of determining local features of each vertex based on the vertex features and the connectivity relationships between the vertices, and a step of determining local features of the detected object based on the local features of each vertex.

[0068] It should be noted that global features are used to represent the overall features of the object to be detected, such as the color, texture, and shape features of the object to be detected, while local features are used to represent the detailed features of the object to be detected, i.e., features extracted from local regions of the object to be detected, such as features extracted from edges, corners, points, lines, curves, and special attribute regions of the object to be detected. For example, if the object to be detected is a face, global features may be the size, shape, and position of the five senses of the face, while local features may be the distribution of facial muscles under different expressions, changes in the shape of the five senses, etc. Because global features are pixel-level low-level visible features, they have good invariance, simple calculation, and intuitive representation, but are not applicable when there is object overlap and occlusion. Meanwhile, local image features are characterized by being abundant in an image and having low correlation between features, and even if some features disappear when there is object overlap and occlusion, the detection and matching of other features is not affected. In this way, by extracting global features and local features from the detection object, it is possible to obtain richer and more accurate feature amounts for the detection object, thereby improving the accuracy of keypoint detection results.

[0069] Next, a process for determining the local features of each vertex based on the vertex features and the connection relationships between the vertices, and a process for determining the local features of the detection object based on the local features of each vertex will be described.

[0070] The process of determining the local features of each vertex based on the vertex features and the connection relationships between the vertices will now be described. Referring to Fig. 5, Fig. 5 is a flowchart of determining the local features of each vertex provided in an embodiment of the present application. Based on Fig. 5, the process of determining the local features of each vertex based on the vertex features and the connection relationships between the vertices is realized by steps 1031 to 1033. Referring to Fig. 5, the following processing is performed for each vertex:

[0071] In step 1031, a vertex is determined as a reference vertex, and based on the vertex characteristics of each vertex of the three-dimensional mesh, the vertex characteristics of the reference vertex and the vertex characteristics of other vertices are determined, where the other vertices are any vertices other than the reference vertex.

[0072] For example, if the number of vertices in a three-dimensional network is N, the characteristics of each vertex are h, and the dimension is F, then the following is obtained.

[0073]

number

[0074] where vertex i is the reference node and h i is a vector of size F, i.e., the features of reference node i, vertex j is another vertex, and h j is a vector of size F, i.e., the features of other node j, and there exists an edge connection relationship between vertex i and vertex j.

[0075] In step 1032, a correlation value between the reference vertex and the other vertices is determined based on the vertex features of the reference vertex, the vertex features of the other vertices, and the connection relationships between the vertices, and the correlation value is intended to indicate the magnitude of the correlation between the reference vertex and the other vertices.

[0076] In some embodiments, the process of determining the correlation value between the reference vertex and other vertices based on the vertex features of the reference vertex, the vertex features of the other vertices, and the connection relationships between the vertices may include employing an attention mechanism to determine the correlation degree between the reference vertex and other vertices based on the vertex features of the reference vertex, the vertex features of the other vertices, and the connection relationships between the vertices, where the correlation degree is an index for measuring the strength of the association between the reference vertex and other vertices, and the magnitude of the correlation degree can be calculated by the following formula:

[0077]

number

[0078] where W is a weighting matrix of size F × F, and h i is the vertex feature of reference vertex i, and h j is the vertex feature of another vertex j, attention indicates that attention mechanism is adopted for processing, and e ij indicates the degree of correlation between the reference vertex and other vertices.

[0079] In some other embodiments, the process of determining a correlation value between a reference vertex and other vertices based on the vertex features of the reference vertex, the vertex features of the other vertices, and the connection relationships between the vertices may include the steps of determining a connected reference vertex and other vertices based on the connection relationships between the vertices; performing similarity matching for the reference vertex and the corresponding other vertices based on the vertex features of the connected reference vertex and the vertex features of the other vertices to obtain a similarity between the reference vertex and the corresponding other vertices (where a similarity corresponding to each of the other vertices is obtained); and determining this similarity as a correlation between the reference vertex and the corresponding other vertex.

[0080] Next, the correlation degree is normalized to obtain a correlation value between the reference vertex and the other vertices.

[0081]

number

[0082] Here, Softmax j indicates the adoption of normalization processing, and α ij denotes the correlation value between nodes i and j, exp denotes an exponential function with the natural constant e as the base, and N i denotes the domain consisting of all other nodes that are connected to the reference node i, and q denotes any vertex in said domain.

[0083] For example, refer to FIG. 6, which is a schematic diagram of determining the correlation degree between a reference vertex and other vertices by adopting the attention mechanism provided in the embodiment of the present application. Based on FIG. 6, α ij indicates the correlation value between nodes i and j, and Whi indicates the vertex feature corresponding to the reference vertex i, and Wh j denotes the vertex feature corresponding to another vertex j, a is a weight vector, and Wh i and Wh j After determining the correlation between the reference vertex and other vertices based on j Processing, i.e., normalization processing, is performed to obtain correlation values ​​between the reference vertex and other vertices.

[0084] It should be noted that the process of adopting the attention mechanism to determine the correlation degree between the reference vertex and other vertices can be specifically as follows: i ,Wh j Then, the feature obtained by the splicing is calculated with the weight vector a, whose dimension is 2F, and the correlation value between the reference vertex and other vertices is obtained using the activation function.

[0085]

number

[0086] where N i denotes a domain consisting of all other nodes connected to the reference node i, and q denotes any vertex in the domain;

number

[0087] It should be noted that, regarding the method for determining the correlation degree, it is also possible to directly calculate the correlation degree between the reference vertex and other vertices based on the vertex features of the reference vertex, the vertex features of the other vertices, and the connection relationships between the vertices. Here, there are various methods for calculating the correlation degree, such as Pearson correlation coefficient, Spearman's rank correlation coefficient, etc.

[0088] In step 1033, local features of the reference vertex are determined based on the correlation values ​​and the vertex features of the other vertices.

[0089] In actual implementation, after obtaining the correlation value, if the number of other vertices is one, the process of determining the local feature of the reference vertex based on the correlation value and the vertex features of the other vertices may include: multiplying the correlation value by the vertex features of the other vertices to obtain the multiplication result; and determining the local feature corresponding to the reference vertex based on the multiplication result.

[0090]

number

[0091] where σ is the activation function and α ij is the correlation value between the reference vertex i and another vertex j, and Wh j denotes the vertex feature corresponding to another vertex j,

number

[0092] It should be noted that when the number of other vertices is multiple, the process of determining the local features of the reference vertex based on the correlation value and the vertex features of the other vertices may include: for each other vertex, multiplying the correlation value by the vertex features of the corresponding other vertex to obtain a multiplication result of the other vertex; performing cumulative addition on the multiplication results of each other vertex to obtain an addition result; and determining the local features corresponding to the reference vertex based on the addition result.

[0093]

number

[0094] where σ is the activation function and α ij is the correlation value between the reference vertex i and another vertex j, and Wh j denotes the vertex feature corresponding to another vertex j, and N i denotes the domain consisting of all other nodes that have a connection relationship with the reference node i.

[0095] Regarding the process of determining the local features of the object to be detected based on the local features of each vertex, specifically, feature fusion is performed on the local features of each vertex based on the local features of each vertex to obtain fusion features, and the fusion features are used as the local features of the object to be detected.

[0096] In step 104, the key points of the object are detected based on the vertex features, the global features, and the local features, and the positions of the key points of the object on the object are obtained.

[0097] In some embodiments, the process of detecting keypoints of the object based on vertex features, global features, and local features to obtain the positions of the keypoints on the object may include: performing feature splicing on the vertex features, global features, and local features to obtain splice features of the object; and performing detection on keypoints of the object based on the splice features to obtain the positions of the keypoints on the object. Since the splice features include feature information of the vertex features, global features, and local features of the object, detecting keypoints of the object based on the splice features combines the feature information of the vertex features, global features, and local features. This means that the detection of keypoints of the object can be performed with richer feature information, improving the accuracy of keypoint detection results.

[0098] It should be noted that in the related art method based on 3D coordinate regression, it is difficult to accurately define a keypoint based on its pixel location because points surrounding the keypoint may also be similar to the keypoint. On the other hand, the 3D heat map in this application is a statistical diagram that displays multiple data by coloring them into color blocks, i.e., each data is represented according to a specified color mapping rule. For example, large values ​​are represented by dark colors and small values ​​by light colors, or large values ​​are represented by warm colors and small values ​​by cool colors. In this way, by outputting the 3D heat map and displaying the possibility of a keypoint belonging to each vertex, the local accuracy of the detection results can be further ensured.

[0099] 7, which is a schematic diagram illustrating the positions of key points on a detection object provided in an embodiment of the present application. Based on FIG. 7, the black dots in FIG. 7 are key points. When the detection object is a face, the positions of the key points illustrated in FIG. 7 may be the positions of the five senses on the face. Here, the black dot in the dotted frame 701 is a key point indicating the position of the forehead on the face, the black dots in the dotted frame 702 and 703 are key points indicating the positions of the eyes on the face, the black dots indicated by 704 and 705 are key points indicating the positions of the ears on the face, the black dot in the dotted frame 706 is a key point indicating the position of the nose on the face, the black dot in the dotted frame 707 is a key point indicating the position of the mouth on the face, the black dots indicated by 708 and 709 are key points indicating the positions of the cheeks on the face, and the black dot in the dotted frame 710 is a key point indicating the position of the chin on the face. Here, through the output layer, detection is performed on the positions of the five senses of the object to be detected based on the vertex features, global features, and local features, and the probability that a keypoint exists at each vertex in the three-dimensional mesh, i.e., the probability that each vertex of the three-dimensional mesh is a keypoint corresponding to each of the positions of the five senses, is obtained, and a three-dimensional heat map corresponding to the three-dimensional mesh is generated based on each probability.Furthermore, based on the three-dimensional heat map, the positions of the keypoints of the object to be detected on the object to be detected, i.e., the keypoints corresponding to each of the positions of the five senses, are determined, and the vertex with the largest probability is selected from the multiple probabilities and determined as the corresponding keypoint, and the positions of the five senses are thereby determined based on the keypoints.

[0100] In some embodiments, the keypoint detection method herein may be further applied to a three-dimensional network model. The three-dimensional network model includes at least a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, and an output layer. Referring to FIG. 8, FIG. 8 is a schematic diagram illustrating the configuration of a three-dimensional network model provided in an embodiment of the present application. Based on FIG. 8, the process of performing feature extraction on the vertices of the three-dimensional mesh to obtain vertex features of the three-dimensional mesh may include performing feature extraction on the vertices of the three-dimensional mesh via the first feature extraction layer to obtain vertex features of the three-dimensional mesh; performing global feature extraction on the detected object based on the vertex features to obtain global features of the detected object; performing local feature extraction on the detected object based on the vertex features and the connection relationships between the vertices; and performing global feature extraction on the detected object based on the vertex features via the second feature extraction layer to obtain local features of the detected object. The process of detecting key points of the detected object based on the vertex features, global features, and local features and obtaining the positions of the key points of the detected object on the detected object may include a step of combining the vertex features, global features, and local features through an output layer to detect key points of the detected object and obtain the positions of the key points of the detected object on the detected object.

[0101] In this way, by detecting the positions of key points on the detection object using a three-dimensional network model, the accuracy of the detected positions is improved.

[0102] In some embodiments, the third feature extraction layer herein may include at least two third feature extraction sub-layers and a feature splicing sub-layer. For example, refer to FIG. 9, which is a schematic diagram illustrating the configuration of the third feature extraction layer provided in an embodiment of the present application. Based on FIG. 9, the process of determining a local feature of each vertex based on vertex features and inter-vertex connection relationships via the third feature extraction layer may include the following steps for each vertex via each third feature extraction sub-layer: determining the vertex as a reference vertex, and determining the vertex features of the reference vertex and vertex features of other vertices based on the vertex features of each vertex of the three-dimensional mesh; determining a correlation value between the reference vertex and other vertices based on the vertex features of the reference vertex, the vertex features of the other vertices, and the inter-vertex connection relationships; determining a local sub-feature of the reference vertex based on the correlation value and the vertex features of the other vertices; and splicing the local sub-features obtained via each third feature extraction sub-layer to obtain a local feature of the reference vertex via the feature splicing sub-layer.

[0103]

number

[0104] where k is the number of layers in the third feature extraction sublayer, and N i denotes the domain consisting of all other nodes connected to the reference node i, σ is the activation function, and α ij is the correlation value between the reference vertex i and another vertex j, and Wh j indicates the vertex feature corresponding to another vertex j, and concat indicates the adoption of splicing processing.

[0105] It should be noted that here, the process of determining the correlation value between the reference vertex and the other vertices based on the vertex features of the reference vertex, the vertex features of the other vertices, and the connection relationships between the vertices is the same as the above-mentioned process, and at the same time, the process of determining the local sub-features of the reference vertex based on the correlation value and the vertex features of the other vertices is also the same as the process of determining the local features of the reference vertex based on the above-mentioned correlation value and the vertex features of the other vertices, which will not be described again.

[0106] In some embodiments, the three-dimensional network model further comprises a first feature splice layer, a second feature splice layer, and a fourth feature extraction layer. For example, refer to FIG. 10, which is a schematic diagram showing the configuration of a three-dimensional network model provided in an embodiment of the present application. Based on FIG. 10 , the process of combining the vertex features, global features, and local features to perform detection on key points of the detected object through the output layer and obtaining the location of the key points on the detected object may include: performing feature splicing on the vertex features, global features, and local features through a first feature splicing layer to obtain spliced ​​features of the detected object; performing local feature extraction on the detected object based on the spliced ​​features through a fourth feature extraction layer to obtain target local features of the detected object; performing feature splicing on the splice features, global features, and target local features through a second feature splicing layer to obtain target spliced ​​features of the detected object; and performing detection on key points of the detected object based on the target spliced ​​features through the output layer to obtain the location of the key points of the detected object on the detected object.

[0107] It should be noted that the three-dimensional network model includes a fifth feature extraction layer and a third feature splicing layer, and the fifth feature extraction layer performs local feature extraction on the detected object based on the target splicing feature to obtain a second target local feature, and then the third feature splicing layer performs feature splicing on the target splicing feature, the second target local feature, and the global feature to obtain a second target splicing feature, and finally the output layer performs detection on the key points of the detected object based on the second target splicing feature to obtain the location of the key points of the detected object on the detected object. Here, the process of determining the local features of the detected object and the corresponding splicing features in the three-dimensional network model can have multiple feature extraction layers and feature splicing layers, and the process of obtaining the final splicing feature through multiple feature extraction layers and feature splicing layers is the same as the above process, and the embodiments of the present application will not repeat the description thereof.

[0108] It should be noted that the fourth feature extraction layer and the fifth feature extraction layer have the same layer structure and feature processing process as the third feature extraction layer. Meanwhile, the second feature splice layer and the third feature splice layer have the same layer structure and feature processing process as the first feature splice layer. The fourth feature extraction layer performs further feature processing on the spliced ​​feature to obtain a more accurate target local feature. At the same time, the second feature splice layer performs feature splicing on the spliced ​​feature, global features, and the obtained target local feature, and detects key points of the object to be detected based on the target splice feature obtained by feature splicing. Correspondingly, the fifth feature extraction layer performs further feature processing on the target splice feature to obtain a more accurate second target local feature. At the same time, the third feature splice layer performs feature splicing on the target splice feature, global features, and the obtained second target local feature, and detects key points of the object to be detected based on the second target splice feature obtained by feature splicing.

[0109] In this way, by setting up a feature extraction layer with the same structure and a feature splice layer with the same structure, and repeatedly performing local feature extraction and corresponding feature splicing on the detection target multiple times, the accuracy of the extracted features is improved, and the accuracy of the keypoint detection results is also improved.

[0110] In some embodiments, before detecting the key points of the object to be detected based on the three-dimensional network model, the three-dimensional network model needs to be trained, and then detecting the key points of the object to be detected based on the trained three-dimensional network model. Specifically, referring to Figure 11, Figure 11 is a flowchart of the training process of the three-dimensional network model provided in the embodiments of the present application. Based on Figure 11, the training process of the three-dimensional network model can be realized by the following steps:

[0111] In step 201, an object training sample with server labels is obtained, where the labels are to indicate the actual locations of the keypoints of the object training sample.

[0112] In step 202, a training 3D mesh for representing an object training sample is obtained, and the vertices and the connectivity between the vertices of the training 3D mesh are determined.

[0113] It should be noted that after obtaining the training 3D mesh for representing the object training sample, the training 3D mesh can also be data augmented, so that the augmented training 3D mesh can be used to train the 3D network model. Specifically, the method of data augmenting the training 3D mesh can be divided into surface simplification and densification.

[0114] In some embodiments, when performing face simplification on the training 3D mesh, an edge optimization method can be used, i.e., searching for the smallest edge between each vertex and merging the two corresponding vertices into one vertex. Specifically, the edges between any two vertices are obtained, compared, and based on the comparison result, the smallest edge from each edge is selected as the target edge. Then, the two vertices corresponding to the target edge are obtained and merged into one vertex, thereby obtaining an expanded training 3D mesh. For example, refer to FIG. 12, which is a schematic diagram illustrating the face simplification of a 3D mesh provided in an embodiment of the present application. Based on Figure 12, there are ten vertices, v1 to v10, and ten edges, v1v2, v1v3, v1v4, v1v10, v1v9, v1v2, v1v8, v5v2, v7v2, and v6v2, are formed based on these ten vertices. The edge between v1 and v2 is the smallest edge, so these two vertices can then be merged into one vertex v to obtain an expanded 3D training mesh.

[0115] In some other embodiments, when face densification is performed on a training 3D mesh, the barycentric coordinates of faces with larger areas are calculated first, and then the original triangular face is divided into three based on the barycentric coordinates. Specifically, at least one face is obtained, and then the faces are compared. Based on the comparison result, the face with the largest area is selected from the multiple faces as the target face, and the barycentric coordinates of the target face and three vertices corresponding to the target face are determined. The original triangular face is divided into three based on the barycentric coordinates and three vertices. For example, refer to FIG. 13, which is a schematic diagram illustrating face densification of a 3D mesh provided in an embodiment of the present application. Based on FIG. 13, there are nine vertices, A to I, and based on these nine vertices, eight triangular faces are formed, namely, the face between vertices A, B, and C, the face between vertices A, B, and I, the face between vertices H, B, and I, the face between vertices H, B, and G, the face between vertices F, B, and G, the face between vertices F, B, and E, the face between vertices D, B, and E, and the face between vertices D, B, and C. Here, the face between vertices A, B, and C is the target face with the largest area, and P, the center of gravity of the target face, and the corresponding vertices A, B, and C are determined. The original target face is divided into three parts based on P, A, B, and C, and an expanded 3D training mesh is obtained.

[0116] It should be noted that the data augmentation process for the training 3D mesh can be terminated by presetting a target number of vertices. Specifically, in the data augmentation process for the training 3D mesh, the number of vertices of the augmented training 3D mesh is obtained, and this number of vertices is compared with a predefined target number of vertices. Based on the comparison result, data augmentation for the training 3D mesh is terminated. When face simplification is performed on the training 3D mesh, if the comparison result indicates that the number of vertices is smaller than the target number of vertices, data augmentation for the training 3D mesh is terminated. When face densification is performed on the training 3D mesh, if the comparison result indicates that the number of vertices is greater than the target number of vertices, data augmentation for the training 3D mesh is terminated.

[0117] In step 203, feature extraction is performed on the vertices of the object training sample through the first feature extraction layer to obtain the vertex features of the training 3D mesh.

[0118] In step 204, through the second feature extraction layer, global feature extraction is performed on the object training sample based on the vertex features of the training 3D mesh to obtain global features of the object training sample; and through the third feature extraction layer, local feature extraction is performed on the object training sample based on the vertices of the training 3D mesh and the connection relationships between the vertices to obtain local features of the object training sample.

[0119] In step 205, detection is performed on the keypoints of the object training sample based on the vertex features of the training 3D mesh, the global features of the object training sample, and the local features of the object training sample via the output layer, and the positions of the keypoints of the object training sample on the object training sample are determined.

[0120] In actual implementation, the three-dimensional network model further includes a first feature splice layer, and the process of performing detection on key points of the object training sample based on the vertex features of the training three-dimensional mesh, the global features of the object training sample, and the local features of the object training sample via the output layer, and determining the positions of the key points of the object training sample on the object training sample may include: performing feature splicing on the vertex features of the training three-dimensional mesh, the global features of the object training sample, and the local features of the object training sample via the first feature splice layer, and obtaining spliced ​​features of the object training sample; and performing detection on key points of the object training sample based on the spliced ​​features of the object training sample via the output layer, and determining the positions of the key points of the object training sample on the object training sample.

[0121] In step 206, the difference between the keypoint positions and labels of the object training sample is obtained, and a 3D network model is trained based on the difference to obtain a target 3D network model, which is configured to perform keypoint detection on the detected object and obtain the positions of the keypoints of the detected object on the detected object.

[0122] Continuing, the keypoint detection method provided in the embodiment of the present application will be described below. Referring to Fig. 14, Fig. 14 is a flowchart of the keypoint detection method provided in the embodiment of the present application. Based on Fig. 14, the keypoint detection method provided in the embodiment of the present application is implemented by a client and a server in cooperation with each other.

[0123] In step 301, the client obtains labeled object training samples in response to an upload operation of labeled object training samples.

[0124] In actual implementation, the client may be a keypoint detection client located in a terminal, and a user triggers an upload function option of the human-computer interaction interface of the client based on the human-computer interaction interface, causing the client to display an object selection interface on the human-computer interaction interface, and the user uploads object training samples with labels from the local terminal based on the object selection interface, causing the client to obtain the uploaded object training samples.

[0125] In some embodiments, the object training samples may also be obtained by photographing with a camera communicatively connected to the terminal, where the camera photographs and obtains the object training samples, labels the object training samples, and transmits the labeled object training samples to the terminal, where they are automatically uploaded to the client by the terminal.

[0126] In step 302, the client sends object training samples to the server.

[0127] In step 303, the server inputs the received object training samples into the 3D network model.

[0128] In step 304, detection is performed on the keypoints of the object training samples based on the 3D network model, and the positions of the keypoints of the object training samples are obtained.

[0129] In step 305, the differences between the positions and labels of the keypoints of the object training samples are obtained, and a 3D network model is trained based on the differences.

[0130] In actual implementation, the server completes the training of the 3D network model by repeating the above training process until the loss function converges.

[0131] In step 306, the server generates a notification message indicating that the 3D network model training is complete.

[0132] In step 307, a suggestion message is sent to the client.

[0133] In step 308, the client acquires the point cloud data corresponding to the detected object in response to the upload operation of the point cloud data corresponding to the detected object.

[0134] In actual implementation, the point cloud data corresponding to the detected object may be pre-stored locally on the terminal, may be obtained from an external source (such as the Internet), or may be collected in real time, for example, by a three-dimensional scanning device such as a three-dimensional scanner.

[0135] In step 309, the client transmits point cloud data corresponding to the detected object to the server in response to the detection command for the key points of the detected object.

[0136] In actual implementation, the detection command for the key points of the detection object may be automatically generated by the client under certain trigger conditions, for example, after the client obtains point cloud data corresponding to the detection object, the detection command for the key points of the detection object may be automatically generated; it may be sent to the client from another device communicatively connected to the terminal; or it may be generated by the user by triggering a corresponding decision function option based on the human-computer interaction interface of the client.

[0137] In step 310, the server inputs the received point cloud data corresponding to the detected object into the three-dimensional network model to enable the three-dimensional network model to perform keypoint detection on the detected object, and obtains a three-dimensional heat map to indicate the locations of the keypoints of the detected object on the detected object.

[0138] In step 311, a three-dimensional heat map is sent to the client to indicate the locations of the key points of the detected object on the detected object.

[0139] In step 312, the client displays a three-dimensional heat map to indicate the locations of the key points of the detected object on the detected object.

[0140] In actual implementation, the client may display the 3D heatmap on the client's human-computer interaction interface, store the 3D heatmap locally on the terminal, or transmit the 3D heatmap to other devices communicatively connected to the terminal.

[0141] By applying the above embodiment of the present application, a 3D mesh corresponding to the detected object is obtained, and then a dual-path feature extraction layer is constructed to extract global and local features of the detected object based on the vertex features and the connection relationships between the vertices obtained from the 3D mesh, respectively, and thereby obtain the locations of keypoints on the detected object based on the vertex features obtained from the 3D mesh and the extracted global and local features. In this way, more feature information of the detected object can be extracted through the multi-layer feature extraction layer, and then keypoints of the detected object based on the rich feature information can be detected, greatly improving the accuracy of 3D keypoint detection.

[0142] Next, an exemplary application example of the embodiment of the present application in a real application scenario will be described.

[0143] The inventors have discovered that keypoint detection for 3D facial characters can generally be divided into two major categories. The first category is based on traditional geometric analysis. It generally utilizes sharp edge detection, curvature calculation, dihedral angle calculation, normal vector calculation, and certain specific geometric rules to directly locate semantic keypoints on a 3D head model. For example, it can be assumed that the largest vertex in the z direction of a 3D coordinate system is the nose tip keypoint. A sharp edge below the nose tip can be detected and roughly located in the approximate regions of the left and right mouth keypoints through symmetry. The second category is based on deep learning. It essentially renders a 3D head model into a 2D image and uses a 2D convolutional neural network to extract features and detect corresponding keypoints. It is noteworthy that these methods can also be divided into different combinations depending on whether they utilize multi-view detection and whether they directly regress 3D keypoints. For example, a common combination method is to render only the front view of a 3D head model, record the rendering projection relationship, then detect the 2D keypoint coordinates in the 2D front view, and finally back-project them into 3D space based on the known projection relationship to obtain the final 3D keypoint coordinates. Another combination method is to render multiple views (front, side, etc.) and then input them into different branches of a neural network model, so that the neural network model can combine the features from both views to directly regress the 3D keypoint coordinates.

[0144] However, for the first type of method, traditional keypoint location methods based on geometric analysis rely heavily on manually set rules. For example, when detecting sharp edges, a single threshold must be specified. This empirical value makes it difficult to apply to head models with different morphologies, making this method less robust. For the second type of method, methods based on 2D convolutional neural networks have achieved great success in traditional 2D image keypoint detection tasks. However, directly applying 2D convolutional neural networks to 3D keypoint detection has various limitations and drawbacks. First, the number of obtainable 3D face models is far fewer than face pictures, meaning that the dataset is insufficient, making it difficult for neural networks to fulfill their role. Second, rendering a 3D face-head model into a 2D picture inevitably results in a loss of 3D geometric information. For example, in a frontal view, information about the occipital region is inevitably lacking. Therefore, when detecting occipital keypoints, such information is naturally insufficient. Third, when adopting a multi-view method to avoid the problem of insufficient information, features are extracted using a multi-branch network, and finally, a neural network is fused to regress to 3D coordinates. In this way, the neural network needs to learn the intrinsic relationship between different views, which may cause the problem of difficult convergence and increase the difficulty of learning.

[0145] Based on this, embodiments of the present application provide a keypoint detection method, device, electronic device, computer-readable storage medium, and computer program product that can effectively solve various drawbacks of the above technical methods. Specifically, first, the 3D face model dataset is expanded by simplifying and densifying surfaces, thereby solving the problem of the relative lack of a 3D head model dataset, and ensuring training data for supervised deep learning. Second, by directly applying a neural convolution module in 3D space based on a graph neural network structure, the problem of the natural loss of 3D geometric information in detection methods in the 2D space of rendering views is avoided, and the problem of difficulty in learning the inherent relationships between different views is also solved. Finally, the traditional 2D heat map is expanded into a 3D heat map, and the 3D heat map can better ensure the local accuracy of detection results compared to methods that directly regress 3D coordinates.

[0146] Next, the technical solution of the present application will be described from a product perspective. This application proposes a 3D facial keypoint detection method based on a graph neural network structure and a 3D heat map. This method can be integrated into a character animation tool set and, in cooperation with a non-rigid registration algorithm, can complete the deformation matching process between different head models. The specific product form here is a control. In response to a trigger operation of the control, a keypoint detection request carrying relevant data of the 3D head model to be detected can be sent to a remote server where the technical solution of the present application is deployed, and the returned result can be obtained. The remote server deployment method eliminates the need for local plug-in code updates and facilitates iterative optimization of the algorithm, thereby saving local computer resources.

[0147] Next, the technical solution of the present application will be described from a technical aspect.

[0148] First, we will explain the graph convolutional neural network structure in the technical solution of this application. Specifically, a three-dimensional model (three-dimensional network model) naturally has a graph structure relationship, which is not compact and regularly arranged like the pixels of a two-dimensional image, making it inappropriate to directly adopt a conventional convolutional neural network. Therefore, a classical graph attention network (GAT) is introduced. Here, the GAT-based network included in the graph convolutional neural network structure assumes that a graph structure (three-dimensional mesh) includes N nodes (vertices), each node has a feature vector (vertex feature) of h, and the dimension is F, as shown in Equation 1. Assuming that node j is a neighbor of node i (i.e., i and j are edge-connected), the attention mechanism can be used to calculate the importance (correlation value) of node j relative to node i, as shown in Equation 2 and Equation 3. Specifically, the process of calculating the importance of node j relative to node i using the attention mechanism can be as follows: As shown in Equation 4, the features Wh of nodes i and j are used to calculate the importance (correlation value) of node j relative to node i. i and Wh j Then, the feature obtained by the splicing is dot-producted with the weight vector a, whose dimension is 2F. As a result, the feature vector (local feature) of node i is determined based on the importance of node j to node i, as shown in Equation 6.

[0149] In practical applications, a multi-layer GAT splicing method is used, i.e., K feature vectors (local sub-features) corresponding to node i are obtained using K attention mechanisms, as shown in Equation 7, and then the K feature vectors are spliced ​​together to finally obtain the feature vector (local feature) corresponding to node i. In this way, GAT does not rely on the complete graph structure, but only on edges (connections between vertices), improving the flexibility of the keypoint detection process. At the same time, attention mechanisms can also be used to assign different weights to different neighboring nodes, improving the accuracy of the keypoint detection process.

[0150] After describing the Graph Attention Network (GAT), reference will now be made to FIG. 15, which is a schematic diagram illustrating the configuration of a graph convolutional neural network provided in an embodiment of the present application. Here, a 3D head model keypoint automatic detection neural network shown in FIG. 15 is constructed based on the GAT. Based on FIG. 15, the input data, i.e., vertex data, is N×(6+X) (vertices of the 3D mesh), where N represents the number of vertices in the 3D model (3D mesh), 6 represents the dimension occupied by the vertex coordinates and normal vectors, and X includes other characteristics such as the curvature and texture information of the 3D head model vertices (3D mesh). These other characteristics can be adjusted according to the data and task. Generally, the more input characteristics there are, the more useful the neural network is for learning. Meanwhile, A ij is a vertex connectivity matrix (connectivity between vertices), its size is N × N, and its value is 0 or 1. For example, if two vertices i and j are connected, A ij is 1, otherwise it is 0.

[0151] As shown in Figure 15, the multilayer perceptron (MLP) represents a multi-layer fully connected sensing layer. The vertex data (vertices of a 3D mesh) first passes through an MLP module with a hidden layer dimension of [128, 64] to obtain primitive hidden layer features X1 (vertex features). This is then divided into two paths (global feature extraction and local feature extraction). One path passes through the MLP module ([512, 1024]) and performs max pooling on the output feature X2 to obtain global feature information X3, which is shared among all N vertices to determine global feature N × X3. The other path passes through three sets of GAT modules, each of which contains an eight-layer attention-based network (heads). The output layers of the three sets of GAT modules are spliced ​​together to determine local features. Finally, the dual-path features are spliced ​​and input into the final MLP module ([1024,512,K]) to obtain the final N × K (K is the number of keypoints) 3D heatmap data. This data is visualized on a 3D head model to obtain N 3D heatmaps.

[0152] It should be noted that due to the characteristics of the GAT module and the MLP module, the same network structure does not need to have a fixed number of vertices N, so whether in the training stage or in the actual use stage, 3D face models with different numbers of vertices can be used as input to the neural network model, thereby improving the applicability of the present application.

[0153] Next, the 3D heat map in the technical solution of the present application will be described. Because a 3D mesh heat map does not have a compact structure like a 2D image coordinate, the 2D heat map uses Euclidean distance, while the 3D heat map uses geodesic distance. In this way, at the 3D mesh level, verifying the shortest path on the mesh graph structure based on the geodesic distance between two points can better reflect the characteristics of the 3D surface than the Euclidean distance between two points. For example, refer to FIG. 16, which is a comparison diagram between the geodesic distance and the Euclidean distance provided in an embodiment of the present application. Based on FIG. 16, the straight line between two vertices shown in 1602 is the Euclidean distance, and the curve shown in 1601 is the corresponding geodesic distance.

[0154] It should be noted that, once a graph neural network has completed training and is ready for use, the 3D heat map output from the neural network needs to be further converted into the final 3D keypoint coordinates. Traditionally, the method for converting a 2D heat map into 2D coordinates involves first obtaining the vertex coordinates where the maximum probability is located (known as the argmax method), and then weighting the softmax probability expectation of multiple vertex coordinates (i.e., the soft-argmax method) to obtain the final 3D keypoint coordinates. Considering that the soft-argmax method involves weighting multiple 3D coordinates, the result of which may not necessarily fit within the 3D mesh plane, we directly adopt the argmax method to determine the final 3D keypoint coordinates.

[0155] Finally, the data expansion method of the present invention will be described.

[0156] Unlike 2D facial images and 2D keypoint data, it is extremely difficult to obtain large quantities of 3D facial mesh data. Data scarcity is a major problem plaguing supervised neural network learning. Only if the dataset is large enough to cover different facial morphologies can a graph neural network achieve sufficient detection capabilities from it. However, the difficulty of obtaining a 3D facial keypoint dataset is reflected in several aspects. Specifically, 3D mesh facial data itself is created by art staff, and the creation process is tedious. On the other hand, because generating a 2D image requires only pressing the shutter of a camera, 2D facial images are already abundant, both on the Internet and in public academic datasets, while the corresponding 3D facial data is extremely scarce. Second, for keypoint detection tasks, keypoints must be manually labeled in advance (or labeling can be achieved by rudimentary automatic detection of existing algorithms followed by minor manual post-processing). On the other hand, 2D keypoint labeling has been performed by many people, and the labeling tools are not complicated. Essentially, it is only necessary to label specific pixels in a picture. However, marking keypoints in a 3D mesh is more difficult, for example, it is difficult to identify the facial contour. Therefore, when 3D data itself is insufficient, it is not possible to develop a corresponding labeling tool for manually labeling 3D keypoints based on an existing 3D face and head model. Based on this, the technical solution of the present application performs data augmentation on existing 3D face model data based on surface simplification and densification, thereby providing standardized and reasonable training data for a graph neural network.

[0157] Here, data expansion methods can be divided into face simplification and densification. Face simplification can be based on the edge optimization method, that is, as shown in FIG. 12, the smallest edge between nodes is searched for each time and merged into one vertex. Face densification, as shown in FIG. 13, prioritizes calculating the barycentric coordinates of faces with large areas, and then divides the original triangular faces into three based on the barycentric coordinates. Here, both densification and face simplification can control the end of the operation using the final target number of vertices.

[0158] In this way, the present invention can automatically detect specific key points of a 3D game head model, providing an accurate and reliable key point basis for subsequent 3D head model registration work. Compared to traditional methods of manual labeling and head model registration, the present invention can avoid excessive manual work and automatically complete key point-dependent tasks such as 3D head model registration. This significantly reduces the effort of art staff and speeds up the entire production process related to model character animation.

[0159] Furthermore, since this application is based on supervised deep learning of graph neural networks, it can accurately predict the positions of 3D keypoints and has high robustness. At the same time, the forward calculation speed of the deep learning model is extremely fast, and the entire algorithm can complete automatic labeling in just one second, whereas manual work usually takes several minutes. Therefore, this application has great practical value in terms of efficiency. Furthermore, this application does not limit the number of vertices of the input 3D face model, and after supervised learning training, the generated deep learning model can be widely used for the task of automatically detecting 3D head model keypoints with different vertex densities, and has high applicability.

[0160] By applying the above embodiment of the present application, a 3D mesh corresponding to the detected object is obtained, and then a dual-path feature extraction layer is constructed to extract global and local features of the detected object based on the vertex features and the connection relationships between the vertices obtained from the 3D mesh, respectively, and thereby obtain the locations of keypoints on the detected object based on the vertex features obtained from the 3D mesh and the extracted global and local features. In this way, more feature information of the detected object can be extracted through the multi-layer feature extraction layer, and then keypoints of the detected object based on the rich feature information can be detected, greatly improving the accuracy of 3D keypoint detection.

[0161] Next, an exemplary structure in which the keypoint detection device 455 provided in the embodiments of the present application is implemented as a software module will be described. In some embodiments, as shown in FIG. 2, the software modules stored in the keypoint detection device 455 in the memory 450 include an acquisition module 4551, a first feature extraction module 4552, a second feature extraction module 4553, and an output module 4554.

[0162] The acquisition module 4551 is configured to acquire a three-dimensional mesh for representing the detected object, and determine vertices and connection relationships between the vertices of the three-dimensional mesh; a first feature extraction module 4552 configured to perform feature extraction on vertices of the three-dimensional mesh to obtain vertex features of the three-dimensional mesh; the second feature extraction module 4553 is configured to perform global feature extraction on the detection object based on the vertex features to obtain global features of the detection object, and to perform local feature extraction on the detection object based on the vertex features and the connection relationships between the vertices to obtain local features of the detection object; The output module 4554 is configured to perform detection on key points of the detected object based on the vertex features, the global features, and the local features, and obtain positions of the key points of the detected object on the detected object.

[0163] In some embodiments, the acquisition module 4551 is further configured to perform the steps of scanning the detected object with a three-dimensional scanning device to obtain point cloud data of a geometric surface of the detected object, and constructing a three-dimensional mesh corresponding to the detected object based on the point cloud data.

[0164] In some embodiments, the second feature extraction module 4553 is further configured to perform the steps of determining local features of each of the vertices based on the vertex features and the connectivity relationships between the vertices, and determining local features of the detected object based on the local features of each of the vertices.

[0165] In some embodiments, the second feature extraction module 4553 is further configured to perform, for each of the vertices, a process of determining the vertex as a reference vertex, and determining, based on vertex features of each vertex in the three-dimensional mesh, vertex features of the reference vertex and vertex features of other vertices, where the other vertices are any vertices other than the reference vertex; a process of determining, based on the vertex features of the reference vertex, the vertex features of the other vertices and the connection relationships between the vertices, a correlation value between the reference vertex and the other vertices, where the correlation value indicates a magnitude of correlation between the reference vertex and the other vertex; and a process of determining local features of the reference vertex based on the correlation value and the vertex features of the other vertices.

[0166] In some embodiments, the second feature extraction module 4553 is further configured to perform the steps of employing an attention mechanism to determine the correlation degree between the reference vertex and the other vertices based on the vertex features of the reference vertex, the vertex features of the other vertices, and the connection relationships between the vertices; and performing a normalization process on the correlation degree to obtain a correlation value between the reference vertex and the other vertices.

[0167] In some embodiments, when the number of the other vertices is one, the second feature extraction module 4553 is further configured to perform the steps of multiplying the correlation value and the vertex feature of the other vertex to obtain a multiplication result, and determining a local feature corresponding to the reference vertex based on the multiplication result.

[0168] In some embodiments, when the number of the other vertices is multiple, the second feature extraction module 4553 is further configured to perform the steps of: multiplying, for each of the other vertices, the correlation value by the vertex feature of the corresponding other vertex to obtain a multiplication result of the other vertex; performing cumulative addition on the multiplication results of each of the other vertices to obtain an addition result; and determining a local feature corresponding to the reference vertex based on the addition result.

[0169] In some embodiments, the second feature extraction module 4553 is further configured to perform the steps of: performing feature fusion on the local features of each of the vertices based on the local features of each of the vertices to obtain a fusion feature; and setting the fusion feature as a local feature of the detected object.

[0170] In some embodiments, the output module 4554 is further configured to perform the steps of: performing feature splicing on the vertex features, the global features, and the local features to obtain splice features of the detected object; and performing detection on key points of the detected object based on the splice features to obtain positions of the key points of the detected object on the detected object.

[0171] In some embodiments, the output module 4554 is further configured to perform the steps of: detecting key points of the detected object based on the vertex features, the global features, and the local features, and obtaining a probability that the key points are present at each of the vertices in the three-dimensional mesh; generating a three-dimensional heat map corresponding to the three-dimensional mesh based on the probabilities; and determining a position of the key points of the detected object on the detected object based on the three-dimensional heat map.

[0172] In some embodiments, the keypoint detection apparatus is applied to a three-dimensional network model, the three-dimensional network model comprising at least a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, and an output layer, wherein the first feature extraction module 4552 is further configured to perform feature extraction on vertices of the three-dimensional mesh via the first feature extraction layer to obtain vertex features of the three-dimensional mesh, the second feature extraction module 4553 is further configured to perform global feature extraction on the detected object based on the vertex features via the second feature extraction layer to obtain global features of the detected object, and to perform local feature extraction on the detected object based on the vertex features and the connection relationships between the vertices via the third feature extraction layer to obtain local features of the detected object, and the output module 4554 is further configured to detect keypoints of the detected object based on the vertex features, the global features, and the local features via the output layer to obtain positions of the keypoints of the detected object on the detected object.

[0173] In some embodiments, the three-dimensional network model further comprises a first feature splice layer, a second feature splice layer, and a fourth feature extraction layer, and the output module 4554 is further configured to perform: performing feature splicing on the vertex features, the global features, and the local features via the first feature splice layer to obtain splice features of the detected object; performing local feature extraction on the detected object based on the splice features via the fourth feature extraction layer to obtain target local features of the detected object; performing feature splicing on the splice features, the global features, and the target local features via the second feature splice layer to obtain target splice features of the detected object; and performing detection on key points of the detected object based on the target splice features via the output layer to obtain positions of the key points of the detected object on the detected object.

[0174] Next, an exemplary structure of a training device for a 3D network model provided in an embodiment of the present application implemented as a software module will be described, where the 3D network model includes at least a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, and an output layer, and the training device includes an acquisition module, an acquisition module, a first feature extraction module, a second feature extraction module, an output module, and an update module; The acquisition module is configured to acquire object training samples having labels, the labels being for indicating actual locations of keypoints of the object training samples; the acquisition module is configured to acquire a training 3D mesh to represent the object training sample, and determine vertices and connectivity relationships between vertices of the training 3D mesh; the first feature extraction module is configured to perform feature extraction on vertices of the object training samples via the first feature extraction layer to obtain vertex features of the training 3D mesh; the second feature extraction module is configured to perform global feature extraction on the object training sample based on vertex features of the training 3D mesh through the second feature extraction layer, to obtain global features of the object training sample; and to perform local feature extraction on the object training sample based on vertices and inter-vertex connection relationships of the training 3D mesh through the third feature extraction layer, to obtain local features of the object training sample; the output module is configured to perform detection on keypoints of the object training samples based on vertex features of the training 3D mesh, global features of the object training samples, and local features of the object training samples via the output layer, and determine positions of the keypoints of the object training samples on the object training samples; The update module is configured to obtain differences between the positions of keypoints of the object training sample and the label, train the 3D network model based on the differences, and obtain a target 3D network model, and the target 3D network model is configured to perform keypoint detection on the detected object and obtain the positions of keypoints of the detected object on the detected object.

[0175] An embodiment of the present application further provides an electronic device, the electronic device comprising: a memory configured to store computer-executable instructions; and a processor that, when executing the computer-executable instructions stored in the memory, performs the keypoint detection method or the three-dimensional network model training method described in the embodiments of the present application, for example, the keypoint detection method shown in FIG. 3 or the three-dimensional network model training method shown in FIG. 11.

[0176] An embodiment of the present application provides a computer program product or a computer program, the computer program product or the computer program including computer-executable instructions stored in a computer-readable storage medium, wherein a processor of an electronic device reads and executes the computer-executable instructions from the computer-readable storage medium, thereby causing the electronic device to perform the keypoint detection method or the 3D network model training method according to the embodiment of the present application, such as the keypoint detection method shown in FIG. 3 or the 3D network model training method shown in FIG.

[0177] An embodiment of the present application provides a computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a processor, causes the processor to perform the keypoint detection method or the method for training a three-dimensional network model described in the embodiment of the present application, such as the keypoint detection method shown in FIG. 3 or the method for training a three-dimensional network model shown in FIG. 11.

[0178] In some embodiments, the computer-readable storage medium may be memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic memory, optical disk, or CD-ROM, or may be various devices including one or any combination of the above memories.

[0179] In some embodiments, the computer-executable instructions take the form of programs, software, software modules, scripts, or code, and may be written in any type of programming language (including compiled or interpreted, declarative or procedural languages), and may be arranged in any form, including as an independent program, or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0180] As an example, computer-executable instructions do not necessarily correspond to a file in a file system, but may be stored as part of a file that holds other programs or data, for example, within one or more scripts within a HyperText Markup Language (HTML) document, within a single file dedicated to the program being discussed, or within multiple joint files (such as files that store one or more modules, subroutines, or code portions).

[0181] As an example, the executable instructions may be arranged to be executed on a single electronic device, may be arranged to be executed on multiple electronic devices located at a single location, or may be arranged to be executed on multiple electronic devices distributed across multiple locations and interconnected via a communications network.

[0182] As described above, the embodiments of the present application have the following beneficial effects.

[0183] (1) By extracting richer feature information of the detection object through multi-layer feature extraction layers, and then performing detection on the keypoints of the detection object with rich feature information, the accuracy of 3D keypoint detection can be significantly improved.

[0184] (2) GAT improves the flexibility of the keypoint detection process by relying only on edges and not on the complete graph structure. At the same time, it can also employ an attention mechanism to assign different weights to different neighboring nodes, thereby improving the accuracy of the keypoint detection process.

[0185] (3) Automatic detection of specific key points on a 3D game head model can provide an accurate and reliable key point basis for subsequent 3D head model registration work. Compared to traditional methods of manual labeling and head model registration, this application can avoid excessive manual work and automatically complete key point-dependent tasks such as 3D head model registration. This significantly reduces the effort of art staff and speeds up the entire production process related to model character animation.

[0186] (4) Because it is based on supervised deep learning of a graph neural network, it can accurately predict the location of 3D keypoints and has high robustness. At the same time, the forward calculation speed of the deep learning model is extremely fast, and the entire algorithm can complete automatic labeling in just one second, whereas manual labeling usually takes several minutes. This makes the present application highly practical in terms of efficiency. Furthermore, the present application does not limit the number of vertices in the input 3D face model. After supervised learning training, the generated deep learning model can be widely used in the task of automatically detecting 3D head model keypoints with different vertex densities, making it highly applicable.

[0187] The above is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and scope of the present application shall be included in the scope of protection of the present application. [Explanation of symbols]

[0188] 100 Keypoint Detection System 200 servers 300 Network 400 terminals 401 Client 401-1 Display Interface 410 processor 420 Network Interface 430 User Interface 431 Output Device 432 Input Device 440 Bus System 450 memory 451 Operating Systems 452 Network Communication Module 453 Display Module 454 Input Processing Module 455 Keypoint Detector 4551 Acquisition Module 4552 First feature extraction module 4553 Second Feature Extraction Module 4554 Output Module

Claims

1. 1. A keypoint detection method implemented by an electronic device, comprising: Obtaining a three-dimensional mesh to represent the detected object, and determining the vertices and the connection relationships between the vertices of the three-dimensional mesh; performing feature extraction on the vertices of the three-dimensional mesh to obtain vertex features of the three-dimensional mesh; performing global feature extraction on the detection object based on the vertex features to obtain global features of the detection object; and performing local feature extraction on the detection object based on the vertex features and the connection relationships between the vertices to obtain local features of the detection object; detecting keypoints of the object based on the vertex features, the global features, and the local features, and obtaining positions of the keypoints of the object on the object.

2. The step of obtaining a three-dimensional mesh to represent the detected object includes: scanning the detection object with a three-dimensional scanning device to obtain point cloud data of a geometric surface of the detection object; constructing a three-dimensional mesh corresponding to the detection object based on the point cloud data; The keypoint detection method of claim 1 .

3. The step of extracting local features from the detection object based on the vertex features and the connection relationships between the vertices to obtain local features of the detection object includes: determining local features of each vertex based on the vertex features and the connectivity relationships between the vertices; determining a local feature of the object based on the local feature of each vertex; The keypoint detection method according to claim 1 or 2.

4. determining a local feature of each vertex based on the vertex features and the connection relationships between the vertices, For each of the vertices, determining the vertex as a reference vertex, and determining vertex characteristics of the reference vertex and vertex characteristics of other vertices based on vertex characteristics of each vertex in the three-dimensional mesh, wherein the other vertices are any vertices other than the reference vertex; determining a correlation value between the reference vertex and the other vertex based on vertex features of the reference vertex, vertex features of the other vertex, and connection relationships between the vertices, the correlation value indicating the magnitude of the correlation between the reference vertex and the other vertex; determining a local feature corresponding to the reference vertex based on the correlation value and vertex features of the other vertices; The keypoint detection method according to claim 3 .

5. The step of determining a correlation value between the reference vertex and the other vertex based on the vertex characteristics of the reference vertex, the vertex characteristics of the other vertex, and the connection relationship between the vertices includes: Employing an attention mechanism to determine the correlation degree between the reference vertex and the other vertices based on the vertex features of the reference vertex, the vertex features of the other vertices, and the connection relationships between the vertices; performing a normalization process on the correlation degree to obtain a correlation value between the reference vertex and the other vertex; The keypoint detection method according to claim 4.

6. When the number of the other vertices is one, the step of determining a local feature corresponding to the reference vertex based on the correlation value and vertex features of the other vertices includes: multiplying the correlation value by the vertex feature of the other vertex to obtain a multiplication result; determining a local feature corresponding to the reference vertex based on the multiplication result; The keypoint detection method according to claim 4.

7. When the number of the other vertices is plural, the step of determining a local feature corresponding to the reference vertex based on the correlation value and vertex features of the other vertices includes: For each other vertex, multiplying the correlation value by the vertex feature of the corresponding other vertex to obtain a multiplication result of the other vertex; performing cumulative addition on the multiplication results of each of the other vertices to obtain an addition result; and determining a local feature corresponding to the reference vertex based on the summation result. The keypoint detection method according to claim 4.

8. The step of determining a local feature of the detection object based on the local feature of each vertex includes: performing feature fusion on the local features of each vertex based on the local features of each vertex to obtain a fusion feature; and determining the fusion feature as a local feature of the object to be detected. The keypoint detection method according to claim 3 .

9. The step of detecting key points of the object to be detected based on the vertex features, the global features, and the local features, and obtaining positions of the key points of the object to be detected on the object to be detected, includes: performing feature splicing on the vertex features, the global features, and the local features to obtain spliced ​​features of the object; detecting key points of the object to be detected based on the splice features, and obtaining positions of the key points of the object to be detected on the object to be detected; A method for detecting keypoints according to any one of claims 1 to 8.

10. The step of detecting key points of the object to be detected based on the vertex features, the global features, and the local features, and obtaining positions of the key points of the object to be detected on the object to be detected, includes: detecting key points of the object based on the vertex features, the global features, and the local features, and obtaining a probability that the key points exist at each vertex in the 3D mesh; generating a three-dimensional heat map corresponding to the three-dimensional mesh based on the probabilities; and determining a position of a key point of the detected object on the detected object based on the three-dimensional heat map. A method for detecting keypoints according to any one of claims 1 to 9.

11. The keypoint detection method is applied to a three-dimensional network model, and the three-dimensional network model includes at least a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, and an output layer; The step of performing feature extraction on the vertices of the three-dimensional mesh to obtain vertex features of the three-dimensional mesh includes: performing feature extraction on vertices of the three-dimensional mesh via the first feature extraction layer to obtain vertex features of the three-dimensional mesh; The step of performing global feature extraction on the detection object based on the vertex features to obtain global features of the detection object, and performing local feature extraction on the detection object based on the vertex features and the connection relationships between the vertices to obtain local features of the detection object includes: performing global feature extraction on the detection object based on the vertex features via the second feature extraction layer to obtain global features of the detection object; and performing local feature extraction on the detection object based on the vertex features and the connection relationships between the vertices via the third feature extraction layer to obtain local features of the detection object, The step of detecting key points of the object to be detected based on the vertex features, the global features, and the local features, and obtaining positions of the key points of the object to be detected on the object to be detected, includes: performing detection on key points of the object to be detected based on the vertex features, the global features, and the local features through the output layer, and obtaining positions of the key points of the object to be detected on the object to be detected; A method for detecting keypoints according to any one of claims 1 to 10.

12. the three-dimensional network model further comprises a first feature splice layer, a second feature splice layer, and a fourth feature extraction layer; performing detection on key points of the detection object based on the vertex features, the global features, and the local features through the output layer, and obtaining positions of the key points of the detection object on the detection object, performing feature splicing on the vertex features, the global features, and the local features through the first feature splice layer to obtain spliced ​​features of the detected object; performing local feature extraction on the detection object based on the splice feature through the fourth feature extraction layer to obtain a target local feature of the detection object; performing feature splicing on the splice feature, the global feature, and the target local feature through the second feature splice layer to obtain a target splice feature of the detection object; detecting key points of the object to be detected based on the target splice features through the output layer, and obtaining positions of the key points of the object to be detected on the object to be detected; The keypoint detection method of claim 11.

13. A method for training a three-dimensional network model executed by an electronic device, wherein the three-dimensional network model includes at least a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, and an output layer, and the method for training the three-dimensional network model includes: obtaining object training samples with labels, the labels being for indicating actual locations of keypoints of the object training samples; obtaining a training 3D mesh to represent the object training sample, and determining vertices and connectivity relationships between vertices of the training 3D mesh; performing feature extraction on the vertices of the object training samples via the first feature extraction layer to obtain vertex features of the training 3D mesh; performing global feature extraction on the object training samples based on vertex features of the training 3D mesh via the second feature extraction layer to obtain global features of the object training samples; and performing local feature extraction on the object training samples based on vertex features and inter-vertex connection relationships of the training 3D mesh via the third feature extraction layer to obtain local features of the object training samples; via the output layer, performing detection on keypoints of the object training samples based on vertex features of the training 3D mesh, global features of the object training samples, and local features of the object training samples, and determining positions of the keypoints of the object training samples on the object training samples; A method for training a three-dimensional network model, comprising: a step of obtaining a difference between the positions of keypoints of the object training sample and the label, and training the three-dimensional network model based on the difference to obtain a target three-dimensional network model, wherein the target three-dimensional network model is configured to perform keypoint detection on a detected object and obtain the positions of keypoints of the detected object on the detected object.

14. A keypoint detection device, comprising: an acquisition module; a first feature extraction module; a second feature extraction module; and an output module; The acquisition module is configured to acquire a three-dimensional mesh representing the detected object and determine vertices and connectivity relationships between the vertices of the three-dimensional mesh; the first feature extraction module is configured to perform feature extraction on vertices of the three-dimensional mesh to obtain vertex features of the three-dimensional mesh; the second feature extraction module is configured to perform global feature extraction on the detection object based on the vertex features to obtain global features of the detection object, and to perform local feature extraction on the detection object based on the vertex features and connectivity relationships between the vertices to obtain local features of the detection object; The output module is configured to detect keypoints of the detection object based on the vertex features, the global features, and the local features, and obtain positions of the keypoints of the detection object on the detection object.

15. A training device for a three-dimensional network model, the three-dimensional network model comprising at least a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, and an output layer, the training device for the three-dimensional network model comprising: an acquisition module, an acquisition module, a first feature extraction module, a second feature extraction module, an output module, and an update module; The acquisition module is configured to acquire object training samples having labels, the labels being for indicating actual locations of keypoints of the object training samples; the acquisition module is configured to acquire a training 3D mesh to represent the object training sample, and determine vertices and connectivity relationships between vertices of the training 3D mesh; the first feature extraction module is configured to perform feature extraction on vertices of the object training samples via the first feature extraction layer to obtain vertex features of the training 3D mesh; the second feature extraction module is configured to perform global feature extraction on the object training samples through the second feature extraction layer based on vertex features of the training 3D mesh to obtain global features of the object training samples; and to perform local feature extraction on the object training samples through the third feature extraction layer based on vertices and inter-vertex connection relationships of the training 3D mesh to obtain local features of the object training samples; the output module is configured to perform detection on keypoints of the object training samples based on vertex features of the training 3D mesh, global features of the object training samples, and local features of the object training samples via the output layer, and determine positions of the keypoints of the object training samples on the object training samples; The update module is configured to obtain a difference between the position of the keypoints of the object training sample and the label, and train the 3D network model based on the difference to obtain a target 3D network model, and the target 3D network model is configured to perform keypoint detection on a detected object and obtain the position of the keypoints of the detected object on the detected object.

16. a memory configured to store computer-executable instructions; and a processor that, when executing computer-executable instructions stored in the memory, performs the keypoint detection method of any one of claims 1 to 12 or the method for training a three-dimensional network model of claim 13.

17. A computer-readable storage medium having stored thereon computer-executable instructions for causing a processor to perform the keypoint detection method of any one of claims 1 to 12 or the method for training a three-dimensional network model of claim 13.

18. A computer program product comprising a computer program or computer executable instructions for causing a processor to perform the keypoint detection method of any one of claims 1 to 12 or the method for training a three-dimensional network model of claim 13.

Images