Training methods, devices, electronic equipment, and storage media for keypoint detection models

Abstract

This invention relates to a training method, apparatus, electronic device, and storage medium for a keypoint detection model. The method includes: acquiring a dense training sample set and a sparse training sample set, wherein each training sample in the dense training sample set is labeled with a dense set of keypoints, and each training sample in the sparse training sample set is labeled with a sparse set of keypoints; training an initial model using the dense training sample set to obtain a first model, the first model including at least a shallow base network layer and a deep base network layer; constructing a second model using the sparse training sample set and the deep base network layer of the first model; and constructing a keypoint detection model based on the shallow base network layer of the first model and the second model, the keypoint detection model being used to detect keypoints in an image. This improves the accuracy of the keypoint detection model.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a training method, apparatus, electronic device and storage medium for a key point detection model. Background Technology

[0002] Facial landmark detection refers to the precise localization of facial regions, including facial features and contours such as cheeks, eyebrows, eyes, mouth, and nose. Since the first application of CNN (Convolutional Neural Network) technology to facial landmark detection in 2013, this technology has undergone rapid development, and the technical methods have become relatively mature. Both fast detection technologies applied to mobile devices and high-precision detection technologies applied to the cloud have achieved rapid progress.

[0003] With the development of facial landmark technology, it has been applied to various scenarios. For example, the landmark technology can be used to divide the face into areas such as eyebrows, eyes, mouth, and cheeks to achieve facial beautification effects. The outer contour area of ​​the face can be used for face slimming. In addition, there are many other application scenarios, which are widely distributed in the fields of short videos, long videos, and photography.

[0004] However, existing facial landmark detection technologies still suffer from inaccuracies and jitter in some landmarks in practical applications, especially in edge-test models. To meet the speed requirements of edge testing, the models are often lightweight. For example, with 106 landmarks, although the landmarks are marked on the edge of the face contour, the discreteness of the landmarks means that the gaps between the landmarks are large. This makes it difficult for the landmark detection model to express accurate semantic information, resulting in low accuracy and a tendency to make errors. Summary of the Invention

[0005] In view of this, in order to solve the technical problem that the key points extracted by the key point detection model in the prior art often cannot express accurate semantic information, have low accuracy, and are prone to deviation, the present invention provides a training method, device, electronic device and storage medium for a key point detection model.

[0006] In a first aspect, embodiments of the present invention provide a training method for a keypoint detection model, the method comprising:

[0007] Obtain a dense training sample set and a sparse training sample set, wherein each training sample in the dense training sample set is labeled with a dense keypoint set, and each training sample in the sparse training sample set is labeled with a sparse keypoint set.

[0008] The initial model is trained using the dense training sample set to obtain a first model, which includes at least a shallow base network layer and a deep base network layer.

[0009] A second model is constructed using the sparse training sample set and the deep base network layers of the first model;

[0010] Based on the shallow base network layer of the first model and the second model, a key point detection model is constructed, which is used to detect key points in an image.

[0011] As one possible implementation, constructing a second model using the sparse training sample set and the deep base network layers of the first model includes:

[0012] The deep basic network layer in the first model is re-initialized using the deep basic network layer in the first model to obtain the re-initialized initial model;

[0013] The re-initialized initial model is trained using the sparse training sample set to obtain the second model.

[0014] As one possible implementation, training the initial model using the dense training sample set to obtain the first model includes:

[0015] The shallow base network layer and dense keypoint prediction layer in the initial model are trained using the dense training sample set to obtain the first sub-model, and the shallow base network layer, deep base network layer, and dense keypoint prediction layer in the initial model are trained using the dense training sample set to obtain the second sub-model.

[0016] The first model is constructed based on the first sub-model and the second sub-model.

[0017] As one possible implementation, constructing a second model using the sparse training sample set and the deep base network layers of the first model includes:

[0018] The deep basic network layer of the first model and the sparse keypoint prediction layer in the initial model are combined to construct a third sub-model;

[0019] The third sub-model is trained using the sparse training sample set to obtain the second model;

[0020] The keypoint detection model is constructed based on the shallow base network layer of the first model and the second model, including:

[0021] The shallow base network layer of the first model and the second model are combined to construct a key point detection model.

[0022] Secondly, embodiments of the present invention provide a method for detecting key points, the method comprising:

[0023] Acquire the target image;

[0024] The target image is input into a keypoint detection model trained according to any one of the methods described in the first aspect above, to obtain a keypoint set of the target image.

[0025] Thirdly, embodiments of the present invention provide a training apparatus for a keypoint detection model, the apparatus comprising:

[0026] The acquisition module is used to acquire a dense training sample set and a sparse training sample set. Each training sample in the dense training sample set is labeled with a dense keypoint set, and each training sample in the sparse training sample set is labeled with a sparse keypoint set.

[0027] The training module is used to train the initial model using the dense training sample set to obtain a first model, wherein the first model includes at least a shallow base network layer and a deep base network layer.

[0028] The first construction module is used to construct the second model using the sparse training sample set and the deep base network layer of the first model;

[0029] The second building module is used to build a key point detection model based on the shallow base network layer of the first model and the second model. The key point detection model is used to detect key points in an image.

[0030] As one possible implementation, the first building module includes:

[0031] An initialization submodule is used to re-initialize the deep basic network layer in the initial model using the deep basic network layer in the first model, so as to obtain a re-initialized initial model.

[0032] The training submodule is used to train the re-initialized initial model using the sparse training samples to obtain the second model.

[0033] As one possible implementation, the training module includes:

[0034] The training submodule is used to train the shallow base network layer and dense keypoint prediction layer in the initial model using the dense training sample set to obtain a first sub-model, and to train the shallow base network layer, deep base network layer and dense keypoint prediction layer in the initial model using the dense training sample set to obtain a second sub-model.

[0035] A submodule is constructed to build the first model based on the first submodel and the second submodel.

[0036] As one possible implementation, the first building module is specifically used for:

[0037] The deep basic network layer of the first model and the sparse keypoint prediction layer in the initial model are combined to construct a third sub-model;

[0038] The third sub-model is trained using the sparse training sample set to obtain the second model;

[0039] The second building module is specifically used for:

[0040] The shallow base network layer of the first model and the second model are combined to construct a key point detection model.

[0041] Fourthly, embodiments of the present invention provide a key point detection device, the device comprising:

[0042] The image acquisition module is used to acquire the target image;

[0043] An input module is used to input the target image into a keypoint detection model trained according to any one of the methods described in the first aspect above, to obtain a keypoint set of the target image.

[0044] Fifthly, embodiments of the present invention provide an electronic device, including: a processor and a memory, wherein the processor is configured to execute a training program for a keypoint detection model or a keypoint detection program stored in the memory, so as to implement the training method for the keypoint detection model as described in the first aspect, or the keypoint detection method as described in the second aspect.

[0045] In a sixth aspect, embodiments of the present invention provide a storage medium storing one or more programs, which can be executed by one or more processors to implement the training method of the key point detection model described in any one aspect of the first aspect, or the key point detection method described in the second aspect.

[0046] The technical solution provided by this invention obtains a dense training sample set and a sparse training sample set. The initial model is trained using the dense training sample set to obtain a first model, which includes at least a shallow base network layer and a deep base network layer. A second model is constructed using the sparse training sample set and the deep base network layer of the first model. Based on the shallow base network layer of the first model and the second model, a keypoint detection model is constructed. This keypoint detection model is used to detect keypoints in an image. This technical solution, when training the keypoint detection model, divides the base network layer in the initial model into a shallow base network layer and a deep base network layer. After training both the shallow and deep base network layers using a dense training sample set, the shallow base network layer is fixed, and the deep base network layer is trained using a sparse training sample set. This improves the convergence speed of the keypoint detection model. Furthermore, the base network layer trained with the dense training sample set can more effectively constrain the position of keypoints, making the keypoints extracted from the target image by the obtained keypoint detection model more accurate, thus improving the accuracy of the keypoint detection model. Attached Figure Description

[0047] Figure 1 A flowchart illustrating an embodiment of a training method for a keypoint detection model provided by the present invention;

[0048] Figure 2 is a schematic diagram of facial key points provided in an embodiment of the present invention;

[0049] Figure 3 A flowchart illustrating an embodiment of a key point detection method provided by this invention;

[0050] Figure 4 This is a schematic diagram of the initial model structure of a key point detection model provided in an embodiment of the present invention;

[0051] Figure 5 A block diagram illustrating an embodiment of a training device for a keypoint detection model provided in this invention;

[0052] Figure 6 A block diagram illustrating an embodiment of a key point detection device provided in this invention;

[0053] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] The training method of the key point detection model provided by the present invention will be further explained below with reference to the accompanying drawings and specific embodiments. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0056] See Figure 1 This is a flowchart illustrating an embodiment of a training method for a keypoint detection model provided by an embodiment of the present invention. Figure 1 As shown, the process may include the following steps:

[0057] Step 101: Obtain the dense training sample set and the sparse training sample set.

[0058] Each training sample in the aforementioned dense training sample set is labeled with a dense set of keypoints. For example, in the facial keypoint training sample set, the training sample can be labeled with a dense set of facial keypoints, for example... Figure 2A The 826 points of the face shown.

[0059] Each training sample in the aforementioned sparse training sample set is labeled with a sparse set of keypoints. For example, in the face keypoint training sample set, the training samples can be labeled with a sparse set of face keypoints, such as... Figure 2B The face shown has 106 key points. The number of key points in the dense keypoint cluster is greater than the number of key points in the sparse keypoint cluster.

[0060] In one embodiment, the executing entity of this invention may obtain the aforementioned dense training sample set and sparse training sample set from a preset database.

[0061] Step 102: Train the initial model using a dense training sample set to obtain a first model, which includes at least a shallow base network layer and a deep base network layer.

[0062] Step 103: Construct the second model using the sparse training sample set and the deep base network layer of the first model.

[0063] Step 104: Based on the shallow basic network layer of the first model and the second model, construct a key point detection model. The key point detection model is used to detect key points in the image.

[0064] The following provides a unified explanation of steps 102 to 104:

[0065] The aforementioned keypoint detection model is used to detect keypoints in an image, such as facial keypoints in a face image. Optionally, the aforementioned keypoint detection model can be used to detect sparse keypoint sets in an image.

[0066] The initial model mentioned above refers to the untrained model.

[0067] In one embodiment, the initial model may include a base network layer and a fully connected layer. The base network layer, which may be a MobileNetV3 model, is used to extract basic image features. The fully connected layer is used to reduce the dimensionality of the extracted image features.

[0068] Furthermore, to improve the accuracy of the keypoint detection model, the execution entity of this embodiment can divide the aforementioned basic network layer into a shallow basic network layer and a deep basic network layer. The aforementioned shallow basic network layer refers to the basic network layer located in the first stage of the basic network layer, and the aforementioned deep basic network layer refers to the basic network layer located in the second stage of the basic network layer.

[0069] Then, the shallow base network layer and the deep base network layer can be connected to a fully connected layer to form two training branches: the first branch consists of the shallow base network layer and the fully connected layer; the second branch consists of the shallow base network layer, the deep base network layer, and the fully connected layer.

[0070] Then, the two training branches can be trained separately using a dense training sample set to obtain the first model. It can be understood that the first model can include at least a shallow base network layer and a deep base network layer, and both the shallow base network layer and the deep base network layer are base network layers trained with a dense training sample set.

[0071] Based on this, in order to accelerate the convergence speed of the keypoint detection model during training, the executing entity of this embodiment can initialize the deep basic network layer in the initial model using the deep basic network layer in the first model, thus obtaining a re-initialized initial model. The re-initialized initial model may include untrained shallow basic network layers, deep basic network layers trained on a dense training sample set, and untrained fully connected layers.

[0072] Subsequently, the re-initialized initial model can be trained using a sparse training sample set to obtain a second model. This second model may include shallow base network layers, deep base network layers trained first with a dense training sample set and then with a sparse training sample set, and fully connected layers trained with a sparse training sample set. Optionally, the shallow base network layers may be trained with or without a sparse training sample set; this embodiment of the invention does not impose any limitations on this.

[0073] Since the shallow base network layer in the first model is trained through the two training branches mentioned above, it can effectively constrain the position of key points, resulting in a higher accuracy of the trained key point detection model. For example, when training a face key point detection model, it can make the position of the identified key points more closely match the facial contour edges.

[0074] To address this, a keypoint detection model can be constructed based on the shallow basic network layer of the first model and the second model.

[0075] In another embodiment, the initial model may include a base network layer, a dense keypoint prediction layer, and a sparse keypoint prediction layer. The dense keypoint prediction layer can be used to train a dense keypoint detection model, and the sparse keypoint prediction layer can be used to train a sparse keypoint detection model.

[0076] Furthermore, to improve the accuracy of model detection, the execution entity of this embodiment can divide the aforementioned basic network layer into a shallow basic network layer and a deep basic network layer. The aforementioned shallow basic network layer refers to the basic network layer located in the first stage of the basic network layer, and the aforementioned deep basic network layer refers to the basic network layer located in the second stage of the basic network layer.

[0077] Then, the shallow base network layer and the deep base network layer can be connected to a dense keypoint prediction layer to form two training branches: the first training branch consists of the shallow base network layer and the dense keypoint prediction layer; the second training branch consists of the shallow base network layer, the deep base network layer, and the dense keypoint prediction layer.

[0078] Based on this, the two training branches mentioned above can be trained separately using a dense training sample set to obtain the first model. It can be understood that the first model may include a shallow base network layer and a dense keypoint prediction layer trained with a dense training sample set, a shallow base network layer and a deep base network layer trained with a dense training sample set, and a dense keypoint prediction layer.

[0079] Specifically, the first training branch of the initial model, namely the shallow base network layer and the dense keypoint prediction layer, can be trained using the aforementioned dense training sample set to obtain the first sub-model. Then, the second training branch of the initial model, namely the shallow base network layer, the deep base network layer, and the dense keypoint prediction layer, can be trained using the dense training sample set to obtain the second sub-model.

[0080] Then, the first model can be constructed based on the first and second sub-models. Optionally, the first and second sub-models can be directly used to construct the first model.

[0081] Based on this, in order to accelerate the convergence speed of the key point detection model when training the key point detection model, the execution subject of this embodiment of the invention can use the deep basic network layer in the first model and the sparse key point prediction layer in the initial model to construct a third sub-model, and use the sparse training sample set to train the third sub-model to obtain the second model.

[0082] Since the shallow base network layer in the first model is trained through the two training branches mentioned above, it can effectively constrain the position of key points, resulting in a higher accuracy of the trained key point detection model. For example, when training a face key point detection model, it can make the position of the identified key points more closely match the facial contour edges.

[0083] In response, the shallow basic network layer of the first model and the second model can be combined to construct a key point detection model.

[0084] The technical solution provided by this invention involves obtaining a dense training sample set and a sparse training sample set, training an initial model using the dense training sample set to obtain a first model, which includes at least a shallow base network layer and a deep base network layer. A second model is then constructed using the sparse training sample set and the deep base network layer of the first model. Based on the shallow base network layer of the first model and the second model, a key point detection model is constructed, which is used to detect key points in an image. This technical solution divides the base network layers in the initial model into shallow and deep base network layers when training the keypoint detection model. First, it trains both the shallow and deep base network layers using a dense training sample set. Then, it trains the deep base network layers using a sparse training sample set. Finally, it directly constructs the keypoint detection model based on the shallow base network layers trained with the dense training sample set. This improves the convergence speed of the keypoint detection model, and the base network layers trained with the dense training sample set can more effectively constrain the positions of keypoints, making the keypoints extracted from the target image by the obtained keypoint detection model more accurate, thus improving the accuracy of the keypoint detection model.

[0085] See Figure 3 The above is a flowchart of an embodiment of a key point detection method provided by the present invention. Figure 3 The process shown is in Figure 1 Based on the process shown, using Figure 1 The keypoint detection model obtained from the illustrated process enables the detection of keypoints. For example... Figure 3 As shown, the process may include the following steps:

[0086] Step 301: Obtain the target image.

[0087] The target image mentioned above refers to the acquired object image for which key point extraction is to be performed.

[0088] In one embodiment, the executing entity of this invention may be an electronic device, which may be hardware or software that supports network connectivity to provide various network services. When the control terminal is hardware, it may be various electronic devices with displays, including but not limited to smartphones, tablets, laptops, desktop computers, etc. When the control terminal is software, it may be installed in the electronic devices listed above.

[0089] In one embodiment, the executing entity of this invention can acquire the target image using a preset image acquisition device.

[0090] In another embodiment, the executing entity of this invention may obtain the target image input by the user.

[0091] Step 302: Input the target image into the trained key point detection model to obtain the key point set of the target image.

[0092] The above key point detection model can be used for Figure 1 The process shown demonstrates the training of a keypoint detection model. This model can be applied to smart terminals to detect sparse keypoint sets in target images.

[0093] Based on this, the execution subject of this embodiment of the invention can input the acquired target image into the key point detection model to obtain the key point set of the target image.

[0094] The technical solution provided in this embodiment of the invention acquires a target image and inputs the target image into... Figure 1The keypoint detection model trained using the illustrated process yields a keypoint set for the target image. This technical solution, by inputting the target image into the trained keypoint detection model, utilizes a keypoint detection model obtained by training shallow and deep base network layers on a dense training sample set, followed by training the deep base network layer on a sparse sample training set, and finally training the shallow base network layer based on the dense training sample set. Therefore, the resulting keypoint detection model effectively constrains the location of keypoints, improving the accuracy of the keypoint detection model. This achieves more accurate extraction of the keypoint set from the target image.

[0095] See Figure 4 This is a schematic diagram of the initial model structure of a key point detection model provided in an embodiment of the present invention. Figure 4 As shown, the initial model structure may include the following:

[0096] Existing sparse keypoint detection models may include Figure 4 The diagram shows a basic network layer and a sparse keypoint prediction layer, which may include a fully connected layer and a sparse keypoint prediction result module.

[0097] Based on this, Figure 4 The structure shown divides the basic network layer into basic network layer 1 and basic network layer 2 (i.e., shallow basic network layer and deep basic network layer), and adds dense keypoint prediction layers to the shallow basic network layer and the deep basic network layer respectively. The dense keypoint prediction layer may include a fully connected layer and a dense keypoint prediction result module.

[0098] In this context, D can represent a higher feature dimension for keypoints, N1 represents a sparse keypoint feature dimension, and N2 represents a dense keypoint feature dimension. For example, with 826 dense keypoints and 106 sparse keypoints, N2 could be 826*2, meaning 826 dense keypoints are predicted, including both x and y coordinate parameters; N1 could be 106*2, meaning 106 points are predicted with their x and y coordinates. The feature dimension D is typically 256.

[0099] Training the keypoint detection model based on the initial model described above can be divided into two stages: In the first stage, a dense keypoint detection model can be trained on two dense keypoint prediction layers using a dense training sample set, without adding any sparse training sample set. During training, only the shallow base network layer and the dense keypoint prediction layer, as well as the shallow base network layer, the deep base network layer, and the dense keypoint prediction layer, can be trained, while the sparse keypoint prediction layer is not trained. This continues until the initial model fully converges on the dense training sample set.

[0100] In the second stage, after the dense keypoint prediction layers have been trained, the two dense keypoint prediction layers mentioned above can be deleted. The parameters of the shallow base network layers are fixed, and the keypoint detection model is trained on the deep base network layers and the sparse keypoint prediction layers using a sparse training sample set. During training, only the sparse training sample set is used; no dense training sample set is added. Furthermore, the deep base network layers trained in the first stage are used to initialize the deep base network layers in this training process.

[0101] The initial model and training method for the keypoint detection model provided in this invention first train a shallow base network layer, a deep base network layer, and a dense keypoint prediction layer using a dense training sample set. After the dense keypoint prediction layer training converges, the dense keypoint prediction layer is deleted, and then a sparse keypoint prediction layer is trained. During training, the deep base network layer trained with the dense training sample set is used to initialize the model, and then the sparse keypoint detection layer is trained using the sparse training sample set until the model converges. This method, based on the learned dense training sample set, then learns from the sparse training sample set, which can effectively accelerate the convergence speed of the sparse keypoint prediction layer training. At the same time, it effectively constrains the position of discrete points, strengthens the spatial distance and position constraints of discrete points, and improves the accuracy of the keypoint detection model. Furthermore, it has strong applicability, does not modify the model structure in the inference stage, and only affects the training process.

[0102] See Figure 5 This is a block diagram illustrating an embodiment of a training device for a keypoint detection model provided by an embodiment of the present invention. Figure 5 As shown, the device includes:

[0103] The acquisition module 51 is used to acquire a dense training sample set and a sparse training sample set, wherein each training sample in the dense training sample set is labeled with a dense key point set, and each training sample in the sparse training sample set is labeled with a sparse key point set.

[0104] Training module 52 is used to train the initial model using the dense training sample set to obtain a first model, the first model including at least a shallow base network layer and a deep base network layer.

[0105] The first construction module 53 is used to construct a second model using the sparse training sample set and the deep basic network layer of the first model;

[0106] The second construction module 54 is used to construct a key point detection model based on the shallow base network layer of the first model and the second model, wherein the key point detection model is used to detect key points in an image.

[0107] As one possible implementation, the first building module 53 includes (not shown in the figure):

[0108] An initialization submodule is used to re-initialize the deep basic network layer in the initial model using the deep basic network layer in the first model, so as to obtain a re-initialized initial model.

[0109] The training submodule is used to train the re-initialized initial model using the sparse training samples to obtain the second model.

[0110] As one possible implementation, the training module 52 includes (not shown in the figure):

[0111] The training submodule is used to train the shallow base network layer and dense keypoint prediction layer in the initial model using the dense training sample set to obtain a first sub-model, and to train the shallow base network layer, deep base network layer and dense keypoint prediction layer in the initial model using the dense training sample set to obtain a second sub-model.

[0112] A submodule is constructed to build the first model based on the first submodel and the second submodel.

[0113] As one possible implementation, the first building module 53 is specifically used for:

[0114] The deep basic network layer of the first model and the sparse keypoint prediction layer in the initial model are combined to construct a third sub-model;

[0115] The third sub-model is trained using the sparse training sample set to obtain the second model;

[0116] The second building module 54 is specifically used for:

[0117] The shallow base network layer of the first model and the second model are combined to construct a key point detection model.

[0118] See Figure 6 This is a block diagram illustrating an embodiment of a key point detection device provided by an embodiment of the present invention. Figure 6 As shown, the device includes:

[0119] Image acquisition module 61 is used to acquire the target image;

[0120] The input module 62 is used to input the target image into the key point detection model trained according to the method described above, so as to obtain the key point set of the target image.

[0121] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 7The illustrated electronic device 700 includes at least one processor 701, a memory 702, at least one network interface 704, and a user interface 703. The various components in the electronic device 700 are coupled together via a bus system 705. It is understood that the bus system 705 is used to implement communication between these components. In addition to a data bus, the bus system 705 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 7 The general labeled all buses as Bus System 705.

[0122] The user interface 703 may include a display, keyboard, or clicking device (e.g., mouse, trackball, touchpad, or touchscreen).

[0123] It is understood that the memory 702 in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 702 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0124] In some implementations, memory 702 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 7021 and application program 7022.

[0125] The operating system 7021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 7022 includes various applications, such as a media player and a browser, used to implement various application functions. The program implementing the method of this embodiment can be included in the application program 7022.

[0126] In this embodiment of the invention, by calling the program or instructions stored in the memory 702, specifically the program or instructions stored in the application program 7022, the processor 701 executes the method steps provided in each method embodiment, including, for example:

[0127] Obtain a dense training sample set and a sparse training sample set, wherein each training sample in the dense training sample set is labeled with a dense keypoint set, and each training sample in the sparse training sample set is labeled with a sparse keypoint set.

[0128] The initial model is trained using the dense training sample set to obtain a first model, which includes at least a shallow base network layer and a deep base network layer.

[0129] A second model is constructed using the sparse training sample set and the deep base network layers of the first model;

[0130] Based on the shallow base network layer of the first model and the second model, a key point detection model is constructed, which is used to detect key points in an image.

[0131] or,

[0132] Acquire the target image;

[0133] The target image is input into the keypoint detection model trained according to the method described above to obtain the keypoint set of the target image.

[0134] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 701. Processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 701 or by instructions in software form. The processor 701 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 702. Processor 701 reads the information in memory 702 and, in conjunction with its hardware, completes the steps of the above method.

[0135] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0136] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0137] The electronic device provided in this embodiment may be as follows: Figure 7 The electronic device shown can perform the following: Figure 1 All steps of the training method for the keypoint detection model, or Figure 3 All steps of the key point detection method are implemented to achieve... Figure 1 Training methods for key point detection models or Figure 3 For details on the technical effectiveness of the detection method for key points, please refer to [link / reference]. Figure 1 and Figure 3 The relevant descriptions are presented concisely and will not be elaborated upon here.

[0138] This invention also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; the memory may also include combinations of the above types of memory.

[0139] When one or more programs in the storage medium can be executed by one or more processors to implement the above-mentioned key point detection model training method or key point detection method executed on the electronic device side.

[0140] The processor is used to execute the training program of the keypoint detection model or the keypoint detection method stored in the memory, so as to implement the following steps of the keypoint detection model training method or the keypoint detection method executed on the electronic device side:

[0141] Obtain a dense training sample set and a sparse training sample set, wherein each training sample in the dense training sample set is labeled with a dense keypoint set, and each training sample in the sparse training sample set is labeled with a sparse keypoint set.

[0142] The initial model is trained using the dense training sample set to obtain a first model, which includes at least a shallow base network layer and a deep base network layer.

[0143] A second model is constructed using the sparse training sample set and the deep base network layers of the first model;

[0144] Based on the shallow base network layer of the first model and the second model, a key point detection model is constructed, which is used to detect key points in an image.

[0145] or,

[0146] Acquire the target image;

[0147] The target image is input into the keypoint detection model trained according to the method described above to obtain the keypoint set of the target image.

[0148] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0149] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0150] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for training a key point detection model, the method comprising: The method comprises: obtaining a dense training sample set and a sparse training sample set, each training sample in the dense training sample set being labeled with a dense key point set, and each training sample in the sparse training sample set being labeled with a sparse key point set; training an initial model by using the dense training sample set to obtain a first model, the first model comprising at least a shallow basic network layer and a deep basic network layer; the initial model comprising a basic network layer and a full connection layer, the basic network layer being used for extracting basic features of an image; the shallow basic network layer refers to a basic network layer located at a first stage in the basic network layer, and the deep basic network layer refers to a basic network layer located at a second stage in the basic network layer; constructing a second model by using the sparse training sample set and the deep basic network layer of the first model; constructing a key point detection model based on the shallow basic network layer of the first model and the second model, the key point detection model being used for detecting key points in an image; the training of the initial model by using the dense training sample set to obtain the first model comprises: training the shallow basic network layer and a dense key point prediction layer in the initial model by using the dense training sample set to obtain a first sub-model, and training the shallow basic network layer, the deep basic network layer, and the dense key point prediction layer in the initial model by using the dense training sample set to obtain a second sub-model; constructing the first model based on the first sub-model and the second sub-model.

2. The method of claim 1, wherein, the construction of the second model by using the sparse training sample set and the deep basic network layer of the first model comprises: reinitializing the deep basic network layer in the initial model by using the deep basic network layer in the first model to obtain a reinitialized initial model; training the reinitialized initial model by using the sparse training sample set to obtain the second model.

3. The method of claim 1, wherein, the construction of the second model by using the sparse training sample set and the deep basic network layer of the first model comprises: constructing the deep basic network layer of the first model and a sparse key point prediction layer in the initial model as a third sub-model; training the third sub-model by using the sparse training sample set to obtain the second model; the construction of the key point detection model based on the shallow basic network layer of the first model and the second model comprises: constructing the shallow basic network layer of the first model and the second model as the key point detection model.

4. A method of detecting a key point, characterized by, The method comprises: obtaining a target image; inputting the target image into the key point detection model trained according to the method in any one of claims 1 to 3 to obtain a key point set of the target image. 5.A device for training a key point detection model, characterized in that, The device comprises: an obtaining module configured to obtain a dense training sample set and a sparse training sample set, each training sample in the dense training sample set being labeled with a dense key point set, and each training sample in the sparse training sample set being labeled with a sparse key point set; The training module is configured to train an initial model by using the dense training sample set to obtain a first model, the first model comprising at least a shallow basic network layer and a deep basic network layer; the initial model comprising a basic network layer and a full connection layer, the basic network layer being configured to extract basic features of an image; the shallow basic network layer being a basic network layer at a first stage in the basic network layer, and the deep basic network layer being a basic network layer at a second stage in the basic network layer; The first constructing module is configured to construct a second model by using the sparse training sample set and the deep basic network layer of the first model; The second constructing module is configured to construct a key point detection model based on the shallow basic network layer of the first model and the second model, the key point detection model being configured to detect key points in an image; The training of the initial model by using the dense training sample set to obtain the first model comprises: training the shallow basic network layer and a dense key point prediction layer in the initial model by using the dense training sample set to obtain a first sub-model, and training the shallow basic network layer, the deep basic network layer, and the dense key point prediction layer in the initial model by using the dense training sample set to obtain a second sub-model; constructing the first model based on the first sub-model and the second sub-model.

6. The apparatus of claim 5, wherein, The first constructing module comprises: an initialization sub-module configured to re-initialize the deep basic network layer in the initial model by using the deep basic network layer in the first model to obtain a re-initialized initial model; a training sub-module configured to train the re-initialized initial model by using the sparse training sample set to obtain the second model.

7. An apparatus for detecting a key point, characterized by, The apparatus comprises: an image acquisition module configured to acquire a target image; an input module configured to input the target image into a key point detection model trained according to the method in any one of claims 1-3 to obtain a key point set of the target image.

8. An electronic device, comprising: comprise: a processor and a memory, the processor being configured to execute a training program of the key point detection model or a detection program of key points stored in the memory to implement the training method of the key point detection model in any one of claims 1-3 or the detection method of key points in claim 4.

9. A storage medium, characterized by The storage medium stores one or more programs, and the one or more programs are executable by one or more processors to implement the training method of the key point detection model in any one of claims 1-3 or the detection method of key points in claim 4.

Images