Key point detection method, device, equipment and computer readable medium

Abstract

The embodiments of the present disclosure disclose a key point detection method, device, electronic device and computer-readable medium. A specific implementation of the method includes: extracting features from the image to be detected to obtain image features; inputting the image features into a pre-trained heat map regression network to obtain a key point heat map; inputting the output result of the intermediate layer of the heat map regression network into a pre-trained coordinate regression network to obtain a key point coordinate set; based on the key point heat map and the key point coordinate set, generating key point position information corresponding to the image to be detected. This implementation achieves the requirements of simultaneously meeting the accuracy rate and the correlation relationship between key points.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to key point detection methods, devices, equipment, and computer-readable media. Background Art

[0002] Keypoint detection is widely used in various computer vision tasks. Related keypoint detection techniques include coordinate regression and heatmap regression.

[0003] However, when the above method is used for key point detection, the following technical problems often occur:

[0004] When the coordinate regression method is used, the coordinates of the key points are not accurate enough. When the heat map regression method is used, the correlation between the key points is weak. In other words, the relevant detection methods cannot simultaneously meet the requirements of accuracy and correlation between key points. Summary of the invention

[0005] The content section of this disclosure is used to introduce concepts in a brief form, and these concepts will be described in detail in the specific implementation section below. The content section of this disclosure is not intended to identify the key features or essential features of the technical solution for which protection is sought, nor is it intended to limit the scope of the technical solution for which protection is sought. Some embodiments of the present disclosure propose key point detection methods, devices, electronic devices, and computer-readable media to solve one or more of the technical problems mentioned in the above background technology section.

[0006] In a first aspect, some embodiments of the present disclosure provide a key point detection method, which includes: extracting features from an image to be detected to obtain image features; inputting the image features into a pre-trained heat map regression network to obtain a key point heat map; inputting the output result of the intermediate layer of the heat map regression network into a pre-trained coordinate regression network to obtain a key point coordinate set; based on the key point heat map and the key point coordinate set, generating key point position information corresponding to the image to be detected.

[0007] In a second aspect, some embodiments of the present disclosure provide a key point detection device, comprising: an extraction unit, configured to perform feature extraction on an image to be detected to obtain image features; a heat map generation unit, configured to input image features into a pre-trained heat map regression network to obtain a key point heat map; a coordinate generation unit, configured to input an output result of an intermediate layer of the heat map regression network into a pre-trained coordinate regression network to obtain a set of key point coordinates; and a position information generation unit, configured to generate key point position information corresponding to the image to be detected based on the key point heat map and the key point coordinate set.

[0008] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0009] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0010] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the key point detection method of some embodiments of the present disclosure, the heat map regression network and the coordinate regression network are combined, thereby combining the advantages of the two methods. Specifically, the key point coordinates predicted by the heat map regression network are accurate, and the correlation between the key points predicted by the coordinate regression network is strong. Therefore, the requirements of accuracy and correlation between key points can be met at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0012] Figure 1 is a schematic diagram of an application scenario of a key point detection method of some embodiments of the present disclosure;

[0013] Figure 2 is a flow chart of some embodiments of the key point detection method according to the present disclosure;

[0014] Figure 3 The hand key point detection results obtained by using the heat map regression network are shown as an example;

[0015] Figure 4 The hand key point detection results obtained by using the coordinate regression network are shown as an example;

[0016] Figure 5 are flow charts of other embodiments of the key point detection method according to the present disclosure;

[0017] Figure 6 is a schematic structural diagram of some embodiments of the key point detection device according to the present disclosure;

[0018] Figure 7 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0020] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0021] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0022] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0023] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0024] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0025] Figure 1 It is a schematic diagram of an application scenario of the key point detection method of some embodiments of the present disclosure.

[0026] The executor of the key point detection method can be any computing device. The above computing device can be hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or it can be implemented as a single server or a single terminal device. When the computing device is embodied as software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here.

[0027] exist Figure 1In the application scenario, the computing device may first input the image to be detected 101 into the feature extraction network 101 for feature extraction. On this basis, the obtained image features may be input into the pre-trained heat map regression network 103 to obtain the key point heat map 104. In addition, the computing device may input the intermediate layer ( Figure 1 The output result of the first layer (taking the third layer as an example) is input into the pre-trained coordinate regression network 105 to obtain the key point coordinate set 106. Then, the computing device can generate the key point position information 107 corresponding to the image to be detected based on the key point heat map 104 and the key point coordinate set 106. For the convenience of explanation, the key point position information 107 can be visualized on the image to be detected, as shown in 108 in the figure.

[0028] Continue to refer Figure 2 , shows a process 200 of some embodiments of the key point detection method according to the present disclosure. The key point detection method comprises the following steps:

[0029] Step 201, extracting features from the image to be detected to obtain image features.

[0030] In some embodiments, the execution subject of the key point detection method can use various feature extraction algorithms to extract features from the image to be detected to obtain image features. For example, the image to be detected can be input into a convolutional neural network to obtain image features. For another example, image features can also be extracted through algorithms such as color histograms and color correlation graphs. Among them, the image to be detected can be any image. For example, in the scene of gesture recognition, the image to be detected can be an image currently captured by a camera. Of course, it can also be an image that has been preprocessed from the captured image, etc. For another example, it can also be an image in a gallery specified by the user.

[0031] In some optional implementations of some embodiments, before extracting features from the image to be detected and obtaining image features, the above method may also include: performing target part detection on the original image to be detected to obtain an image area showing the target part; scaling the image area to the target size to obtain the image to be detected. In practice, there may be a lot of content displayed in the original image to be detected. For example, in the scenario of hand key point detection, in addition to the hand, the original image to be detected may also show legs, body, head, etc. Other content may interfere with the key point detection of the target part. Therefore, the target part detection can be performed first to obtain the image area showing the target part. On this basis, in order to facilitate unified processing, the image area can be scaled to the target size to obtain the image to be detected.

[0032] In some optional implementations of some embodiments, key point position information corresponding to the image to be detected is generated based on the key point heat map and the key point coordinate set, including: generating key point position information corresponding to the original image to be detected based on the key point heat map and the key point coordinate set. In these optional implementations, since the image to be detected is obtained based on the original image to be detected, the key point position information corresponding to the original image to be detected can be generated as needed.

[0033] In some optional implementations of some embodiments, the key point heat map and the key point coordinate set are respectively mapped to the image to be detected to obtain a target image containing the heat map mapping key point set and the coordinate mapping key point set, including: mapping the key point heat map and the key point coordinate set to the original image to be detected to obtain a target image containing the heat map mapping key point set and the coordinate mapping key point set.

[0034] Step 202: input the image features into a pre-trained heat map regression network to obtain a key point heat map.

[0035] In some embodiments, the above-mentioned execution entity can input image features into a pre-trained heat map regression network to obtain a key point heat map. Among them, the heat map regression network can be used to predict a two-dimensional heat map. The principle is to generate a heat map (heatmap) using a two-dimensional Gaussian function to generate the coordinate position of the standard (groundTruth) of the key point, and finally take the position coordinates with the highest activation value in the map as the final key point coordinates. As an example, the heat map regression network may include multiple (for example, 3) deconvolution layers and an output layer. In addition, as needed, the heat map network may also include network structures such as residual blocks.

[0036] In practice, the prediction accuracy of the heatmap regression network is high, but it cannot predict the correlation between key points well. Figure 3 The following example shows the hand key point detection results obtained using the heat map regression network. In practice, in order to facilitate the subsequent use of the key point detection results for gesture detection and other processing, the key points are generally numbered. In this example, the three key points on the index finger are numbered 1-3, and the three key points on the middle finger are numbered 4-6. Figure 3 As shown in 301. However, Figure 3As shown in 302, the hand key point detection result obtained by using the heat map regression network, although the positions of each key point are relatively accurate as a whole, the positions of the key points numbered 2 and 4 have changed. If the key points are connected in sequence, it can be intuitively seen that the connection lines between the key points have changed significantly. The connection lines between the key points can represent the association relationship between the key points. If gesture detection is performed subsequently, the association relationship between the key points needs to be used. For example, to determine whether the user is "making a yes", it is necessary to determine the relative position relationship between the connection lines of the middle finger and index finger key points.

[0037] Step 203, input the output result of the intermediate layer of the heat map regression network into the pre-trained coordinate regression network to obtain a set of key point coordinates.

[0038] In some embodiments, the above-mentioned execution entity can input the output result of the intermediate layer of the above-mentioned heat map regression network into a pre-trained coordinate regression network to obtain a set of key point coordinates. Among them, the output result of any intermediate layer can be selected to input the coordinate regression network. As an example, the network structure of the coordinate regression network may include multiple convolutional layers and pooling layers, and a reshape layer. In practice, the training sample set can be used to pre-train the initial coordinate regression network and the heat map regression network through some machine learning methods. For example, the initial coordinate regression network and the heat map regression network can be trained by back propagation and stochastic gradient descent methods, and the coordinate regression network is obtained when the training stop conditions are met. Among them, the above-mentioned initial coordinate regression network and the heat map regression network can be trained separately, or they can be trained jointly.

[0039] In practice, the prediction accuracy of the coordinate regression network is low, but it can predict the association relationship between key points well. Figure 4 The hand key point detection results obtained by using the coordinate regression network are shown as an example. Figure 3 Similarly, the three key points on the index finger are numbered 1-3, and the three key points on the middle finger are numbered 4-6, as shown in Figure 401. The hand key point detection result obtained by using the coordinate regression network is shown in Figure 402. The connection lines between the key points do not change significantly, but the position coordinate prediction of some key points (such as the key point numbered 4) has a large offset.

[0040] Step 204: Generate key point position information corresponding to the image to be detected based on the key point heat map and the key point coordinate set.

[0041] In some embodiments, the above-mentioned execution subject can generate key point position information corresponding to the image to be detected based on the key point heat map and the key point coordinate set. As an example, the above-mentioned execution subject performs coordinate regression on the key point heat map. Specifically, the position coordinates with the highest activation value in the key point heat map can be first obtained, and the average value can be taken with the key point position information. On this basis, the obtained average value can be used as the key point position information corresponding to the image to be detected.

[0042] The method provided by some embodiments of the present disclosure combines the advantages of the heat map regression network and the coordinate regression network, thereby combining the advantages of the two methods. Thus, the requirements of accuracy and the correlation between key points can be met at the same time. In this process, compared with inputting image features into two branch networks separately, using the output results of the middle layer of the heat map network to further perform coordinate regression helps to integrate the capabilities of the two networks, thereby better meeting the requirements of accuracy and the correlation between key points.

[0043] Further references Figure 5 , which shows a process 500 of another embodiment of a key point detection method. The process 500 of the key point detection method includes the following steps:

[0044] Step 501: extract features from the image to be detected to obtain image features.

[0045] In some embodiments, the key point detection method is executed on the execution body (eg Figure 1 server as shown).

[0046] Step 502: Input the image features into a pre-trained heat map regression network to obtain a key point heat map.

[0047] Step 503, input the output result of the intermediate layer of the heat map regression network into the pre-trained coordinate regression network to obtain a set of key point coordinates.

[0048] In some embodiments, the specific implementation of steps 501-503 and the technical effects thereof can be referred to in Figure 2 The corresponding embodiments will not be described in detail here.

[0049] Step 504 , respectively mapping the key point heat map and the key point coordinate set to the image to be detected, to obtain a target image containing the key point set mapped by the heat map and the key point set mapped by the coordinates.

[0050] In some embodiments, for the key point heat map, the execution body of the key point detection method can take at least one position with the highest activation value in the map, and on this basis, obtain the heat map mapping key point in the image to be detected through a certain mapping. For each key point coordinate in the key point coordinate set, the coordinate mapping key point can be obtained through a certain mapping. Among them, according to actual needs, the mapping may include matrix transformation, multiplication with fixed coefficients, etc. It can be understood that the target image is obtained by mapping the key point heat map and the key point coordinate set to the image to be detected.

[0051] Step 505 , selecting a key point from each key point group in the target image as a target key point, and obtaining a target key point set, wherein each key point group includes a corresponding heat map mapping key point and a coordinate mapping key point.

[0052] In some embodiments, since the heat map mapping key point set and the coordinate mapping key point set are both prediction results for the key points in the image to be detected. Therefore, for the same position (such as the fingertip of the thumb), there will be a heat map mapping key point and a coordinate mapping key point, that is, a key point group corresponding to it. The two key points in the key point group corresponding to the same position also correspond to each other. For each key point group, a key point can be selected as the key point corresponding to this position, that is, the target key point. Since there are multiple positions in the target image, there are multiple key point groups. Therefore, a target key point set will be obtained. As an example, a key point can be randomly selected as the target key point. In this way, from a holistic perspective, the combination of the two networks can be achieved.

[0053] In some optional implementations of some embodiments, based on the distance between two key points in each key point group, a key point is selected from the key point group as the target key point. As an example, the distance between the two key points may be the Euclidean distance.

[0054] As an example, in response to determining that the distance is less than or equal to a preset threshold, it means that the prediction results of the two networks are relatively similar. In other words, no matter which network's prediction result is selected, the location information of the key point is relatively accurate. At this time, the coordinate mapping key points in the key point group are preferentially determined as target key points. Since the coordinate mapping key points can also meet the requirements of the association relationship, the dual requirements of accuracy and the association relationship between key points can be met at the same time. In response to determining that the distance is greater than the preset threshold, the heat map mapping key points in the key point group are determined as target key points. In these implementations, the selection of key points is achieved by setting a threshold, which can further take into account the requirements of accuracy and the association relationship between key points.

[0055] Step 506: determine the position information of each target key point in the target key point set as the key point position information corresponding to the image to be detected.

[0056] In some optional implementations of some embodiments, the heat map regression network and the coordinate regression network are trained by the following steps: training the initial heat map regression network with a first learning rate until the convergence condition is met to obtain an intermediate heat map regression network; jointly training the intermediate heat map regression network and the initial coordinate regression network with a second learning rate until the training end condition is met to obtain a heat map regression network and a coordinate regression network, wherein the second learning rate is less than the first learning rate.

[0057] In practice, since the heat map regression network is more sensitive to the location of key points, while the coordinate regression network is more sensitive to the relationship between key points, the convergence directions of the two networks during training are different. Based on this, if the two networks are trained directly, it is equivalent to converging in two directions at the same time, which will inevitably affect the overall convergence speed and the accuracy of the network's prediction results.

[0058] In addition, as an important hyperparameter in deep learning, the learning rate determines whether the objective function can converge to the local minimum and when it converges to the minimum. A suitable learning rate can make the objective function converge to the local minimum in a suitable time.

[0059] Based on this, in these implementations, the initial heat map regression network can be first trained with a larger first learning rate, so that the heat map regression network converges quickly and learns the key point position information first. On this basis, the intermediate heat map regression network and the initial coordinate regression network are jointly trained with a smaller second learning rate, so that the correlation between key points can be learned while finding the local optimal solution, thereby simultaneously meeting the requirements of accuracy and the correlation between key points.

[0060] and Figure 2 Compared with the description of some corresponding embodiments, Figure 5 The process 500 of the key point detection method in some corresponding embodiments obtains the target key point by selecting the key point from the key point group. From the overall point of view, there must be some key point heat maps and some key point coordinate sets, so that the advantages of the two networks can be combined to simultaneously meet the requirements of accuracy and the correlation between key points.

[0061] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a key point detection device. These device embodiments are similar to Figure 2Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0062] like Figure 6 As shown, the key point detection device 600 of some embodiments includes: an extraction unit 601, a heat map generation unit 602, a coordinate generation unit 603 and a position information generation unit 604. Among them, the extraction unit 601 is configured to extract features of the image to be detected to obtain image features. The heat map generation unit 602 is configured to input the image features into a pre-trained heat map regression network to obtain a key point heat map. The coordinate generation unit 603 is configured to input the output result of the intermediate layer of the heat map regression network into a pre-trained coordinate regression network to obtain a key point coordinate set. The position information generation unit 604 is configured to generate key point position information corresponding to the image to be detected based on the key point heat map and the key point coordinate set.

[0063] In an optional implementation of some embodiments, the position information generation unit 604 is further configured to: map the key point heat map and the key point coordinate set to the image to be detected, respectively, to obtain a target image containing a heat map mapping key point set and a coordinate mapping key point set; select a key point from each key point group in the target image as a target key point to obtain a target key point set, wherein each key point group includes a corresponding heat map mapping key point and a coordinate mapping key point; determine the position information of each target key point in the target key point set as the key point position information corresponding to the image to be detected.

[0064] In an optional implementation of some embodiments, the position information generating unit 604 is further configured to: select a key point from the key point group as a target key point based on the distance between two key points in each key point group.

[0065] In an optional implementation of some embodiments, the location information generation unit 604 is further configured to: in response to determining that the distance is less than or equal to a preset threshold, determine the heat map mapping key point in the key point group as the target key point; in response to determining that the distance is greater than the preset threshold, determine the coordinate mapping key point in the key point group as the target key point.

[0066] In an optional implementation of some embodiments, the heat map regression network includes multiple deconvolution layers; and the coordinate generation unit 603 is configured to: input the output result of the last deconvolution layer among the multiple deconvolution layers into the coordinate regression network to obtain a set of key point coordinates.

[0067] In an optional implementation of some embodiments, the heat map regression network and the coordinate regression network are trained by the following steps: training the initial heat map regression network with a first learning rate until a convergence condition is met to obtain an intermediate heat map regression network; jointly training the intermediate heat map regression network and the initial coordinate regression network with a second learning rate until a training end condition is met to obtain a heat map regression network and a coordinate regression network, wherein the second learning rate is less than the first learning rate.

[0068] In an optional implementation of some embodiments, the device 600 further includes: a detection unit and a scaling unit. The detection unit is configured to perform target part detection on the original image to be detected to obtain an image area showing the target part. The scaling unit is configured to scale the image area to a target size to obtain the image to be detected. The position information generating unit 604 is further configured to: generate key point position information corresponding to the original image to be detected based on the key point heat map and the key point coordinate set.

[0069] In an optional implementation of some embodiments, the position information generation unit 604 is further configured to: map the key point heat map and the key point coordinate set to the original image to be detected, respectively, to obtain a target image containing a heat map mapping key point set and a coordinate mapping key point set.

[0070] It is understood that the units described in the device 600 are similar to those described in the reference Figure 2 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 600 and the units included therein, and will not be described in detail here.

[0071] Reference below Figure 7 , which shows a structural schematic diagram of an electronic device 700 suitable for implementing some embodiments of the present disclosure. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0072] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0073] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 7 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0074] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0075] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0076] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0077] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: extracts features from the image to be detected to obtain image features; inputs the image features into a pre-trained heat map regression network to obtain a key point heat map; inputs the output result of the intermediate layer of the heat map regression network into a pre-trained coordinate regression network to obtain a key point coordinate set; and generates key point position information corresponding to the image to be detected based on the key point heat map and the key point coordinate set.

[0078] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as a separate software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0079] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0080] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor including an extraction unit, a heat map generation unit, a coordinate generation unit, and a position information generation unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the extraction unit may also be described as a "unit for extracting features from an image to be detected".

[0081] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0082] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A key point detection method, comprising: Perform feature extraction on the image to be detected to obtain image features; Inputting the image features into a pre-trained heat map regression network to obtain a key point heat map; Inputting the output result of the intermediate layer of the heat map regression network into a pre-trained coordinate regression network to obtain a set of key point coordinates; Based on the key point heat map and the key point coordinate set, generating key point position information corresponding to the image to be detected, including: A heat map mapping key point set and a coordinate mapping key point set are obtained through mapping, and the key point selection is realized according to the relationship between the distance between two key points in the key point group and the preset threshold, and the key point position information corresponding to the image to be detected is determined according to the selected key point position information, wherein each key point group includes corresponding heat map mapping key points and coordinate mapping key points.

2. The method according to claim 1, wherein: The generating the key point position information corresponding to the image to be detected based on the key point heat map and the key point coordinate set includes: Mapping the key point heat map and the key point coordinate set to the image to be detected respectively, to obtain a target image including the key point set mapped by the heat map and the key point set mapped by the coordinates; Selecting a key point from each key point group in the target image as a target key point to obtain a target key point set, wherein each key point group includes a corresponding heat map mapping key point and a coordinate mapping key point; The position information of each target key point in the target key point set is determined as the key point position information corresponding to the image to be detected.

3. The method according to claim 2, wherein: The step of selecting a key point from each key point group in the target image as a target key point comprises: Based on the distance between two key points in each key point group, a key point is selected from the key point group as a target key point.

4. The method according to claim 3, wherein: The selecting a key point from each key point group as a target key point based on the distance between two key points in each key point group includes: In response to determining that the distance is less than or equal to a preset threshold, determining the heat map mapping key point in the key point group as a target key point; In response to determining that the distance is greater than the preset threshold, a coordinate mapping key point in the key point group is determined as a target key point.

5. The method according to claim 1, wherein: The heat map regression network includes multiple deconvolution layers; as well as The output result of the intermediate layer of the heat map regression network is input into the coordinate regression network to obtain a set of key point coordinates, including: The output result of the last deconvolution layer among the multiple deconvolution layers is input into the coordinate regression network to obtain a set of key point coordinates.

6. The method according to claim 1, wherein: The heat map regression network and the coordinate regression network are trained by the following steps: The initial heat map regression network is trained with the first learning rate until the convergence condition is met to obtain the intermediate heat map regression network; The intermediate heat map regression network and the initial coordinate regression network are jointly trained at a second learning rate until a training end condition is met, thereby obtaining the heat map regression network and the coordinate regression network, wherein the second learning rate is less than the first learning rate.

7. The method according to claim 2, wherein: Before extracting features from the image to be detected to obtain image features, the method further includes: Performing target part detection on the original image to be detected to obtain an image area showing the target part; Scaling the image area to a target size to obtain the image to be detected; and The generating the key point position information corresponding to the image to be detected based on the key point heat map and the key point coordinate set includes: Based on the key point heat map and the key point coordinate set, the key point position information corresponding to the original image to be detected is generated.

8. The method according to claim 7, wherein: The step of mapping the key point heat map and the key point coordinate set to the image to be detected to obtain a target image including the heat map mapping key point set and the coordinate mapping key point set includes: The key point heat map and the key point coordinate set are respectively mapped to the original image to be detected to obtain a target image containing a heat map mapping key point set and a coordinate mapping key point set.

9. A key point detection device, comprising: An extraction unit is configured to perform feature extraction on the image to be detected to obtain image features; A heat map generation unit is configured to input the image features into a pre-trained heat map regression network to obtain a key point heat map; A coordinate generation unit is configured to input the output result of the intermediate layer of the heat map regression network into a pre-trained coordinate regression network to obtain a set of key point coordinates; A position information generating unit is configured to generate key point position information corresponding to the image to be detected based on the key point heat map and the key point coordinate set, including: A heat map mapping key point set and a coordinate mapping key point set are obtained through mapping, and the key point selection is realized according to the relationship between the distance between two key points in the key point group and the preset threshold, and the key point position information corresponding to the image to be detected is determined according to the selected key point position information, wherein each key point group includes corresponding heat map mapping key points and coordinate mapping key points.

10. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

11. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images