Key point detection methods, devices, electronic equipment, storage media and program products

Abstract

This invention discloses a keypoint detection method, apparatus, electronic device, storage medium, and program product. In this invention, coordinate regression processing is performed on an image to be detected to obtain at least one initial keypoint corresponding to the image. The image to be detected includes an image-carrying object. Based on the initial keypoints, key regions corresponding to each initial keypoint are selected in the image. Feature extraction is performed on each key region to obtain a key feature map corresponding to each key region. Each key feature map is divided into at least one feature sub-map, and feature processing is performed on each feature sub-map to obtain the features of each feature sub-map. Based on the features of each feature sub-map, the target keypoint corresponding to the image-carrying object in each key region is determined. This method saves computational resources required for keypoint detection and reduces the power consumption of electronic devices while ensuring the accuracy and stability of keypoint detection.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to key point detection methods, devices, electronic devices, storage media, and program products. Background Technology

[0002] With the rapid development of science and technology, more and more applications require the integration of key point detection technology. Examples include facial recognition for identity verification and adding special effects to people or animals in short videos.

[0003] Currently, many keypoint detection technologies require the deployment of computationally intensive models or neural networks within devices to achieve high accuracy and stability. However, the computing resources available to users vary significantly between different devices, and mobile terminals generally have limited computing resources. If a computationally intensive model is deployed on a user's device, the device may not be able to provide the necessary computing resources, or even if it can, the power consumption will be high. Summary of the Invention

[0004] The embodiments of the present invention provide a key point detection method, apparatus, electronic device, storage medium and program product, which can save computing resources and power consumption when performing key point detection while ensuring the accuracy and stability of key point detection.

[0005] This invention provides a key point detection method, comprising:

[0006] Coordinate regression processing is performed on the image to be detected to obtain at least one initial key point corresponding to the image to be detected, wherein the image to be detected includes an image carrying object;

[0007] In the image to be detected, key regions corresponding to each initial key point are selected based on the initial key points;

[0008] Feature extraction is performed on each of the key regions to obtain the key feature map corresponding to each key region;

[0009] Each of the key feature maps is divided into at least one feature sub-map, and feature processing is performed on each feature sub-map to obtain the features of each feature sub-map.

[0010] Based on the features of each feature sub-image, the target key points corresponding to the image-carrying object in each key region are determined.

[0011] Accordingly, embodiments of the present invention also provide a key point detection device, comprising:

[0012] A coordinate regression unit is used to perform coordinate regression processing on the image to be detected to obtain at least one initial key point corresponding to the image to be detected, wherein the image to be detected includes an image carrying object.

[0013] A region selection unit is used to select key regions corresponding to each initial key point in the image to be detected, based on the initial key points.

[0014] The feature extraction unit is used to extract features from each of the key regions to obtain key feature maps corresponding to each of the key regions.

[0015] A feature processing unit is configured to divide each of the key feature maps into at least one feature sub-map, perform feature processing on each feature sub-map, and obtain the features of each feature sub-map.

[0016] The key point determination unit is used to determine the target key points corresponding to the image-carrying object in each key region based on the features of each feature sub-image.

[0017] Optionally, the key point detection device provided in this embodiment of the invention further includes an object detection unit, used to perform object type detection on the image-carrying object and obtain the predicted object result corresponding to the image-carrying object;

[0018] Based on the predicted object result, the similarity between the image carrying object and the preset image expected object is calculated;

[0019] When the similarity is greater than a preset similarity threshold, the step of selecting key regions corresponding to each initial key point in the image to be detected is performed.

[0020] Optionally, the feature extraction unit is used to extract features from each of the key regions to obtain a discrete feature map corresponding to each key region as a key feature map, wherein the key feature map includes at least one discrete feature point.

[0021] The feature processing unit is used to divide the key feature map into sub-maps based on the discrete feature points in the key feature map to obtain a discrete feature sub-map corresponding to the key feature map, wherein the discrete feature sub-map includes at least one discrete feature point.

[0022] Based on the discrete feature points, preset interpolation coefficients, and weight coefficients corresponding to the discrete feature points in each discrete feature sub-map, interpolation processing is performed on each discrete feature sub-map to obtain the feature sub-map corresponding to each key feature map.

[0023] Each of the aforementioned feature sub-graphs is subjected to feature processing to obtain the features of each of the aforementioned feature sub-graphs.

[0024] Optionally, the feature processing unit is configured to determine the position of the key sub-region corresponding to each feature sub-image in the image to be detected based on each feature sub-image;

[0025] Based on the location of each key sub-region, the mean value of each feature sub-map corresponding to each key feature map is calculated and used as the feature of each feature sub-map.

[0026] Optionally, the key point determination unit is used to determine at least one candidate key point corresponding to the image-carrying object in each key region based on the features of each feature sub-image;

[0027] The key regions corresponding to each initial key point are subjected to distribution mapping processing to obtain the Gaussian distribution function corresponding to each key region.

[0028] Calculate the confidence level of each candidate key point based on the Gaussian distribution function and the candidate key points.

[0029] If the confidence level is greater than the preset confidence threshold, then the candidate key point is taken as the target key point corresponding to the image carrying object.

[0030] Optionally, the key point detection device provided in this embodiment of the invention further includes a first model acquisition unit for acquiring a coordinate regression model;

[0031] Correspondingly, the coordinate regression unit is used to extract features from the image to be detected using the feature extraction parameters of the coordinate regression model, and generate an initial feature map corresponding to the image to be detected.

[0032] Based on the classification mapping parameters of the coordinate regression model, the initial feature points in the initial feature map are classified to obtain at least one initial key point corresponding to the image to be detected.

[0033] Optionally, the key point detection device provided in this embodiment of the invention further includes a second model acquisition unit, used to acquire a key point determination model;

[0034] Correspondingly, the feature extraction unit is used to determine the feature map extraction parameters of the model through the key points and extract the key feature maps corresponding to each key region.

[0035] The feature processing unit is used to determine the graph cutting parameters of the model based on the key points, divide each key feature map, and obtain at least one feature sub-map corresponding to each key feature map.

[0036] The feature processing parameters of the model are determined by the key points, and feature processing is performed on each of the feature sub-graphs to generate the features of each feature sub-graph.

[0037] The key point determination unit is used to determine the target key points corresponding to the image-carrying object in each key region based on the features of each feature sub-image and the key point judgment parameters of the key point determination model.

[0038] Optionally, the key point detection device provided in this embodiment of the invention further includes a model training unit, which includes a model acquisition unit, a first model processing unit, a training region selection unit, a second model processing unit, a joint loss calculation unit, and a model adjustment unit.

[0039] The model acquisition unit is used to acquire the coordinate regression model to be trained, the key point determination model to be trained, and the sample image, wherein the sample image is labeled with actual key points.

[0040] The first model processing unit is used to extract initial key points from the image to be detected using the coordinate regression model to be trained, so as to obtain at least one initial training key point corresponding to the sample image.

[0041] The training region selection unit is used to select, in the sample image, a training key region corresponding to each of the initial training key points, based on the initial training key points.

[0042] The second model processing unit is used to extract target key points from each of the training key regions using the key point determination model to be trained, and to determine the training target key points corresponding to the image to be detected from each of the training key regions.

[0043] The joint loss calculation unit is used to calculate the joint loss of the coordinate regression model to be trained and the key point determination model to be trained based on the actual key points, the initial training key points and the target training key points;

[0044] The model adjustment unit is used to adjust the coordinate regression model to be trained and the key point determination model to be trained according to the joint loss, so as to obtain the trained coordinate regression model and the trained key point determination model.

[0045] Optionally, the coordinate regression model to be trained includes an object prediction sub-model to be trained, and the sample image is labeled with the actual similarity between the sample image carrying the object and the preset expected object of the sample image.

[0046] The key point detection device provided in this embodiment of the invention further includes a first sub-model unit, which is used to predict the object carried by the sample image in the sample image through the object prediction sub-model to be trained, and to obtain the training similarity between the object carried by the sample image and the preset expected object of the sample image.

[0047] The joint loss calculation unit is used to calculate the first key point prediction loss based on the actual key points and the initial training key points.

[0048] Calculate the object prediction loss based on the training similarity and the actual similarity;

[0049] Based on the first keypoint prediction loss and the object prediction loss, the first loss corresponding to the coordinate regression model to be trained is obtained;

[0050] Based on the actual key points and the training target key points, calculate the second loss corresponding to the model based on the key points to be trained;

[0051] Based on the first loss and the second loss, calculate the joint loss of the coordinate regression model to be trained and the keypoint determination model to be trained.

[0052] Optionally, the key point determination model to be trained includes a confidence prediction sub-model to be trained, the sample image is labeled with the actual confidence level corresponding to each key feature point, and the key feature points are obtained by extracting key points from the feature sub-map corresponding to the sample image;

[0053] The key point detection model provided in this embodiment of the invention further includes a second sub-model unit, which is used to calculate the confidence of the training target key points through the confidence prediction sub-model to be trained, and obtain the training confidence corresponding to each training target key point;

[0054] The joint loss calculation unit is used to calculate the first loss corresponding to the coordinate regression model based on the actual key points and the initial training key points;

[0055] Based on the actual key points and the training target key points, calculate the second key point prediction loss corresponding to the key point determination model to be trained;

[0056] Based on the training confidence and the actual confidence corresponding to each training target key point, calculate the confidence prediction loss corresponding to the key point determination model to be trained;

[0057] Based on the second keypoint prediction loss and the confidence prediction loss, the second loss corresponding to the keypoint determination model to be trained is obtained;

[0058] Based on the first loss and the second loss, calculate the joint loss of the coordinate regression model to be trained and the keypoint determination model to be trained.

[0059] Optionally, the image to be detected is a video frame in the video to be detected, the image carrier is the video carrier of the video to be detected, and the video to be detected contains at least two of the video frames;

[0060] The key point detection model provided in this embodiment of the invention further includes a pose tracking unit, which is used to obtain a video frame adjacent to the image to be detected from the video to be detected as a new image to be detected, return to execute the step of performing coordinate regression processing on the image to be detected to obtain at least one initial key point corresponding to the image to be detected, until the target key point corresponding to the image carrying object in the new image to be detected is determined.

[0061] Based on the target key points corresponding to the image-carrying object in the previous image to be detected and the target key points corresponding to the image-carrying object in the new image to be detected, the pose change of the video-carrying object in the video to be detected is determined.

[0062] Optionally, the feature extraction unit is used to divide each of the key regions into sub-regions to obtain a set of sub-regions corresponding to each key region;

[0063] Feature extraction is performed on the sub-regions in each of the sub-region sets to obtain the sub-region feature maps corresponding to each sub-region;

[0064] Based on the sub-region feature maps, determine the key region feature maps for each of the key regions;

[0065] Based on the feature maps of each key region, the region association weights corresponding to each sub-region in each set of sub-regions are determined. The region association weights are used to indicate the association relationships between the sub-regions in the set of sub-regions corresponding to each key region.

[0066] Based on the regional association weights, the feature maps of each key region are weighted and calculated to obtain the discrete feature maps corresponding to each key region.

[0067] Accordingly, embodiments of the present invention also provide an electronic device, including a memory and a processor; the memory stores an application program, and the processor is used to run the application program in the memory to perform the steps in any of the key point detection methods provided in the embodiments of the present invention.

[0068] Accordingly, embodiments of the present invention also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the key point detection methods provided in the embodiments of the present invention.

[0069] Furthermore, embodiments of the present invention also provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in any of the key point detection methods provided in embodiments of the present invention.

[0070] The scheme of this invention can perform coordinate regression processing on the image to be detected to obtain at least one initial key point corresponding to the image to be detected. The image to be detected includes an image carrying object. In the image to be detected, key regions corresponding to each initial key point are selected according to the initial key points. Features are extracted from each key region to obtain key feature maps corresponding to each key region. Each key feature map is divided into at least one feature sub-map. Feature processing is performed on each feature sub-map to obtain the features of each feature sub-map. Based on the features of each feature sub-map, the target key points corresponding to the image carrying object in each key region are determined. Since this invention first obtains initial key points that can guarantee the shape of the image carrying object through coordinate regression processing, key points are reselected more accurately from small regions based on the initial key points. Moreover, the calculation method of coordinate regression can save a lot of computation. Therefore, while ensuring the accuracy and stability of key point detection, the computational resources required for key point detection are saved, and the power consumption of electronic devices is reduced. Attached Figure Description

[0071] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1 This is a schematic diagram of a scenario for the key point detection method provided in an embodiment of the present invention;

[0073] Figure 2 This is a flowchart of the key point detection method provided in the embodiments of the present invention;

[0074] Figure 3 This is a schematic diagram of the key point detection model provided in an embodiment of the present invention;

[0075] Figure 4 This is another flowchart of the key point detection method provided in the embodiments of the present invention;

[0076] Figure 5 This is a schematic diagram of the key point detection device provided in an embodiment of the present invention;

[0077] Figure 6 This is another structural schematic diagram of the key point detection device provided in the embodiment of the present invention;

[0078] Figure 7 This is a schematic diagram of the structure of the model training unit provided in an embodiment of the present invention;

[0079] Figure 8 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0080] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0081] This invention provides a keypoint detection method, apparatus, electronic device, and computer-readable storage medium. Specifically, this invention provides a keypoint detection method applicable to a keypoint detection apparatus, which can be integrated into an electronic device.

[0082] The electronic device can be a terminal or other device, including but not limited to mobile terminals and fixed terminals. For example, mobile terminals include but are not limited to smartphones, smartwatches, tablets, laptops, smart vehicles, etc., while fixed terminals include but are not limited to desktop computers, smart TVs, etc.

[0083] The electronic device can also be a server or other similar device. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, but it is not limited to these.

[0084] The key point detection method of this invention can be implemented by a server or by a terminal and a server together.

[0085] The following example illustrates the keypoint detection method implemented jointly by a terminal and a server.

[0086] like Figure 1 As shown, the key point detection system provided in this embodiment of the invention includes a terminal 10 and a server 20, etc.; the terminal 10 and the server 20 are connected through a network, such as through a wired or wireless network, etc., wherein the terminal 10 can exist as a terminal for a user to send an image to be detected to the server 20.

[0087] Terminal 10 can be a terminal for users to upload images to be detected, and is used to send the images to be detected to server 20.

[0088] Server 20 can be used to perform coordinate regression processing on the image to be detected to obtain at least one initial key point corresponding to the image to be detected. The image to be detected includes an image carrying object. In the image to be detected, key regions corresponding to each initial key point are selected according to the initial key points. Features are extracted from each key region to obtain key feature maps corresponding to each key region. Each key feature map is divided into at least one feature sub-map. Feature processing is performed on each feature sub-map to obtain the features of each feature sub-map. Based on the features of each feature sub-map, the target key points corresponding to the image carrying object in each key region are determined.

[0089] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0090] The embodiments of the present invention will be described from the perspective of a key point detection device, which can be integrated into a server or terminal.

[0091] like Figure 2 As shown, the specific process of the key point detection method in this embodiment can be as follows:

[0092] 201. Perform coordinate regression processing on the image to be detected to obtain at least one initial key point corresponding to the image to be detected, wherein the image to be detected includes the image carrying object.

[0093] In this embodiment of the invention, key points are important information points related to the analysis of image-carrying objects in the image to be detected. They can express the combination or contextual relationship of image information within a certain neighborhood range in the image to be detected. For example, key points can be points where the image-carrying object expresses certain features, or they can be edge points that distinguish the image-carrying object from other areas in the image to be detected, or they can be points or blocks that can represent the image-carrying object in the same or similar form in other similar images containing the image-carrying object, and so on.

[0094] Understandably, each keypoint can be a location in a specific image to be detected, or it can be a region consisting of at least one point, and so on.

[0095] The image to be detected is the image for which key point detection is required, and the image carrying the key point can have one or more objects. Specifically, the image carrying the key point can be a person, animal, plant, etc., and this embodiment of the invention does not limit this.

[0096] In practical applications, to reduce the demand for computing resources, coordinate regression can be used to directly process the image to be detected. There are various implementation methods; for example, the step of "performing coordinate regression on the image to be detected to obtain at least one initial key point corresponding to the image" can include:

[0097] Image features are extracted from the image to be detected to obtain an initial feature distribution of the image to be detected, which includes at least one initial feature point.

[0098] Based on the initial feature distribution, the initial feature points are clustered to obtain at least one cluster center and the cluster range corresponding to the cluster center;

[0099] Based on the cluster centers and the corresponding cluster ranges, at least one initial key point corresponding to the image to be detected is determined.

[0100] For example, when using Faster RCNN (Regions with CNN features) for coordinate regression processing, we can first use a set of feature extraction layers to extract the initial feature distribution (feature map) of the image to be detected. Then, based on the preset hyperparameter anchor, we can determine at least one anchor region center and region range (cluster center and the cluster range corresponding to the cluster center) in the initial feature distribution. Then, we can perform linear regression calculation on the anchor region to obtain the initial key points.

[0101] Alternatively, the YOLO algorithm can be used for coordinate regression processing to extract the initial feature distribution of the image to be detected. The k-means algorithm is then used to cluster the feature points in the initial feature distribution. Based on the clustering results, the hyperparameter anchor is determined, and the clustering range is determined based on the hyperparameter anchor. Linear regression is then performed on the clustering range to obtain the initial key points.

[0102] Among them, since the coordinate regression processing method mainly focuses on the positioning accuracy of key points, it can well ensure the shape of the image carrying object corresponding to the detected key points (i.e., shape preservation), avoiding the situation of poor key point positioning or the occurrence of points that do not preserve shape.

[0103] In some optional embodiments, coordinate regression can be implemented using a neural network or machine learning model, and may include the following before step 201:

[0104] Obtain the coordinate regression model;

[0105] Correspondingly, the step "perform coordinate regression processing on the image to be detected to obtain at least one initial key point corresponding to the image to be detected" may include:

[0106] The feature extraction parameters of the coordinate regression model are used to extract features from the image to be detected, generating an initial feature map corresponding to the image to be detected.

[0107] Based on the classification mapping parameters of the coordinate regression model, the initial feature points in the initial feature map are classified to obtain at least one initial key point corresponding to the image to be detected.

[0108] In this context, a feature map, also known as a feature image, refers to a set of feature points in an image to be detected. The number of initial feature maps can be related to the number of feature extraction layers in the coordinate regression model. For example, three feature extraction layers can generate three initial feature maps. Alternatively, feature fusion processing can be performed on the outputs of the three feature extraction layers to obtain an initial feature map corresponding to the entire feature extraction layer, and so on.

[0109] Feature extraction parameters may include, but are not limited to: the number of feature extraction layers in the coordinate regression model, the size of the convolutional kernel, the number of output channels of the convolutional layer, etc. Classification mapping parameters may include, but are not limited to: the number of categories, the mapping calculation formula, etc.

[0110] Understandably, in order to ensure the accuracy of the coordinate regression model, it is generally trained before its formal application. For details of the training process, please refer to the following examples.

[0111] 202. In the image to be detected, select the key regions corresponding to each initial key point based on the initial key points.

[0112] Specifically, key regions corresponding to each initial key point can be selected from the image to be detected based on the region selection parameters. Step 202 includes:

[0113] In the image to be detected, determine the location of each initial key point;

[0114] Based on the location of each initial keypoint and the region selection parameters, select the key region corresponding to each initial keypoint.

[0115] The region selection parameter can be a specific numerical value. For example, the region selection parameter can be 16, which means selecting a 16×16 square region with the initial key point as the center of the region; or, the region selection parameter can be an array. For example, the region selection parameter can be (16, 20), which means selecting a 16×20 rectangular region with the initial key point as the top left corner of the region, and so on.

[0116] The region selection parameters can be manually set by technicians or obtained through pre-training to achieve the best key region selection effect. This embodiment of the invention does not limit this.

[0117] In related technologies, coordinate regression methods primarily focus on the accuracy of keypoint localization, regardless of whether the object in the input image is the object to be detected. In other words, coordinate regression methods cannot provide confidence levels for the current input, and the output keypoints may not match the expected object in the actual application.

[0118] Therefore, object detection can be performed on the image carrying object in the image to be detected to determine whether the image carrying object matches the expected object. That is, before the step "selecting the key regions corresponding to each initial key point in the image to be detected", the embodiments of the present invention may further include a step of judging the image carrying object:

[0119] Perform object type detection on the image carrying the object to obtain the predicted object result corresponding to the image carrying the object;

[0120] Based on the predicted object results, the similarity between the image carrying object and the preset image expected object is calculated;

[0121] When the similarity is greater than the preset similarity threshold, the step of selecting the key regions corresponding to each initial key point in the image to be detected is performed.

[0122] The predicted object result can be the object type to which the image carries the object, such as the image carrying the object belonging to the first type (cat) or the second type (dog), or the predicted object result can be the contour data of the image carrying the object, and so on.

[0123] Correspondingly, when the predicted object result is an object type, the object type can be compared with the preset image expected object type. If the types are the same, the similarity is 1; if the types are different, the similarity is 0. Alternatively, the type features of the object type (e.g., the overall features of a dog) can be compared with the target type features of the image expected object (e.g., the overall features of a cat) to obtain the similarity between the type features and the target type features, which is used as the similarity between the image carrying object and the image expected object.

[0124] Alternatively, when the predicted object is the contour data of the image carrying object, the similarity between this contour data and the expected contour data of the image carrying object can be calculated to obtain the similarity between the image carrying object and the image expecting object.

[0125] It is understandable that in actual images to be detected, there may be situations where the objects in the image are partially occluded. In this case, the objects in the image can be divided into parts, and the similarity of the divided parts can be compared with the corresponding parts in the expected objects in the image.

[0126] In practical applications, the step of determining the object carried by the image can be performed simultaneously with the coordinate regression processing of the image to be detected, or the order of execution of the two can be unrestricted.

[0127] In some alternative embodiments, such as Figure 3 As shown, the step of determining the object carried by an image can be implemented using a neural network or machine learning model. Therefore, the coordinate regression model to be trained can include an object prediction sub-model to be trained, where the sample image is labeled with the actual key points in the sample image and the actual similarity between the object carried by the sample image and the preset expected object in the sample image;

[0128] The steps for determining the object carried by an image using an object prediction sub-model may specifically include:

[0129] The object prediction sub-model predicts the image carrying object in the image to be detected, and obtains the similarity between the image carrying object and the preset expected image object.

[0130] During training, the object prediction sub-model to be trained can be used to predict the object carried by the sample image in the sample image, and the training similarity between the object carried by the sample image and the preset expected object of the sample image can be obtained.

[0131] Based on this, the loss of the coordinate regression model to be trained can be calculated through the following steps:

[0132] Calculate the prediction loss for the first key point based on the actual key points and the initial key points during training;

[0133] Calculate the object prediction loss based on the training similarity and the actual similarity;

[0134] Based on the first keypoint prediction loss and the object prediction loss, the first loss corresponding to the coordinate regression model to be trained is obtained.

[0135] In addition to the object prediction sub-model to be trained, the coordinate regression model to be trained may also include a coordinate extraction sub-model to be trained. The initial key points for training can be obtained through the coordinate extraction sub-model to be trained.

[0136] Specifically, the calculation process for the first loss can be expressed by the following formula:

[0137]

[0138] Where loss1 represents the first loss, For the predicted initial keypoint, For the i-th labeled actual key point, Let be the actual similarity of the i-th part of the image carrying object. Let be the training similarity of the i-th part of the image carrying object.

[0139] It is understandable that after obtaining the first loss (the loss of the coordinate regression model to be trained), the coordinate regression model to be trained can be adjusted directly. Alternatively, after obtaining the second loss of the keypoint determination model to be trained, the coordinate regression model to be trained and the keypoint determination model to be trained can be jointly adjusted based on the first and second losses.

[0140] 203. Extract features from each key region to obtain the key feature map corresponding to each key region.

[0141] In practical applications, features obtained from image feature extraction are... Figure 1 Generally, it is composed of discrete feature points, and step 203 may specifically include:

[0142] Feature extraction is performed on each key region to obtain a discrete feature map corresponding to each key region as a key feature map. The key feature map includes at least one discrete feature point.

[0143] However, when performing feature processing directly on each sub-map after dividing the discrete feature map, quantization errors can occur during rounding operations because the sub-map positions may contain floating-point numbers. Therefore, the discrete feature map can be converted into a continuous feature map to solve the problem of accumulated errors caused by quantization operations.

[0144] Correspondingly, the step "dividing each key feature map into at least one feature sub-map, performing feature processing on each feature sub-map to obtain the features of each feature sub-map" includes:

[0145] Based on the discrete feature points in the key feature map, the key feature map is divided into sub-maps to obtain discrete feature sub-maps corresponding to the key feature map. Each discrete feature sub-map includes at least one discrete feature point.

[0146] Based on the discrete feature points in each discrete feature sub-map, the preset interpolation coefficients, and the weight coefficients corresponding to the discrete feature points, interpolation processing is performed on each discrete feature sub-map to obtain the feature sub-maps corresponding to each key feature map.

[0147] Feature processing is performed on each feature sub-image to obtain the features of each feature sub-image.

[0148] In some optional examples, feature processing can be performed directly on the feature submap after obtaining it, or the feature submap can be further divided into regions after obtaining continuous feature submaps, and feature processing can be performed on the smaller regions, and so on.

[0149] The interpolation process for discrete feature submaps can be expressed by the following formula:

[0150]

[0151] Specifically, IC(x,y,i,j) are the interpolation coefficients, IC(x,y,i,j) = max(0,1-|xi|)×max(0,1-|yj|), f(x,y) is the continuous feature map, and w(i,j) is the discrete feature map. This bilinear interpolation method allows for surface quantization, and the resulting feature map can be considered continuous.

[0152] In some optional embodiments, multi-head attention networks or self-attention networks can be used to extract features from each key region to obtain the region association weights of each sub-region corresponding to each key region. For example, the specific extraction process can be to convert the sub-regions of the key region into sub-region feature maps, and then use the sub-region feature maps as the region association weights of each sub-region in the sub-region set.

[0153] That is, the step "extracting features from each key region to obtain discrete feature maps corresponding to each key region as key feature maps" can include:

[0154] Each key region is divided into sub-regions, resulting in a set of sub-regions corresponding to each key region.

[0155] Feature extraction is performed on the sub-regions in each set of sub-regions to obtain the sub-region feature map corresponding to each sub-region;

[0156] Based on the feature maps of the sub-regions, determine the key region feature maps for each key region;

[0157] Based on the feature maps of each key region, the region association weights corresponding to each sub-region in each sub-region set are determined. The region association weights are used to indicate the association relationships between the sub-regions in the sub-region set corresponding to each key region.

[0158] Based on the regional association weight, the feature maps of each key region are weighted and calculated to obtain the discrete feature maps corresponding to each key region.

[0159] In this context, a sub-region is a region obtained by dividing the key region. The specific rules for dividing the region can be set by technical personnel according to the actual application, such as dividing the key region with a fixed area, or dividing the key region into a fixed number of sub-regions, etc. This embodiment of the invention does not limit this.

[0160] Among them, the key region feature map can be obtained by directly concatenating vectors from the sub-region feature maps, or by performing vector processing such as weighted calculation on the sub-region feature maps.

[0161] For example, a multi-head attention network can be used to convert the key region feature map into a query vector (q), a key vector (k), and a value vector (v). Specifically, a self-attention network can be used to fuse the key region feature map with transformation parameters of multiple dimensions to obtain the query vector (q), the key vector (k), and the value vector (v). The query vector (q), the key vector (k), and the value vector (v) can then be used as the region association weights of the sub-regions in the sub-region set.

[0162] 204. Divide each key feature map into at least one feature sub-map, perform feature processing on each feature sub-map, and obtain the features of each feature sub-map.

[0163] Specifically, the total eigenvalues ​​and the area of ​​the corresponding regions in each feature sub-graph can be calculated as the features of each feature sub-graph. That is, the step "perform feature processing on each feature sub-graph to obtain the features of each feature sub-graph" can include:

[0164] Based on each feature sub-image, determine the location of the key sub-region corresponding to each feature sub-image in the image to be detected;

[0165] Based on the location of each key sub-region, the mean value of each feature sub-map corresponding to each key feature map is processed, and the mean value of each feature sub-map is calculated as the feature of each feature sub-map.

[0166] The process of averaging can be represented by the following formula:

[0167] Feature =

[0168] The numerator represents the integration of eigenvalues ​​of continuous feature subgraphs, where (x1, y1) and (x2, y2) are the top-left and bottom-right boundary points of the regions corresponding to the feature subgraphs, respectively.

[0169] In some optional examples, the determination of target key points can be achieved through neural networks or their learning models. Before the step "extracting features from each key region to obtain the key feature map corresponding to each key region", the method also includes:

[0170] Key points are obtained to determine the model;

[0171] Correspondingly, steps 203, 204, and 205 can be implemented using key points to determine the model, specifically including:

[0172] The feature map extraction parameters of the model are determined by key points, and the key feature maps corresponding to each key region are extracted.

[0173] Based on the key points, the graph cutting parameters of the model are determined, and each key feature map is divided to obtain at least one feature sub-map corresponding to each key feature map.

[0174] The feature processing parameters of the model are determined by key points, and feature processing is performed on each feature sub-map to generate the features of each feature sub-map.

[0175] Based on the features of each feature sub-image and the key points, the key point judgment parameters of the model are determined, and the target key points corresponding to the image-carrying objects in each key region are determined.

[0176] The feature map extraction parameters may include, but are not limited to: the number of feature extraction layers in the keypoint determination model, the size of the convolutional kernel, the number of output channels of the convolutional layer, etc. The graph cutting parameters may include, but are not limited to, the number of cuts, the cut size, etc. The feature processing parameters may include, but are not limited to, the mean calculation formula, the weights for calculating the mean of different regions, etc.

[0177] It is understandable that, in order to ensure the accuracy of the key point determination model, the key point determination model is usually trained before it is formally applied. For the specific training process, please refer to other embodiments.

[0178] 205. Based on the features of each feature sub-image, determine the target key points corresponding to the image-carrying objects in each key region.

[0179] In some embodiments, to ensure the reliability of the final target key points, a confidence check can be performed on the candidate key points extracted from each feature sub-map before determining the target key points. Therefore, step 205 may specifically include:

[0180] Based on the features of each feature sub-image, at least one candidate key point corresponding to the image-carrying object in each key region is determined.

[0181] The key regions corresponding to each initial key point are processed by distribution mapping to obtain the Gaussian distribution function corresponding to each key region.

[0182] Calculate the confidence level of each candidate key point based on each Gaussian distribution function and each candidate key point;

[0183] If the confidence level is greater than the preset confidence threshold, the candidate key point will be used as the target key point corresponding to the image carrying object.

[0184] Specifically, the distribution mapping process can be used to process the key region into a one-dimensional Gaussian response map (Gaussian distribution function). Based on the Gaussian distribution function, the confidence level of each point within the key region can be simulated.

[0185] In some optional examples, the confidence verification step can be implemented using a neural network or machine learning model. Therefore, the keypoint determination model to be trained includes a confidence prediction sub-model to be trained, sample images labeled with actual keypoints and the actual confidence scores corresponding to each key feature point, and key feature points are obtained by extracting keypoints from the feature sub-maps corresponding to the sample images;

[0186] The confidence verification process may specifically include the following steps:

[0187] The confidence of the training target key points is calculated by using the confidence prediction sub-model to be trained, and the training confidence of each training target key point is obtained.

[0188] Based on this, the calculation process of the loss of the model is determined by the key points to be trained, which may specifically include the following steps:

[0189] Based on the actual key points and the training target key points, calculate the key points to be trained and determine the second key point prediction loss corresponding to the model.

[0190] Based on the training confidence and actual confidence corresponding to each training target key point, calculate the confidence prediction loss corresponding to the model for the key points to be trained;

[0191] Based on the second keypoint prediction loss and the confidence prediction loss, the second loss corresponding to the keypoint determination model to be trained is obtained.

[0192] In addition to the object prediction sub-model to be trained, the coordinate regression model to be trained may also include a coordinate extraction sub-model to be trained. The initial key points for training can be obtained through the coordinate extraction sub-model to be trained.

[0193] Specifically, the calculation process for the second loss can be expressed by the following formula:

[0194]

[0195] Where loss2 represents the second loss, For the predicted initial keypoint, For the i-th labeled actual key point, This represents the actual confidence level corresponding to the i-th key feature point in the image to be detected. Let be the training confidence level corresponding to the i-th training target feature point in the image to be detected.

[0196] It is understandable that after obtaining the first loss (the loss of the coordinate regression model to be trained), the key point determination model to be trained can be adjusted directly. Alternatively, after obtaining the first loss of the coordinate regression model to be trained, the coordinate regression model to be trained and the key point determination model to be trained can be jointly adjusted based on the first loss and the second loss.

[0197] In some optional examples, the step of jointly training the coordinate regression model and the keypoint determination model to be trained may specifically include:

[0198] Obtain the coordinate regression model to be trained, the key point determination model to be trained, and the sample images, which are labeled with actual key points;

[0199] The coordinate regression model to be trained is used to extract the initial key points of the image to be detected, so as to obtain at least one initial training key point corresponding to the sample image.

[0200] In the sample image, the training key region corresponding to each initial training key point is selected based on the initial training key points;

[0201] The key point determination model under training extracts target key points from each training key region, and determines the training target key points corresponding to the image to be detected from each training key region.

[0202] Based on the actual key points, the initial key points of training, and the target key points of training, calculate the joint loss of the coordinate regression model to be trained and the key point determination model to be trained.

[0203] Based on the joint loss, the coordinate regression model and the keypoint determination model to be trained are adjusted to obtain the trained coordinate regression model and the trained keypoint determination model.

[0204] The joint loss can be obtained by weighting the first loss and the second loss, or by directly adding the first loss and the second loss, and so on.

[0205] In some optional examples, the image to be detected is a video frame in the video to be detected, the image carrier is the video carrier of the video to be detected, and the video to be detected contains at least two video frames;

[0206] Following the step of "determining the target key points corresponding to the image-carrying object in each key region based on the features of each feature sub-image", the embodiments of the present invention may further include:

[0207] The process involves obtaining video frames adjacent to the image to be detected from the video to be detected as new images to be detected, returning to perform coordinate regression processing on the image to be detected to obtain at least one initial key point corresponding to the image to be detected, until the target key point corresponding to the image-carrying object in the new image to be detected is determined.

[0208] Based on the target key points corresponding to the image-carrying object in the previous image to be detected and the target key points corresponding to the image-carrying object in the new image to be detected, the pose change of the video-carrying object in the video to be detected is determined.

[0209] In practical applications, this embodiment can be well embedded into the detection and tracking framework, enabling key point detection and pose tracking functions with very little computing power, further improving the user experience.

[0210] As can be seen from the above, the scheme of the present invention can perform coordinate regression processing on the image to be detected to obtain at least one initial key point corresponding to the image to be detected. The image to be detected includes an image carrying object. In the image to be detected, key regions corresponding to each initial key point are selected according to the initial key points. Features are extracted from each key region to obtain key feature maps corresponding to each key region. Each key feature map is divided into at least one feature sub-map. Feature processing is performed on each feature sub-map to obtain the features of each feature sub-map. Based on the features of each feature sub-map, the target key points corresponding to the image carrying object in each key region are determined. Since the present invention first obtains initial key points that can guarantee the shape of the image carrying object through coordinate regression processing, and then reselects key points from small regions more accurately based on the initial key points, and the calculation method of coordinate regression can save a lot of computation, the computational resources required for key point detection are saved and the power consumption of electronic devices is reduced while ensuring the accuracy and stability of key point detection.

[0211] Based on the methods described in the preceding embodiments, the following examples will provide further detailed explanations.

[0212] In this embodiment, the combination Figure 1 The system will be explained.

[0213] like Figure 4As shown, the key point detection method in this embodiment can be described in the following specific process:

[0214] 401. The server acquires the image to be detected, performs coordinate regression processing on the image to be detected, and obtains at least one initial key point corresponding to the image to be detected.

[0215] There are several ways to obtain the image to be detected. For example, the image to be detected can be directly received from the user's uploaded video through the terminal, or at least one video frame can be extracted from the video uploaded by the user through the terminal as the image to be detected, or a key point detection request sent by the terminal can be received, which carries the storage address of the image to be detected. Based on the storage address, the image to be detected can be obtained from the terminal's memory, cache, or third-party database, and so on.

[0216] In some optional examples, the server can perform preliminary cropping on the acquired image, retaining only the portion of the image whose object area exceeds a preset area threshold, as the image to be detected.

[0217] 402. The server extracts initial key points from the image to be detected using a coordinate regression model, thereby obtaining at least one initial key point corresponding to the image to be detected.

[0218] In practical applications, coordinate regression models can include coordinate extraction sub-models and object prediction sub-models. The coordinate extraction sub-model can consist of at least one convolutional layer, outputting the coordinates of key points in the image to be detected.

[0219] 403. The server uses the object prediction sub-model to predict the image carrying object in the image to be detected, and obtains the similarity between the image carrying object and the preset image expected object.

[0220] The object prediction sub-model can output a score (similarity) for the object carried by the image, that is, to determine whether the current input is a qualified input, such as whether the object carried by the image is a cat or a dog, and give a score value based on the corresponding detection result.

[0221] 404. When the similarity is greater than the preset similarity threshold, the server selects the key regions corresponding to each initial key point in the image to be detected based on the initial key points.

[0222] The similarity threshold can be preset by technicians, such as a score or similarity. For example, the similarity threshold could be 99 (score) or 95% (similarity), etc.

[0223] Specifically, key regions corresponding to each initial key point can be selected from the image to be detected based on the region selection parameters. Step 202 includes:

[0224] In the image to be detected, determine the location of each initial key point;

[0225] Based on the location of each initial keypoint and the region selection parameters, select the key region corresponding to each initial keypoint.

[0226] The region selection parameter can be a specific numerical value. For example, the region selection parameter can be 16, which means selecting a 16×16 square region with the initial key point as the center of the region; or, the region selection parameter can be an array. For example, the region selection parameter can be (16, 20), which means selecting a 16×20 rectangular region with the initial key point as the top left corner of the region, and so on.

[0227] 405. The server determines the feature map extraction parameters of the model through key points and extracts the key feature maps corresponding to each key region.

[0228] In practical applications, features obtained from image feature extraction are... Figure 1 It is generally composed of discrete feature points, and the key feature map includes at least one discrete feature point.

[0229] For example, the server can use a key point determination model to extract features from key regions in multiple dimensions, obtain local image features in multiple dimensions, and then fuse the local image features to obtain the global image features of the key regions as key feature maps.

[0230] 406. The server determines the graph cutting parameters of the model based on the key points, divides each key feature map, and obtains at least one feature sub-map corresponding to each key feature map.

[0231] Image cutting parameters may include, but are not limited to, the number of cuts, the cutting size, etc.

[0232] It is understandable that, in order to ensure the accuracy of the key point determination model, the key point determination model is usually trained before it is formally applied. For the specific training process, please refer to other embodiments.

[0233] 407. The server determines the feature processing parameters of the model through key points, performs feature processing on each feature sub-graph, and generates the features of each feature sub-graph.

[0234] Specifically, based on the location of each key sub-region, the mean value of each feature sub-map corresponding to each key feature map can be processed to calculate the feature mean value of each feature sub-map as the feature of each feature sub-map.

[0235] 408. The server determines the key point judgment parameters of the model based on the features of each feature sub-image and the key points, and determines the candidate key points corresponding to the image carrying objects in each key region.

[0236] Specifically, the feature sub-map with the largest feature value can be used as a candidate key point.

[0237] Understandably, each keypoint can be a location in a specific image to be detected, or it can be a region consisting of at least one point, and so on.

[0238] Key points can be represented by coordinates or sets of coordinates in the image.

[0239] 409. The server calculates the confidence level of each candidate key point through the confidence prediction sub-model. If the confidence level is greater than the preset confidence threshold, the candidate key point is taken as the target key point corresponding to the image carrying object.

[0240] Specifically, the key regions corresponding to each initial key point can be distributed and mapped to obtain the Gaussian distribution function corresponding to each key region.

[0241] Calculate the confidence level of each candidate key point based on the Gaussian distribution function and each candidate key point.

[0242] In some embodiments, in order to ensure the reliability of the final target key points, the confidence level of the candidate key points extracted from each feature sub-map can be verified before determining the target key points.

[0243] As can be seen from the above, the embodiments of the present invention save the computing resources required for key point detection and reduce the power consumption of electronic devices while ensuring the accuracy and stability of key point detection.

[0244] To better implement the above methods, this embodiment of the invention also provides a key point detection device.

[0245] refer to Figure 5 The device includes:

[0246] The coordinate regression unit 501 can be used to perform coordinate regression processing on the image to be detected to obtain at least one initial key point corresponding to the image to be detected. The image to be detected may include an image carrying object.

[0247] The region selection unit 502 can be used to select key regions corresponding to each initial key point in the image to be detected based on the initial key points.

[0248] The feature extraction unit 503 can be used to extract features from each key region to obtain the key feature map corresponding to each key region.

[0249] The feature processing unit 504 can be used to divide each key feature map into at least one feature sub-map, perform feature processing on each feature sub-map, and obtain the features of each feature sub-map.

[0250] The key point determination unit 505 can be used to determine the target key points corresponding to the image-carrying objects in each key region based on the features of each feature sub-image.

[0251] Optionally, the key point detection device provided in this embodiment of the invention may further include an object detection unit 506, which can be used to perform object type detection on the image-carrying object and obtain the predicted object result corresponding to the image-carrying object;

[0252] Based on the predicted object results, the similarity between the image carrying object and the preset image expected object is calculated;

[0253] When the similarity is greater than the preset similarity threshold, the step of selecting the key regions corresponding to each initial key point in the image to be detected is performed.

[0254] Optionally, the feature extraction unit 503 can be used to extract features from each key region to obtain a discrete feature map corresponding to each key region as a key feature map. The key feature map may include at least one discrete feature point.

[0255] The feature processing unit 504 can be used to divide the key feature map into sub-maps based on the discrete feature points in the key feature map, and obtain the discrete feature sub-maps corresponding to the key feature map. The discrete feature sub-maps may include at least one discrete feature point.

[0256] Based on the discrete feature points in each discrete feature sub-map, the preset interpolation coefficients, and the weight coefficients corresponding to the discrete feature points, interpolation processing is performed on each discrete feature sub-map to obtain the feature sub-maps corresponding to each key feature map.

[0257] Feature processing is performed on each feature sub-image to obtain the features of each feature sub-image.

[0258] Optionally, the feature processing unit 504 can be used to determine the position of the key sub-region corresponding to each feature sub-image in the image to be detected based on each feature sub-image;

[0259] Based on the location of each key sub-region, the mean value of each feature sub-map corresponding to each key feature map is processed, and the mean value of each feature sub-map is calculated as the feature of each feature sub-map.

[0260] Optionally, the key point determination unit 505 can be used to determine at least one candidate key point corresponding to the image-carrying object in each key region based on the features of each feature sub-image.

[0261] The key regions corresponding to each initial key point are processed by distribution mapping to obtain the Gaussian distribution function corresponding to each key region.

[0262] Calculate the confidence level of each candidate key point based on each Gaussian distribution function and each candidate key point;

[0263] If the confidence level is greater than the preset confidence threshold, the candidate key point will be used as the target key point corresponding to the image carrying object.

[0264] Optionally, the key point detection device provided in the embodiments of the present invention, such as... Figure 6 As shown, it may also include a first model acquisition unit 507, which can be used to acquire a coordinate regression model;

[0265] Correspondingly, the coordinate regression unit 501 can be used to extract features from the image to be detected using the feature extraction parameters of the coordinate regression model, and generate the initial feature map corresponding to the image to be detected.

[0266] Based on the classification mapping parameters of the coordinate regression model, the initial feature points in the initial feature map are classified to obtain at least one initial key point corresponding to the image to be detected.

[0267] Optionally, the key point detection device provided in this embodiment of the invention may further include a second model acquisition unit 508, which can be used to acquire a key point determination model;

[0268] Correspondingly, the feature extraction unit 503 can be used to determine the feature map extraction parameters of the model through key points and extract the key feature maps corresponding to each key region.

[0269] The feature processing unit 504 can be used to determine the graph cutting parameters of the model based on key points, divide each key feature map, and obtain at least one feature sub-map corresponding to each key feature map.

[0270] The feature processing parameters of the model are determined by key points, and feature processing is performed on each feature sub-map to generate the features of each feature sub-map.

[0271] The key point determination unit 505 can be used to determine the target key points corresponding to the image-carrying objects in each key region based on the features of each feature sub-image and the key point judgment parameters of the key point determination model.

[0272] Optionally, the key point detection device provided in this embodiment of the invention may further include a model training unit 509, such as... Figure 7 As shown, the model training unit 509 may include a model acquisition unit 5091, a first model processing unit 5092, a training region selection unit 5093, a second model processing unit 5094, a joint loss calculation unit 5095, and a model adjustment unit 5096.

[0273] The model acquisition unit 5091 can be used to acquire the coordinate regression model to be trained, the key point determination model to be trained, and sample images, which are labeled with actual key points.

[0274] The first model processing unit 5092 can be used to extract initial key points from the image to be detected through the coordinate regression model to be trained, so as to obtain at least one initial training key point corresponding to the sample image.

[0275] The training region selection unit 5093 can be used to select the training key region corresponding to each initial training key point in the sample image based on the initial training key points.

[0276] The second model processing unit 5094 can be used to extract target key points from each training key region through the key point determination model to be trained, and determine the training target key points corresponding to the image to be detected from each training key region.

[0277] The joint loss calculation unit 5095 can be used to calculate the joint loss of the coordinate regression model to be trained and the key point determination model to be trained based on the actual key points, the initial training key points and the training target key points.

[0278] The model adjustment unit 5096 can be used to adjust the coordinate regression model and the keypoint determination model to be trained based on the joint loss, so as to obtain the trained coordinate regression model and the trained keypoint determination model.

[0279] Optionally, the coordinate regression model to be trained may include an object prediction sub-model to be trained, and the sample images are labeled with the actual similarity between the sample image carrying the object and the preset expected object of the sample image.

[0280] The model training unit 509 provided in this embodiment of the invention may further include a first sub-model unit 5097, which can be used to predict the object carried by the sample image in the sample image through the object prediction sub-model to be trained, and obtain the training similarity between the object carried by the sample image and the preset expected object of the sample image.

[0281] The joint loss calculation unit 5095 can be used to calculate the first key point prediction loss based on the actual key points and the initial key points during training.

[0282] Calculate the object prediction loss based on the training similarity and the actual similarity;

[0283] Based on the first keypoint prediction loss and the object prediction loss, the first loss corresponding to the coordinate regression model to be trained is obtained;

[0284] Based on the actual key points and the key points of the training target, the second loss corresponding to the model is determined by calculating the key points to be trained.

[0285] Based on the first loss and the second loss, the joint loss of the coordinate regression model to be trained and the key points to be trained is determined.

[0286] Optionally, the key point determination model to be trained may include a confidence prediction sub-model to be trained, the sample images are labeled with the actual confidence level corresponding to each key feature point, and the key feature points are obtained by extracting key points from the feature sub-maps corresponding to the sample images.

[0287] The model training unit 509 provided in this embodiment of the invention may further include a second sub-model unit 5098, which can be used to calculate the confidence of the training target key points through the confidence prediction sub-model to be trained, and obtain the training confidence corresponding to each training target key point.

[0288] The joint loss calculation unit 5095 can be used to calculate the first loss corresponding to the coordinate regression model based on the actual key points and the initial key points of training.

[0289] Based on the actual key points and the training target key points, calculate the key points to be trained and determine the second key point prediction loss corresponding to the model.

[0290] Based on the training confidence and actual confidence corresponding to each training target key point, calculate the confidence prediction loss corresponding to the model for the key points to be trained;

[0291] Based on the second keypoint prediction loss and the confidence prediction loss, the second loss corresponding to the keypoint determination model to be trained is obtained.

[0292] Based on the first loss and the second loss, the joint loss of the coordinate regression model to be trained and the key points to be trained is determined.

[0293] Optionally, the image to be detected is a video frame in the video to be detected, the image carrier is the video carrier of the video to be detected, and the video to be detected contains at least two video frames;

[0294] The key point detection model provided in this embodiment of the invention may further include a pose tracking unit 510, which can be used to obtain a video frame adjacent to the image to be detected from the video to be detected as a new image to be detected, return to perform coordinate regression processing on the image to be detected, and obtain at least one initial key point corresponding to the image to be detected, until the target key point corresponding to the image carrying the object in the new image to be detected is determined.

[0295] Based on the target key points corresponding to the image-carrying object in the previous image to be detected and the target key points corresponding to the image-carrying object in the new image to be detected, the pose change of the video-carrying object in the video to be detected is determined.

[0296] Optionally, the feature extraction unit 503 can be used to divide each key region into sub-regions to obtain a set of sub-regions corresponding to each key region.

[0297] Feature extraction is performed on the sub-regions in each set of sub-regions to obtain the sub-region feature map corresponding to each sub-region;

[0298] Based on the feature maps of the sub-regions, determine the key region feature maps for each key region;

[0299] Based on the feature maps of each key region, the region association weights corresponding to each sub-region in each sub-region set are determined. The region association weights can be used to indicate the association relationships between sub-regions in the sub-region set corresponding to each key region.

[0300] Based on the regional association weight, the feature maps of each key region are weighted and calculated to obtain the discrete feature maps corresponding to each key region.

[0301] As can be seen from the above, by using a key point detection device, the computing resources required for key point detection can be saved and the power consumption of the key point detection device can be reduced while ensuring the accuracy and stability of key point detection.

[0302] Furthermore, embodiments of the present invention also provide an electronic device, which may be a terminal or a server, etc. Figure 8 As shown, it illustrates a structural schematic diagram of the electronic device involved in an embodiment of the present invention, specifically:

[0303] The electronic device may include a radio frequency (RF) circuit 801, a memory 802 including one or more computer-readable storage media, an input unit 803, a display unit 804, a sensor 805, an audio circuit 806, a wireless fidelity (WiFi) module 807, a processor 808 including one or more processing cores, and a power supply 809, etc. Those skilled in the art will understand that... Figure 8 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0304] RF circuit 801 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and hands it over to one or more processors 808 for processing; additionally, it transmits uplink data to the base station. Typically, RF circuit 801 includes, but is not limited to, an antenna, at least one amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, RF circuit 801 can also communicate wirelessly with networks and other devices. Wireless communication can use any communication standard or protocol, including but not limited to GSM, GPRS, CDMA, WCDMA, LTE, email, and SMS.

[0305] The memory 802 can be used to store software programs and modules. The processor 808 executes various functional applications and data processing by running the software programs and modules stored in the memory 802. The memory 802 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device (such as audio data, telephone directory, etc.). In addition, the memory 802 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 802 may also include a memory controller to provide access to the memory 802 for the processor 808 and the input unit 803.

[0306] Input unit 803 can be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, in one embodiment, input unit 803 may include a touch-sensitive surface and other input devices. A touch-sensitive surface, also known as a touch display or touchpad, can collect user touch operations on or near it (e.g., user operations using fingers, styluses, or any suitable object or accessory on or near the touch-sensitive surface) and drive corresponding connection devices according to a pre-set program. Optionally, the touch-sensitive surface may include a touch detection device and a touch controller. The touch detection device detects the user's touch orientation and the signal generated by the touch operation, transmitting the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 808, and can receive and execute commands from the processor 808. Furthermore, various types of touch-sensitive surfaces, such as resistive, capacitive, infrared, and surface acoustic wave, can be used. In addition to the touch-sensitive surface, input unit 803 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0307] Display unit 804 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of electronic devices. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Display unit 804 may include a display panel, optionally configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar form. Furthermore, a touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it transmits the information to processor 808 to determine the type of touch event. Subsequently, processor 808 provides corresponding visual output on the display panel according to the type of touch event. Although in Figure 8 In this context, the touch-sensitive surface and the display panel are two separate components for implementing input and output functions. However, in some embodiments, the touch-sensitive surface and the display panel can be integrated to achieve both input and output functions.

[0308] Electronic devices may also include at least one sensor 805, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel according to the ambient light level, and the proximity sensor can turn off the display panel and / or backlight when the electronic device is moved to the ear. As a type of motion sensor, a gravity acceleration sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometers, taps), etc. Other sensors that may be configured in electronic devices, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0309] Audio circuitry 806, a speaker, and a microphone provide an audio interface between the user and the electronic device. Audio circuitry 806 converts received audio data into electrical signals, transmits them to the speaker, and the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuitry 806, converted back into audio data, and processed by processor 808. The processed data is then transmitted via RF circuitry 801 to, for example, another electronic device, or output to memory 802 for further processing. Audio circuitry 806 may also include an earphone jack to facilitate communication between peripheral headphones and the electronic device.

[0310] WiFi is a short-range wireless transmission technology. Electronic devices using the WiFi module 807 can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 8 WiFi module 807 is shown, but it is understood that it is not a necessary component of an electronic device and can be omitted as needed without changing the nature of the invention.

[0311] The processor 808 is the control center of the electronic device. It connects various parts of the phone via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 802, and by calling data stored in the memory 802, thereby performing overall detection of the phone. Optionally, the processor 808 may include one or more processing cores; preferably, the processor 808 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 808.

[0312] The electronic device also includes a power supply 809 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 808 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 809 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0313] Although not shown, the electronic device may also include a camera, Bluetooth module, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 808 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 802 according to the following instructions, and the processor 808 runs the applications stored in the memory 802 to realize various functions, as follows:

[0314] Coordinate regression processing is performed on the image to be detected to obtain at least one initial key point corresponding to the image to be detected, which includes the image carrying object.

[0315] In the image to be detected, the key regions corresponding to each initial key point are selected based on the initial key points.

[0316] Feature extraction is performed on each key region to obtain the key feature map corresponding to each key region;

[0317] Each key feature map is divided into at least one feature sub-map, and feature processing is performed on each feature sub-map to obtain the features of each feature sub-map;

[0318] Based on the features of each feature sub-image, the target key points corresponding to the image-carrying objects in each key region are determined.

[0319] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0320] To this end, embodiments of the present invention provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the keypoint detection methods provided in the embodiments of the present invention. For example, the instructions can execute the following steps:

[0321] Coordinate regression processing is performed on the image to be detected to obtain at least one initial key point corresponding to the image to be detected, which includes the image carrying object.

[0322] In the image to be detected, the key regions corresponding to each initial key point are selected based on the initial key points.

[0323] Feature extraction is performed on each key region to obtain the key feature map corresponding to each key region;

[0324] Each key feature map is divided into at least one feature sub-map, and feature processing is performed on each feature sub-map to obtain the features of each feature sub-map;

[0325] Based on the features of each feature sub-image, the target key points corresponding to the image-carrying objects in each key region are determined.

[0326] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0327] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0328] Since the instructions stored in the computer-readable storage medium can execute the steps in any of the key point detection methods provided in the embodiments of the present invention, the beneficial effects that any of the key point detection methods provided in the embodiments of the present invention can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0329] According to one aspect of this application, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in the various optional implementations of the above embodiments.

[0330] The foregoing has provided a detailed description of a key point detection method, apparatus, electronic device, storage medium, and computer program provided by embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A key point detection method, characterized in that, include: Coordinate regression processing is performed on the image to be detected to obtain at least one initial key point corresponding to the image to be detected, wherein the image to be detected includes an image carrying object; In the image to be detected, key regions corresponding to each initial key point are selected based on the initial key points; Feature extraction is performed on each of the key regions to obtain the key feature map corresponding to each key region; Each of the key feature maps is divided into at least one feature sub-map, and feature processing is performed on each feature sub-map to obtain the features of each feature sub-map. Based on the features of each feature sub-map, determine the target key points corresponding to the image carrying object in each key region, including: based on the features of each feature sub-map, determine at least one candidate key point corresponding to the image carrying object in each key region; The key regions corresponding to each initial key point are subjected to distribution mapping processing to obtain the Gaussian distribution function corresponding to each key region; based on each Gaussian distribution function and each candidate key point, the confidence level of each candidate key point is calculated; if the confidence level is greater than a preset confidence threshold, the candidate key point is used as the target key point corresponding to the image carrying object.

2. The key point detection method according to claim 1, characterized in that, Before selecting the key regions corresponding to each initial key point in the image to be detected, the method further includes: Perform object type detection on the image-carrying object to obtain the predicted object result corresponding to the image-carrying object; Based on the predicted object result, the similarity between the image carrying object and the preset image expected object is calculated; When the similarity is greater than a preset similarity threshold, the step of selecting key regions corresponding to each initial key point in the image to be detected is performed.

3. The key point detection method according to claim 1, characterized in that, The step of extracting features from each of the key regions to obtain a key feature map corresponding to each key region includes: Feature extraction is performed on each of the key regions to obtain a discrete feature map corresponding to each key region as a key feature map, wherein the key feature map includes at least one discrete feature point. The step of dividing each of the key feature maps into at least one feature sub-map, and performing feature processing on each feature sub-map to obtain the features of each feature sub-map includes: Based on the discrete feature points in the key feature map, the key feature map is divided into sub-maps to obtain discrete feature sub-maps corresponding to the key feature map. Each discrete feature sub-map includes at least one discrete feature point. Based on the discrete feature points, preset interpolation coefficients, and weight coefficients corresponding to the discrete feature points in each discrete feature sub-map, interpolation processing is performed on each discrete feature sub-map to obtain the feature sub-map corresponding to each key feature map. Each of the aforementioned feature sub-graphs is subjected to feature processing to obtain the features of each of the aforementioned feature sub-graphs.

4. The key point detection method according to claim 1, characterized in that, The step of performing feature processing on each of the feature sub-graphs to obtain the features of each feature sub-graph includes: Based on each of the feature sub-maps, determine the position of the key sub-region corresponding to each feature sub-map in the image to be detected; Based on the location of each key sub-region, the mean value of each feature sub-map corresponding to each key feature map is calculated and used as the feature of each feature sub-map.

5. The key point detection method according to claim 1, characterized in that, Before performing coordinate regression processing on the image to be detected to obtain at least one initial key point corresponding to the image to be detected, the method further includes: Obtain the coordinate regression model; The step of performing coordinate regression processing on the image to be detected to obtain at least one initial key point corresponding to the image to be detected includes: The feature extraction parameters of the coordinate regression model are used to extract features from the image to be detected, and an initial feature map corresponding to the image to be detected is generated. Based on the classification mapping parameters of the coordinate regression model, the initial feature points in the initial feature map are classified to obtain at least one initial key point corresponding to the image to be detected.

6. The key point detection method according to claim 5, characterized in that, Before performing feature extraction on each of the key regions to obtain the key feature map corresponding to each key region, the method further includes: Key points are obtained to determine the model; The step of extracting features from each of the key regions to obtain a key feature map corresponding to each key region includes: The feature map extraction parameters of the model are determined by the key points, and the key feature maps corresponding to each key region are extracted. The step of dividing each of the key feature maps into at least one feature sub-map, and performing feature processing on each feature sub-map to obtain the features of each feature sub-map includes: Based on the key points, the graph cutting parameters of the model are determined, and each key feature map is divided to obtain at least one feature sub-map corresponding to each key feature map. The feature processing parameters of the model are determined by the key points, and feature processing is performed on each of the feature sub-graphs to generate the features of each feature sub-graph. The step of determining the target key points corresponding to the image-carrying object in each key region based on the features of each feature sub-map includes: Based on the features of each feature sub-image and the key point determination model, the target key points corresponding to the image-carrying objects in each key region are determined.

7. The key point detection method according to claim 6, characterized in that, Before performing coordinate regression processing on the image to be detected, the method further includes: Obtain the coordinate regression model to be trained, the key point determination model to be trained, and the sample images, wherein the sample images are labeled with actual key points; The coordinate regression model to be trained is used to extract initial key points from the image to be detected, so as to obtain at least one initial training key point corresponding to the sample image. In the sample image, a training key region corresponding to each of the initial training key points is selected based on the initial training key points. The key point determination model to be trained extracts target key points from each of the training key regions, and determines the training target key points corresponding to the image to be detected from each of the training key regions. Based on the actual key points, the initial key points for training, and the target key points for training, calculate the joint loss of the coordinate regression model to be trained and the key point determination model to be trained; Based on the joint loss, the coordinate regression model and the keypoint determination model to be trained are adjusted to obtain the trained coordinate regression model and the trained keypoint determination model.

8. The key point detection method according to claim 7, characterized in that, The coordinate regression model to be trained includes an object prediction sub-model to be trained, and the sample images are labeled with the actual similarity between the sample image carrying the object and the preset expected object of the sample image. Before calculating the joint loss of the coordinate regression model to be trained and the key point determination model to be trained based on the actual key points, the initial training key points, and the target training key points, the method further includes: The object prediction sub-model to be trained is used to predict the object carried by the sample image in the sample image, and the training similarity between the object carried by the sample image and the preset expected object of the sample image is obtained. The step of calculating the joint loss of the coordinate regression model and the keypoint determination model to be trained based on the actual keypoints, the initial training keypoints, and the target training keypoints includes: Calculate the first key point prediction loss based on the actual key points and the initial key points of training; Calculate the object prediction loss based on the training similarity and the actual similarity; Based on the first keypoint prediction loss and the object prediction loss, the first loss corresponding to the coordinate regression model to be trained is obtained; Based on the actual key points and the training target key points, calculate the second loss corresponding to the model based on the key points to be trained; Based on the first loss and the second loss, calculate the joint loss of the coordinate regression model to be trained and the keypoint determination model to be trained.

9. The key point detection method according to claim 7, characterized in that, The key point determination model to be trained includes a confidence prediction sub-model to be trained. The sample image is labeled with the actual confidence level corresponding to each key feature point. The key feature points are obtained by extracting key points from the feature sub-image corresponding to the sample image. Before calculating the joint loss of the coordinate regression model to be trained and the key point determination model to be trained based on the actual key points, the initial training key points, and the target training key points, the method further includes: The confidence score of the training target key points is calculated by the confidence prediction sub-model to be trained, and the training confidence score corresponding to each training target key point is obtained. The step of calculating the joint loss of the coordinate regression model and the keypoint determination model to be trained based on the actual keypoints, the initial training keypoints, and the target training keypoints includes: Based on the actual key points and the initial key points of training, calculate the first loss corresponding to the coordinate regression model; Based on the actual key points and the training target key points, calculate the second key point prediction loss corresponding to the key point determination model to be trained; Based on the training confidence and the actual confidence corresponding to each training target key point, calculate the confidence prediction loss corresponding to the key point determination model to be trained; Based on the second keypoint prediction loss and the confidence prediction loss, the second loss corresponding to the keypoint determination model to be trained is obtained; Based on the first loss and the second loss, calculate the joint loss of the coordinate regression model to be trained and the keypoint determination model to be trained.

10. The key point detection method according to claim 1, characterized in that, The image to be detected is a video frame in the video to be detected, the image carrier is the video carrier of the video to be detected, and the video to be detected contains at least two of the video frames. After determining the target key points corresponding to the image-carrying object in each key region based on the features of each feature sub-map, the method further includes: The video frame adjacent to the image to be detected is obtained from the video to be detected as a new image to be detected. Then, the coordinate regression processing of the image to be detected is performed to obtain at least one initial key point corresponding to the image to be detected, until the target key point corresponding to the image carrying the object in the new image to be detected is determined. Based on the target key points corresponding to the image-carrying object in the previous image to be detected and the target key points corresponding to the image-carrying object in the new image to be detected, the pose change of the video-carrying object in the video to be detected is determined.

11. The key point detection method according to claim 3, characterized in that, The step of extracting features from each of the key regions to obtain discrete feature maps corresponding to each key region as key feature maps includes: Each of the key regions is divided into sub-regions to obtain a set of sub-regions corresponding to each key region. Feature extraction is performed on the sub-regions in each of the sub-region sets to obtain the sub-region feature maps corresponding to each sub-region; Based on the sub-region feature maps, determine the key region feature maps for each of the key regions; Based on the feature maps of each key region, the region association weights corresponding to each sub-region in each set of sub-regions are determined. The region association weights are used to indicate the association relationships between the sub-regions in the set of sub-regions corresponding to each key region. Based on the regional association weights, the feature maps of each key region are weighted and calculated to obtain the discrete feature maps corresponding to each key region.

12. A key point detection device, characterized in that, include: A coordinate regression unit is used to perform coordinate regression processing on the image to be detected to obtain at least one initial key point corresponding to the image to be detected, wherein the image to be detected includes an image carrying object. A region selection unit is used to select key regions corresponding to each initial key point in the image to be detected, based on the initial key points. The feature extraction unit is used to extract features from each of the key regions to obtain key feature maps corresponding to each of the key regions. A feature processing unit is configured to divide each of the key feature maps into at least one feature sub-map, perform feature processing on each feature sub-map, and obtain the features of each feature sub-map. The key point determination unit is used to determine the target key points corresponding to the image carrying object in each key region based on the features of each feature sub-image, including: determining at least one candidate key point corresponding to the image carrying object in each key region based on the features of each feature sub-image. The key regions corresponding to each initial key point are subjected to distribution mapping processing to obtain the Gaussian distribution function corresponding to each key region; based on each Gaussian distribution function and each candidate key point, the confidence level of each candidate key point is calculated; if the confidence level is greater than a preset confidence threshold, the candidate key point is used as the target key point corresponding to the image carrying object.

13. An electronic device, characterized in that, It includes a memory and a processor; the memory stores an application program, and the processor runs the application program within the memory to perform the steps of the keypoint detection method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the keypoint detection method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the key point detection method as described in any one of claims 1 to 11.

Images