Key point detection method and device, electronic equipment and storage medium

Abstract

The application discloses a key point detection method and device, electronic equipment and storage medium. The key point detection method comprises the following steps: identifying key points of a current image frame; if a target key point corresponding to a target point in the key points of the current image frame does not move, performing anti-shake processing on the target key point based on a reference key point corresponding to the target point; the target point is any detection point to be detected in each image frame; the reference key point is a key point corresponding to the target point in the key points of a target image frame; the target image frame is a first image frame in a plurality of specified image frames adjacent to the current image frame and before the current image frame, and the key point corresponding to the target point in the plurality of specified image frames does not move; and the target key point of the current image frame is updated to the key point obtained after the anti-shake processing. The method can improve the stability of key point detection when detecting key points of continuous image frames.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a key point detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid advancement of technology, technologies such as facial recognition and face tracking are attracting increasing attention. Tracking facial targets within consecutive image frames (such as video streams) primarily relies on the locations of facial landmarks determined within the video images. However, in related technologies, the stability of facial landmark detection is poor when performing detection on consecutive image frames. Summary of the Invention

[0003] This application proposes a keypoint detection method, apparatus, electronic device, and storage medium, which can improve the stability of keypoint detection when detecting keypoints in consecutive image frames.

[0004] In a first aspect, embodiments of this application provide a keypoint detection method, the method comprising: identifying keypoints in a current image frame; if a target keypoint corresponding to a target point in the keypoints of the current image frame has not moved, then performing anti-shake processing on the target keypoint based on a reference keypoint corresponding to the target point, wherein the target point is any detection point to be detected in each image frame, the reference keypoint is a keypoint in the keypoints of a target image frame corresponding to the target point, the target image frame is the first image frame among a plurality of specified image frames preceding and adjacent to the current image frame, and the keypoint corresponding to the target point in the plurality of specified image frames has not moved; and updating the target keypoint in the current image frame to the keypoint obtained after the anti-shake processing.

[0005] Secondly, embodiments of this application provide a keypoint detection device, the device comprising: a keypoint recognition module, a stabilization processing module, and a keypoint update module, wherein the keypoint recognition module is used to identify keypoints in the current image frame; the stabilization processing module is used to perform stabilization processing on the target keypoint based on a reference keypoint corresponding to the target keypoint if the target keypoint corresponding to the target keypoint in the keypoints of the current image frame has not moved, wherein the target keypoint is any detection point to be detected in each image frame, the reference keypoint is the keypoint corresponding to the target keypoint in the keypoints of the target image frame, and the target image frame is the first image frame in the image frames before each movement of the keypoint corresponding to the target keypoint; the keypoint update module is used to update the target keypoint in the current image frame to the keypoint obtained after the stabilization processing.

[0006] Thirdly, embodiments of this application provide an electronic device, including: one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to perform the key point detection method provided in the first aspect above.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing program code, which can be invoked by a processor to execute the key point detection method provided in the first aspect above.

[0008] The solution provided in this application identifies key points in the current image frame; if the target key point corresponding to the target point in the key points of the current image frame has not moved, then the target key point is subjected to anti-shake processing based on the reference key point corresponding to the target point. The target point is any detection point to be detected in each image frame, and the reference key point is the key point corresponding to the target point in the key points of the target image frame. The target image frame is the first image frame among multiple specified image frames before and adjacent to the current image frame, and the key point corresponding to the target point in the multiple specified image frames has not moved; the target key point of the current image frame is updated to the key point obtained after anti-shake processing. Therefore, when the key point at any detection point has not moved, the target key point corresponding to the detection point in the current image frame can be stabilized based on the first image frame among the specified image frames in the previous multiple image frames. The key point corresponding to the target point in the specified image frames has not moved, and the target key point is updated based on the key point obtained by the stabilization process. This can avoid large jitter in the key points detected in consecutive image frames and improve the accuracy and stability of key point detection. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart of a key point detection method according to an embodiment of this application is shown.

[0011] Figure 2 A flowchart of a key point detection method according to another embodiment of this application is shown.

[0012] Figure 3A schematic diagram illustrating the effect of the key point detection method provided in the embodiments of this application is shown.

[0013] Figure 4 A flowchart of a key point detection method according to yet another embodiment of this application is shown.

[0014] Figure 5 A block diagram of a key point detection device according to one embodiment of this application is shown.

[0015] Figure 6 This is a block diagram of an electronic device for performing a key point detection method according to an embodiment of this application.

[0016] Figure 7 This is a storage unit in this application embodiment for storing or carrying program code that implements the key point detection method according to this application embodiment. Detailed Implementation

[0017] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0018] For a long time, the lack of stability in keypoint detection (for faces, bodies, etc.) within consecutive image frames has been a persistent problem. For example, even when the face hasn't moved in consecutive image frames, the detected facial keypoints can shift between different frames. If the stability of keypoint detection is insufficient, it will affect the application's performance when further applied based on the detected keypoints. For instance, when applied to facial beautification, it can result in unstable beautification and makeup jitter, leading to a significant negative impact on the user's subjective experience.

[0019] One related technical solution improves the stability of keypoint detection by enhancing the performance of the keypoint detection model itself. Specifically, this is achieved by optimizing the model structure and training methods, and increasing training data to improve the robustness of the detection model, thereby enabling stable detection of keypoints in consecutive image frames. However, achieving stable keypoint detection in this approach typically requires using models with significantly higher computational demands. This usually increases the algorithm's runtime and power consumption on electronic devices such as mobile phones and tablets, and also increases the difficulty of deploying the algorithm.

[0020] Another related technical solution involves adding post-processing algorithms to process the detected key points after detection. Currently, this mainly involves using various filter algorithms, such as low-pass filters, Gaussian filters, Kalman filters, and Euro filters. However, the calculation methods of commonly used filtering algorithms in this approach are quite complex, resulting in unclear effects and side effects. For example, Gaussian and Kalman filters are prone to delays when the face moves quickly, leading to makeup misalignment. Euro filters are relatively sensitive to high-frequency information and have good tracking performance, but their stabilization effect on jittery key points is poor in relatively static states.

[0021] To address the aforementioned problems, the inventors have proposed a keypoint detection method, apparatus, electronic device, and storage medium as described in this application. This method enables the stabilization of the target keypoint corresponding to the detection point in the current image frame, based on the first image frame from among multiple specified image frames preceding the current image frame, when the keypoint at any detection point has not moved. Furthermore, the keypoint is updated based on the stabilization-processed keypoint, thus avoiding significant jitter in keypoints detected across consecutive image frames and improving the accuracy and stability of keypoint detection. The specific keypoint detection method will be described in detail in subsequent embodiments.

[0022] Please see Figure 1 , Figure 1 A flowchart illustrating a keypoint detection method according to an embodiment of this application is shown. In a specific embodiment, the keypoint detection method is applied to, for example... Figure 5 The key point detection device 400 and the electronic device 100 equipped with the key point detection device 400 are shown. Figure 6 The following will use an electronic device as an example to illustrate the specific process of this embodiment. Of course, it is understood that the electronic device used in this embodiment can be a smartphone, tablet computer, smartwatch, e-reader, etc., and is not limited thereto. The following will focus on... Figure 1 The process shown will be described in detail. The key point detection method may specifically include the following steps:

[0023] Step S110: Identify key points in the current image frame.

[0024] In this context, the current image frame refers to the image frame at the current moment among a series of consecutive image frames for keypoint recognition. In other words, the current image frame is also any frame within a series of consecutive image frames. For example, assuming a video includes image frame A at time t-1, image frame B at time t, and image frame C at time t+1, then image frame B is the current image frame when processing up to time t. Keypoints can be the coordinates of key points on the target object to be recognized. For example, if the target object is a face, the keypoints are the coordinates of the face's key points; if the target object is a vehicle, the keypoints are the coordinates of the vehicle's key points. The electronic device can perform keypoint recognition on each image frame in a series of consecutive image frames to obtain the keypoints for each frame. Subsequent steps then process these keypoints to ensure the accuracy of keypoint recognition for each image frame.

[0025] In one possible implementation, the electronic device can perform keypoint detection on the acquired continuous image frames. For example, keypoint detection can be performed on each image frame in a saved video, and further applications can be made based on the detected keypoints for each frame, such as beautification or displaying a detection box for a target object. These continuous image frames can be continuous image frames already captured by the electronic device, such as a captured video; or they can be continuous image frames obtained by the electronic device from other devices, such as a video downloaded from a server.

[0026] In another possible implementation, the aforementioned continuous image frames can also be image frames acquired by the electronic device in real time. In this case, the electronic device can perform key point detection on each acquired image frame in real time, and further apply the detected key points to each image frame. For example, when the electronic device is shooting video, it can perform beautification processing on the faces in each captured image frame; that is, key point detection is performed on each captured image frame in real time.

[0027] In some implementations, an electronic device can identify key points in the current image frame using a pre-trained keypoint detection model, thereby obtaining the key points of the current image frame. Specifically, the electronic device can input the current image frame into the keypoint detection model to obtain the key points of the current image frame output by the keypoint detection model. The specific type of the keypoint detection model is not limited; for example, it can be a Transformer model or other models based on self-attention mechanisms.

[0028] Step S120: If the target key point corresponding to the target point position in the key points of the current image frame has not moved, then the target key point is subjected to anti-shake processing based on the reference key point corresponding to the target point position.

[0029] Wherein, the target point is any detection point in each image frame to be detected, the reference key point is the key point in the target image frame that corresponds to the target point, the target image frame is the first image frame in a set of multiple specified image frames that precede the current image frame and are adjacent to the current image frame, and the key point corresponding to the target point in the multiple specified image frames has not moved.

[0030] In this embodiment, after obtaining the key points of the current image frame, the key points corresponding to the target point in the current image frame can be determined, and it can be determined whether the key points corresponding to the identified target point have moved. If they have moved, it is normal that the position coordinates of the target key point in the current image frame are different from the position coordinates of the target key point in the previous image frame, and therefore no processing is required. If they have not moved, and the position coordinates of the target key point in the current image frame are different from the position coordinates of the target key point in the previous image frame, then the position coordinates of the target key point are inaccurate and need to be processed.

[0031] The movement of keypoints corresponding to the target location refers to the actual movement of the keypoints at that location. For example, when filming a video of the same person, if the user's head moves, then at least some facial keypoints have actually moved. Understandably, when detecting keypoints in consecutive image frames, some keypoints typically won't actually move within a given timeframe. In such cases, if these identified keypoints shift significantly across multiple frames, the application based on those keypoints will perform poorly.

[0032] Furthermore, different points can be understood as key locations (i.e., detection points) of the target object to be identified. For example, for facial key point detection, this can include various key locations of the face, such as the corners of the mouth, the tip of the nose, and the corners of the eyes. The target point is any location among the key locations of the target object to be identified. For key point detection in consecutive image frames, each frame can detect the key point of each of the detection points to be detected. In one approach, an index can be added to each point to distinguish key points from different points. For example, different points can be numbered, and the key points in each image frame can include the key points corresponding to these numbers. For instance, if the target point is the corner of the mouth, and the corner of the mouth is numbered 1, then the target key point is the key point corresponding to number 1. Similarly, in other image frames, if the corner of the mouth exists, the key point corresponding to number 1 can be identified.

[0033] In some implementations, the electronic device can compare the position coordinates of a target key point with the position coordinates of the corresponding key point identified in previous image frames to determine whether the target key point in the current image frame has moved. For example, if the difference between the position coordinates of the target key point and the position coordinates of the corresponding key point identified in the previous image frame is greater than a set distance, it indicates that the target key point in the current image frame has moved; conversely, if the difference between the position coordinates of the target key point and the position coordinates of the corresponding key point identified in the previous image frame is not greater than a set distance, it indicates that the target key point in the current image frame has not moved. Of course, the specific method for determining whether the target key point in the current image frame has moved is not limited.

[0034] When an electronic device determines that the target keypoint corresponding to the target point in the current image frame has not moved, in order to avoid a large deviation in the identified target keypoint if the position coordinates of the target keypoint identified in the current image frame differ from those identified in previous image frames, anti-shake processing can be performed on the target keypoint based on the reference keypoint corresponding to the target point. In the embodiments of this application, the reference keypoint is the keypoint corresponding to the target point in the keypoint of the target image frame, and the target image frame is the first image frame among a number of specified image frames preceding and adjacent to the current image frame, and the keypoint corresponding to the target point in the specified image frames has not moved. That is, the keypoint corresponding to the target point identified in each of the specified image frames has not moved.

[0035] The following describes the possible scenarios for the target image frame.

[0036] In one approach, if the current image frame is the first frame among all consecutive image frames, then the aforementioned target image frame does not exist, meaning the aforementioned reference keypoints do not exist. In this case, anti-shake processing based on the reference keypoints corresponding to the target point is not performed. That is, the target keypoints in the identified keypoints of the current image frame are not updated; instead, the keypoints corresponding to the target point in the final keypoints of the current image frame are directly updated. For example, if consecutive image frames include frames 1 to n, and the current image frame is frame 1, then the aforementioned reference keypoints do not exist. In this case, anti-shake processing based on the reference keypoints corresponding to the target point is not performed.

[0037] In another approach, if the current image frame is not the first frame in a series of consecutive image frames, and the keypoints corresponding to the target location in the image frames preceding the current image frame have not moved, then the specified multiple image frames are the multiple image frames from the first frame to the frame preceding the current image frame in the series of image frames. In other words, the target image frame is the first frame in a series of consecutive image frames. For example, if the series of image frames includes frames 1 to n, the current image frame is the m-th frame, and the target keypoints corresponding to the target location in frames 1 to m-1 have not moved, then the specified multiple image frames are frames 1 to (m-1)-1, meaning the target image frame is the 1st frame. Here, m and n are positive integers, and m is less than n.

[0038] In another approach, if the current image frame is not the first frame in a series of consecutive image frames, and at least one image frame prior to the current image frame has a moving keypoint, then the specified image frames are the multiple image frames from the image frame at the time the keypoint at the target location moved to the image frame preceding the current image frame. In other words, the target image frame is the image frame at the time the keypoint at the target location moved. For example, if the consecutive image frames include frames 1 to n, the current image frame is frame m, and the keypoint at the target location moves at frame t, then the specified image frames are frames t to (m-1)th, meaning the target image frame is frame t. Here, m and n are positive integers, and m is less than t, and t is less than n.

[0039] In some implementations, the anti-shake processing of the target key point is based on the reference key point corresponding to the target point. This can be achieved by performing a weighted calculation based on the position coordinates of the reference key point and the position coordinates of the target key point, such as by performing a weighted summation. The position coordinates obtained by the weighted calculation are then used as the key point obtained after the anti-shake processing. This avoids the final target key point from exhibiting significant jitter compared to the above reference key point, provided that the key point corresponding to the target point itself has not moved.

[0040] Step S130: Update the target key points of the current image frame to the key points obtained after the image stabilization process.

[0041] In this embodiment of the application, when the target key point corresponding to the target point position in the key points of the current image frame has not moved, after the target key point is stabilized based on the reference key point corresponding to the target point position, the key point obtained after stabilization is obtained, that is, the key point corresponding to the target point position of the current image frame after stabilization. The target key point corresponding to the target point position of the current image frame identified above can be updated to the key point obtained after stabilization. That is, the position coordinates of the target key point corresponding to the target point position of the current image frame identified are replaced with the position coordinates of the key point obtained after stabilization.

[0042] In some implementations, since the target points mentioned above are any of the detection points of the target object to be identified, the electronic device can also perform steps S110 and S120 for each detection point. That is, for each detection point, the step of performing anti-shake processing on the target key point based on the reference key point corresponding to the target point is repeatedly executed if the target key point corresponding to the target point in the key points of the current image frame has not moved, until the target key point in the current image frame is updated to the key point obtained after the anti-shake processing, until the above processing is completed for all detection points, thereby avoiding large jitter of the key points corresponding to each detection point compared with the reference key points corresponding to each detection point. For example, the detection points to be identified include point 1, point 2, and point 3. Point 1, point 2, and point 3 can be used as target points respectively, and steps S110 and S120 can be performed.

[0043] The key point detection method provided in this application performs anti-shake processing on the target key point corresponding to the detection point in the current image frame when the key point at any detection point has not moved, based on the key point corresponding to the key point in the first image frame in the image frame before each movement, and updates the target key point. This can avoid large jitter of the key points detected in consecutive image frames and improve the accuracy and stability of key point detection.

[0044] Please see Figure 2 , Figure 2 A flowchart illustrating a key point detection method according to another embodiment of this application is shown. This key point detection method is applied to the aforementioned electronic device, and will be discussed below. Figure 2 The process shown will be described in detail. The key point detection method may specifically include the following steps:

[0045] Step S210: Identify key points in the current image frame.

[0046] Step S220: If the target key point corresponding to the target point in the key points of the current image frame has not moved, then based on the preset weight, the first position coordinate of the target key point and the second position coordinate of the reference key point are weighted and calculated to obtain a new key point as the key point obtained after the image stabilization process.

[0047] Wherein, the target point is any detection point in each image frame to be detected, the reference key point is the key point in the target image frame that corresponds to the target point, the target image frame is the first image frame in a set of multiple specified image frames that precede the current image frame and are adjacent to the current image frame, and the key point corresponding to the target point in the multiple specified image frames has not moved.

[0048] In this embodiment of the application, when performing image stabilization on the target key point based on the reference key point corresponding to the target point, the first position coordinate of the target key point and the second position coordinate of the reference key point can be weighted and calculated based on a preset weight to obtain a new key point as the key point obtained after image stabilization.

[0049] In some implementations, the preset weights may include a first weight corresponding to the reference keypoint and a second weight corresponding to the target keypoint. Optionally, the recognition accuracy of keypoints in the current image frame can be obtained in advance. For example, if a keypoint detection model is used to identify keypoints in the current image frame, the recognition accuracy of the keypoint detection model can be obtained. Then, based on the recognition accuracy, the weights corresponding to the reference keypoints and the target keypoints are determined. The sum of the weights corresponding to the reference keypoints and the target keypoints is 1. The magnitude of the weight corresponding to the reference keypoint is negatively correlated with the magnitude of the recognition accuracy, that is, the magnitude of the weight corresponding to the target keypoint is negatively correlated with the magnitude of the recognition accuracy. In this method, when performing weighted calculations on the first position coordinates of the target keypoint and the second position coordinates of the reference keypoint, the first position coordinates and the second position coordinates can be weighted and summed based on the weights corresponding to the reference keypoints and the target keypoints to obtain a new keypoint as the keypoint obtained after image stabilization.

[0050] In other embodiments, the electronic device can obtain the ratio of a preset constant to the stabilization ratio as a first weight corresponding to the first position coordinate; obtain the difference between the stabilization ratio and the preset constant, and use the ratio of the difference to the stabilization ratio as a second weight corresponding to the second position coordinate; based on the first weight and the second weight, perform a weighted summation on the first position coordinate and the second position coordinate to obtain a new key point as the key point obtained after stabilization processing. Here, the stabilization ratio is a positive number greater than the preset weight, the preset constant can be a fixed value, the magnitude of the first weight is negatively correlated with the magnitude of the stabilization ratio, and the magnitude of the second weight is negatively correlated with the magnitude of the stabilization ratio. Therefore, the first weight and the second weight can be controlled by setting different stabilization ratios. It can be understood that the stabilization ratio can be understood as a parameter used to suppress the jitter of the key point of the target position in the final obtained current image frame relative to the above reference key points; the larger the stabilization ratio, the higher the degree of jitter suppression. Therefore, the stabilization ratio can be set according to the actual recognition accuracy of key points in the current image frame. Specifically, the stabilization ratio can be negatively correlated with the recognition accuracy. That is, the higher the recognition accuracy, the less the suppression of the above-mentioned shaking is required, so the stabilization ratio can be smaller, that is, the larger the first weight and the smaller the second weight; conversely, the lower the recognition accuracy, the larger the stabilization ratio can be, that is, the smaller the first weight and the larger the second weight.

[0051] For example, the first position coordinates of the above target key points are (x i y i The second position coordinate of the reference key point is (x0, y0), and the key point obtained after anti-shake processing is (x0, y0). ’ y ’Given a preset constant of 1 and a stabilization ratio of S, the first weight is 1 / S, the second weight is (S-1) / S, and (x', y') satisfies the following condition:

[0052]

[0053]

[0054] Step S230: Update the target key points of the current image frame to the key points obtained after the image stabilization process.

[0055] In this embodiment, step S230 can be found in other embodiments and will not be repeated here.

[0056] For example, consecutive image frames include a first image frame, a second image frame, a third image frame, and a fourth image frame. See also... Figure 3 P0 is the key point corresponding to the target point of the first image frame, identified in step S210 during key point detection. Since the current image frame is the first frame, the aforementioned reference key point does not exist, therefore no image stabilization is applied to the identified key point P0. P1 is the key point corresponding to the target point of the second image frame, identified in step S210 during key point detection. If the key point corresponding to the target point has not moved, the reference key point is the key point P0 corresponding to the target point in the first image frame. After image stabilization is applied to P1 based on P0, key point P1' is obtained. P2 is the key point corresponding to the target point of the third image frame. For example, when performing keypoint detection, the keypoint corresponding to the target point in the third image frame identified in step S210 is used as a reference if the keypoint has not moved. The reference keypoint is P0, the keypoint corresponding to the target point in the first image frame. Based on P0, P2 is stabilized to obtain keypoint P2'. Similarly, when performing keypoint detection on the fourth image frame, the keypoint corresponding to the target point in step S210 is used as a reference if the keypoint has not moved. The reference keypoint is P0, the keypoint corresponding to the target point in the first image frame. Based on P0, P3 is stabilized to obtain keypoint P3'. It can be seen that, in the above consecutive image frames, when the keypoint corresponding to the target point has not moved, the stabilization process provided in this embodiment allows the final keypoint to be closer to the keypoint in the first image frame, thus avoiding significant jitter in the detected keypoint.

[0057] The key point detection method provided in this application embodiment, for the current image frame, when the key point at any detection point has not moved, uses the reference key point corresponding to the detection point in the first image frame in the image frame before each movement, and performs weighted calculation by combining the first position coordinates of the target key point and the second position coordinates of the reference key point, thereby achieving anti-shaking processing of the target key point corresponding to the detection point. This can avoid large jitter of the key points detected in consecutive image frames, and improve the accuracy and stability of key point detection.

[0058] Please see Figure 4 , Figure 4 A flowchart illustrating a key point detection method according to another embodiment of this application is shown. This key point detection method is applied to the aforementioned electronic device, and will be discussed below. Figure 4 The process shown will be described in detail. The key point detection method may specifically include the following steps:

[0059] Step S310: Identify key points in the current image frame.

[0060] Step S320: If the target key point corresponding to the target point position in the key points of the current image frame has not moved, then the target key point is stabilized based on the reference key point corresponding to the target point position.

[0061] Wherein, the target point is any detection point in each image frame to be detected, the reference key point is the key point in the target image frame that corresponds to the target point, the target image frame is the first image frame in a set of multiple specified image frames that precede the current image frame and are adjacent to the current image frame, and the key point corresponding to the target point in the multiple specified image frames has not moved.

[0062] In some implementations, the distance between the target keypoint and a reference keypoint can be obtained to determine whether the target keypoint corresponding to the target point in the current image frame has moved. After obtaining the distance, it can be compared with a preset distance. If the distance is less than or equal to the preset distance, it is determined that the target keypoint has not moved; otherwise, if the distance is greater than the preset distance, it is determined that the target keypoint has moved. It is understandable that since the reference keypoint is the keypoint corresponding to the target point in the first image frame of a multi-frame specified image frame, and the keypoint corresponding to the target point in the multi-frame specified image frame has not moved, if image stabilization is to be performed on the target keypoint based on the reference keypoint, the distance between the target keypoint and the reference keypoint should be relatively small. Therefore, the distance between the target keypoint and the reference keypoint can be used to determine whether the target keypoint has moved.

[0063] In one possible implementation, the identification of key points in the current image frame can be achieved by inputting the current image frame into a pre-trained keypoint detection model to obtain the key points of the current image frame; that is, by identifying the key points of the current image frame through the keypoint detection model. In this method, the preset distance can be determined by testing the keypoint detection model.

[0064] Optionally, multiple test images can be input into the keypoint detection model to obtain keypoints in each test image. The multiple test images include the same keypoint detection object located at the same position in the image. Based on the keypoints in each test image, the offset distance between keypoints corresponding to the same detection point in every two test images is determined. Based on the offset distance between keypoints corresponding to the same detection point, the preset distance is determined. The average offset distance between all keypoints corresponding to the same detection point can be obtained and used as the preset distance. Understandably, by identifying keypoints of keypoint detection objects in the multiple test images using the keypoint detection model, and since each test image includes the same keypoint detection object located at the same position in the image, the offset distance between keypoints corresponding to the same detection point in each test image can be determined when the keypoint detection model is applied to keypoint recognition in consecutive image frames, assuming the keypoints at the same detection point in different frames have not moved. Based on this, the preset distance used to determine whether the keypoint corresponding to the target point has moved can be set.

[0065] In some implementations, optical flow tracing can also be used to determine whether the keypoints corresponding to the target point have moved. Specifically, the optical flow vector of the target keypoint relative to the keypoints in the previous image frame (the frame preceding the current image frame) can be obtained. If the optical flow vector is less than or equal to a preset vector value, it is determined that the target keypoint has not moved. In essence, by performing optical flow tracing on the keypoints of the target point in each image frame to determine the optical flow vector of the target keypoint in the current image frame relative to the keypoints in the previous frame, and comparing this optical flow vector with a preset vector value to determine whether the keypoint has moved, if the optical flow vector is less than or equal to the preset vector value, it is determined that the target keypoint has not moved; if the optical flow vector is greater than the preset vector value, it is determined that the target keypoint has moved. Optionally, optical flow tracing can be performed using algorithms such as Lucas-Kanade optical flow tracing, KLT optical flow tracing, and pyramid KLT optical flow tracing; the specific optical flow tracing method is not limited.

[0066] Step S330: Update the target key points of the current image frame to the key points obtained after the image stabilization process.

[0067] In this embodiment, step S330 can be found in other embodiments and will not be repeated here.

[0068] Step S340: If the target key point moves, update the reference key point to the target key point.

[0069] In this embodiment, if it is determined that the target keypoint corresponding to the target point in the keypoints of the current image frame has moved, then even if the position coordinates of the target keypoint identified in the current image frame are different from the position coordinates of the target keypoint identified in previous image frames, this is normal and therefore no processing is required for the identified target keypoint. Furthermore, to facilitate processing of the identified target keypoints in subsequent image frames when detecting keypoints, the reference keypoint can be updated to the target keypoint, ensuring that the reference keypoint satisfies the following conditions: the reference keypoint is the keypoint corresponding to the target point in the keypoints of the target image frame, and the target image frame is the first image frame among multiple specified image frames preceding and adjacent to the current image frame.

[0070] The keypoint detection method provided in this application embodiment performs anti-shake processing on the target keypoint corresponding to the detection point in the current image frame when the keypoint at any detection point has not moved, based on the keypoint corresponding to the keypoint in the first image frame before each movement. The target keypoint is then updated, thereby avoiding significant jitter in the keypoints detected in consecutive image frames and improving the accuracy and stability of keypoint detection. Furthermore, in this application embodiment, when the keypoint corresponding to the detection point moves, the reference keypoint is updated to the keypoint corresponding to the detection point identified in the current image frame, thus ensuring the accuracy of the reference keypoint when processing subsequent image frames.

[0071] Please see Figure 5The diagram illustrates a structural block diagram of a key point detection device 400 provided in an embodiment of this application. The key point detection device 400 utilizes the aforementioned electronic equipment and includes: a key point recognition module 410, a stabilization processing module 420, and a key point update module 430. The key point recognition module 410 is used to identify key points in the current image frame; the anti-shake processing module 420 is used to perform anti-shake processing on the target key point based on the reference key point corresponding to the target point if the target key point corresponding to the target point in the key points of the current image frame has not moved. The target point is any detection point to be detected in each image frame, the reference key point is the key point corresponding to the target point in the key points of the target image frame, the target image frame is the first image frame among multiple specified image frames before and adjacent to the current image frame, and the key point corresponding to the target point in the multiple specified image frames has not moved; the key point update module 430 is used to update the target key point of the current image frame to the key point obtained after the anti-shake processing.

[0072] In some implementations, the image stabilization module 420 may be specifically used to: perform a weighted calculation on the first position coordinates of the target key point and the second position coordinates of the reference key point based on a preset weight, and obtain a new key point as the key point obtained after image stabilization.

[0073] In one possible implementation, the image stabilization module 420 may be specifically used to: obtain the ratio of a preset constant to the image stabilization ratio as a first weight corresponding to the first position coordinate; obtain the difference between the image stabilization ratio and the preset constant, and use the ratio of the difference to the image stabilization ratio as a second weight corresponding to the second position coordinate; based on the first weight and the second weight, perform a weighted summation on the first position coordinate and the second position coordinate to obtain a new key point as the key point obtained after the image stabilization process.

[0074] In some embodiments, the keypoint detection device 400 may further include a first distance acquisition module and a first motion determination module. The distance acquisition module is used to acquire the distance between the target keypoint and the reference keypoint before performing image stabilization processing on the target keypoint based on the reference keypoint corresponding to the target point if the target keypoint corresponding to the target point in the keypoint of the current image frame has not moved; the first motion determination module is used to determine that the target keypoint has not moved if the distance is less than or equal to a preset distance.

[0075] In one possible implementation, the keypoint detection device 400 may further include a second distance acquisition module. The keypoint recognition module 410 may specifically be used to: input the current image frame into a pre-trained keypoint detection model to obtain the keypoints of the current image frame. The keypoint recognition module 410 may also be used to input multiple test images into the keypoint detection model to obtain the keypoints of each test image, wherein the multiple test images include the same keypoint detection object located at the same position in the image; the second distance acquisition module is used to determine, based on the keypoints of each test image, the offset distance between keypoints corresponding to the same detection point position in every two test images; and to determine the preset distance based on the offset distance between keypoints corresponding to the same detection point position.

[0076] In some embodiments, the keypoint detection device 400 further includes a vector acquisition module and a second motion determination module. The vector acquisition module is used to acquire the optical flow vector of the keypoint corresponding to the target point in the keypoints of the previous image frame relative to the target point, before performing image stabilization processing on the target keypoint based on the reference keypoint corresponding to the target point if the target keypoint corresponding to the target point in the keypoints of the current image frame has not moved. The previous image frame is located one frame before the current image frame. The second motion determination module is used to determine that the target keypoint has not moved if the optical flow vector is less than or equal to a preset vector value.

[0077] In some implementations, the key point update module 430 may also be used to update the reference key point to the target key point if the target key point moves after the key point of the current image frame has been identified.

[0078] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0079] In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.

[0080] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0081] In summary, the solution provided in this application identifies key points in the current image frame; if the target key point corresponding to the target point in the key points of the current image frame has not moved, then the target key point is subjected to anti-shake processing based on the reference key point corresponding to the target point. The target point is any detection point to be detected in each image frame, and the reference key point is the key point corresponding to the target point in the key points of the target image frame. The target image frame is the first image frame among multiple specified image frames before and adjacent to the current image frame, and the key point corresponding to the target point in the multiple specified image frames has not moved; the target key point of the current image frame is updated to the key point obtained after anti-shake processing. Therefore, when the key point at any detection point has not moved, the target key point corresponding to the detection point in the current image frame can be stabilized based on the first image frame among the specified image frames in the previous multiple image frames. The key point corresponding to the target point in the specified image frames has not moved, and the target key point is updated based on the key point obtained by the stabilization process. This can avoid large jitter in the key points detected in consecutive image frames and improve the accuracy and stability of key point detection.

[0082] Please refer to Figure 6 This document illustrates a structural block diagram of an electronic device according to an embodiment of this application. The electronic device 100 can be a smartphone, tablet computer, smartwatch, e-reader, or other electronic device capable of running applications. The electronic device 100 in this application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications can be stored in the memory 120 and configured to be executed by the one or more processors 110, and the one or more applications are configured to perform the methods described in the foregoing method embodiments.

[0083] Processor 110 may include one or more processing cores. Processor 110 connects to various parts within the electronic device 100 using various interfaces and lines, and performs various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets, or instruction sets stored in memory 120, and by calling data stored in memory 120. Optionally, processor 110 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 110 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 110 and may be implemented separately using a communication chip.

[0084] The memory 120 may include random access memory (RAM) or read-only memory (ROM). The memory 120 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created by the electronic device 100 during use (such as phonebook data, audio and video data, chat log data, etc.).

[0085] Please refer to Figure 7 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable medium 800 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0086] The computer-readable storage medium 800 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 800 has storage space for program code 810 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 810 may be compressed, for example, in a suitable form.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A key point detection method, characterized in that, The method includes: Identify key points in the current image frame; If the target key point corresponding to the target point in the key points of the current image frame has not moved, then the target key point is subjected to anti-shake processing based on the reference key point corresponding to the target point. The target point is any detection point to be detected in each image frame, and the reference key point is the key point corresponding to the target point in the key points of the target image frame. The target image frame is the first image frame among multiple specified image frames that are before the current image frame and adjacent to the current image frame, and the key point corresponding to the target point in the multiple specified image frames has not moved. Update the target key points of the current image frame to the key points obtained after the image stabilization process; The step of performing anti-shake processing on the target key points based on the reference key points corresponding to the target point includes: The ratio of a preset constant to the image stabilization ratio is used as the first weight corresponding to the first position coordinate of the target key point; Obtain the difference between the image stabilization ratio and the preset constant, and use the ratio of the difference to the image stabilization ratio as the second weight corresponding to the second position coordinate of the reference key point; Based on the first weight and the second weight, the first position coordinates and the second position coordinates are weighted and summed to obtain new key points, which are then used as the key points obtained after the image stabilization process.

2. The method according to claim 1, characterized in that, Before performing image stabilization on the target keypoint based on the reference keypoint corresponding to the target keypoint if the target keypoint in the current image frame has not moved, the method further includes: Obtain the distance between the target key point and the reference key point; If the distance is less than or equal to the preset distance, it is determined that the target key point has not moved.

3. The method according to claim 2, characterized in that, The key points for identifying the current image frame include: The current image frame is input into a pre-trained keypoint detection model to obtain the keypoints of the current image frame; The method further includes: Multiple test images are input into the key point detection model to obtain key points for each test image. The multiple test images include the same key point detection object located at the same position in the image. Based on the key points of each test image, determine the offset distance between the key points corresponding to the same detection point in every two test images; The preset distance is determined based on the offset distance between key points corresponding to the same detection point.

4. The method according to claim 1, characterized in that, Before performing image stabilization on the target keypoint based on the reference keypoint corresponding to the target keypoint if the target keypoint in the current image frame has not moved, the method further includes: Obtain the optical flow vector of the key point corresponding to the target key point in the key points of the previous image frame relative to the target key point; the previous image frame is located in the frame before the current image frame. If the optical flow vector is less than or equal to a preset vector value, then it is determined that the target key point has not moved.

5. The method according to any one of claims 1-4, characterized in that, After identifying the key points of the current image frame, the method further includes: If the target key point moves, the reference key point is updated to the target key point.

6. A key point detection device, characterized in that, The device includes: a key point recognition module, a stabilization module, and a key point update module, wherein... The key point recognition module is used to identify key points in the current image frame; The image stabilization module is used to perform image stabilization on the target keypoint based on the reference keypoint corresponding to the target keypoint if the target keypoint corresponding to the target keypoint in the keypoint of the current image frame has not moved. The target keypoint is any detection point to be detected in each image frame, and the reference keypoint is the keypoint corresponding to the target keypoint in the keypoint of the target image frame. The target image frame is the first image frame in a set of multiple specified image frames that are before the current image frame and adjacent to the current image frame, and the keypoint corresponding to the target keypoint in the multiple specified image frames has not moved. The key point update module is used to update the target key points of the current image frame to the key points obtained after the image stabilization process; The image stabilization module is specifically used to obtain the ratio of a preset constant to the image stabilization ratio as a first weight corresponding to the first position coordinate of the target key point; obtain the difference between the image stabilization ratio and the preset constant, and use the ratio of the difference to the image stabilization ratio as a second weight corresponding to the second position coordinate of the reference key point; based on the first weight and the second weight, perform a weighted summation on the first position coordinate and the second position coordinate to obtain a new key point as the key point obtained after the image stabilization process.

7. An electronic device, characterized in that, include: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1-5.

Images