A method for processing key points in an image and a related device

Abstract

This application discloses a method and related apparatus for processing key points in an image, applicable to various scenarios such as video, cloud technology, artificial intelligence, intelligent transportation, and assisted driving. The method includes: obtaining a first region image to which the target key point belongs from the (N-1)th frame image, where N is an integer greater than or equal to 2; obtaining a second region image to which the target key point belongs from the Nth frame image; analyzing the optical flow of the target key point between the first and second region images to determine the predicted position offset information of the target key point; compensating the first predicted position information of the target key point in the (N-1)th frame image with the predicted position offset information to obtain compensated predicted position information of the target key point; and using the compensated predicted position information to temporally smooth the second predicted position information of the target key point in the Nth frame image to obtain smoothed predicted position information of the target key point in the Nth frame image, effectively avoiding delays or jitter in key points in consecutive images.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a method and related apparatus for processing key points in an image. Background Technology

[0002] With the rapid development of video technology, the stability of key points in continuous images of a video is particularly important; for example, when Augmented Reality (AR) technology is applied to videos, the stability of key points in continuous images affects the aesthetics and realism of the AR presentation.

[0003] In related technologies, the predicted location information of key points in an image is generally output through a deep learning model. However, deep learning models do not consider the temporal sequence of images during training, which causes key points in the output continuous images to jitter. In order to make the key points in the output continuous images more stable, the predicted location information of key points in the current frame image is usually temporally smoothed by using the predicted location information of key points in the previous frame image.

[0004] However, when the weight of the predicted location information of key points in the previous frame is large in the aforementioned temporal smoothing method, the smoothed image tends to favor the predicted location information of key points in the previous frame, resulting in a delay in key points in consecutive images. Conversely, when the weight of the predicted location information of key points in the previous frame is small, the smoothed image tends to favor the predicted location information of key points in the current frame, resulting in jitter in key points in consecutive images. In other words, the aforementioned temporal smoothing method has a poor smoothing effect on key points in consecutive images and is not suitable for scenarios where the stability of key points in consecutive images is critical. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a method and related apparatus for processing key points in an image, which can effectively prevent delays or jitters in key points in continuous images, thereby improving the smoothing effect of key points in continuous images and adapting to scenarios where the stability requirements of key points in continuous images are high.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] On the one hand, this application provides a method for processing key points in an image, the method comprising:

[0008] Get the first region image to which the target key points belong in the (N-1)th frame image, where N is an integer greater than or equal to 2;

[0009] Obtain the second region image to which the target key point belongs in the Nth frame image; the position of the second region image in the Nth frame image corresponds to the position of the first region image in the (N-1)th frame image;

[0010] Optical flow processing is performed on the target key points in the first region image and the second region image to determine the predicted position offset information of the target key points;

[0011] The first predicted position information of the target key point in the N-1th frame image is compensated using the predicted position offset information to obtain the compensated predicted position information of the target key point.

[0012] Temporal smoothing is performed based on the compensated predicted location information and the second predicted location information of the target key point in the Nth frame image to obtain the smoothed predicted location information of the target key point in the Nth frame image.

[0013] On the other hand, this application provides a processing apparatus for key points in an image, the apparatus comprising: an acquisition unit, a determination unit, a compensation unit, and a smoothing unit;

[0014] The acquisition unit is used to acquire the first region image to which the target key point belongs in the (N-1)th frame image, where N is an integer greater than or equal to 2;

[0015] The acquisition unit is further configured to acquire a second region image to which the target key point belongs in the Nth frame image; the position of the second region image in the Nth frame image corresponds to the position of the first region image in the (N-1)th frame image;

[0016] The determining unit is used to perform optical flow processing on the target key points in the first region image and the second region image to determine the predicted position offset information of the target key points;

[0017] The compensation unit is used to compensate the first predicted position information of the target key point in the (N-1)th frame image using the predicted position offset information, so as to obtain the compensated predicted position information of the target key point.

[0018] The smoothing unit is used to perform temporal smoothing based on the compensated predicted position information and the second predicted position information of the target key point in the Nth frame image, so as to obtain the smoothed predicted position information of the target key point in the Nth frame image.

[0019] On the other hand, this application provides a computer device for processing key points in an image, the computer device including a processor and a memory:

[0020] The memory is used to store program code and transmit the program code to the processor;

[0021] The processor is configured to execute the image key point processing method described above, according to instructions in the program code.

[0022] On the other hand, embodiments of this application provide a computer-readable storage medium for storing a computer program for performing the image key point processing method described above.

[0023] On the other hand, embodiments of this application provide a computer program product, which includes a computer program or instructions; when the computer program or instructions are executed by a processor, the image key point processing method described above is performed.

[0024] As can be seen from the above technical solution, the first region image to which the target key point belongs is obtained from the (N-1)th frame image, where N is an integer greater than or equal to 2; the second region image to which the target key point belongs is obtained from the Nth frame image, and the position of the second region image in the Nth frame image corresponds to the position of the first region image in the (N-1)th frame image; the optical flow of the target key point between the first and second region images is analyzed to determine the predicted position offset information of the target key point; the first predicted position information of the target key point in the (N-1)th frame image is compensated using the predicted position offset information to obtain the compensated predicted position information of the target key point; the second predicted position information of the target key point in the Nth frame image is temporally smoothed using the compensated predicted position information to obtain the smoothed predicted position information of the target key point in the Nth frame image. It is evident that by using the predicted position offset information of the target key point in the first and second region images to compensate the first predicted position information of the target key point in the (N-1)th frame image, a compensated predicted position information that is closer to the second predicted position information of the target key point in the Nth frame image is obtained, and this compensated predicted position information is used to temporally smooth the second predicted position information, reducing the bias of the temporal smoothing result. Based on this, the method can effectively avoid delays or jitters in key points in continuous images, thereby improving the smoothness of key points in continuous images and adapting to scenarios where the stability requirements of key points in continuous images are high. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram illustrating an application scenario of a method for processing key points in an image, as provided in an embodiment of this application.

[0027] Figure 2A flowchart illustrating a method for processing key points in an image, provided in an embodiment of this application;

[0028] Figure 3 A schematic diagram of a first region image and a second region image provided in an embodiment of this application;

[0029] Figure 4 A schematic flowchart illustrating a method for processing key points in an image in a specific application scenario, as provided in this application embodiment;

[0030] Figure 5 A schematic diagram illustrating a scenario for applying makeup to continuous facial images in a video, provided as an embodiment of this application;

[0031] Figure 6 A schematic diagram of an image key point processing device provided in an embodiment of this application;

[0032] Figure 7 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application;

[0033] Figure 8 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0034] The embodiments of this application will now be described with reference to the accompanying drawings.

[0035] Currently, when applying augmented reality technology in videos, the stability of keypoints in continuous images is critical. Generally, deep learning models output predicted keypoint locations in images. However, these models do not calculate a loss function based on temporal constraints during training, thus neglecting the temporal nature of the images and causing keypoint jitter in the output continuous images. To ensure greater stability of keypoints in the output continuous images, smoothing is necessary, typically achieved through temporal smoothing. This temporal smoothing method involves using the predicted keypoint locations from the previous frame to smooth the predicted keypoint locations in the current frame.

[0036] However, when the weight of the predicted location information of key points in the previous frame is large in the aforementioned temporal smoothing method, the smoothed image tends to favor the predicted location information of key points in the previous frame, resulting in a delay in key points in consecutive images. Conversely, when the weight of the predicted location information of key points in the previous frame is small, the smoothed image tends to favor the predicted location information of key points in the current frame, resulting in jitter in key points in consecutive images. Therefore, the aforementioned temporal smoothing method has a poor smoothing effect on key points in consecutive images and is not suitable for scenarios where the stability of key points in consecutive images is critical.

[0037] In view of this, this application proposes a method and related apparatus for processing key points in an image. By using the predicted position offset information of target key points in a first region image and a second region image, the method compensates for the first predicted position information of target key points in the (N-1)th frame image, obtaining compensated predicted position information that is closer to the second predicted position information of target key points in the Nth frame image. This compensated predicted position information is then used to perform temporal smoothing on the second predicted position information, reducing the bias of the temporal smoothing result. Based on this, the method can effectively avoid delays or jitter in key points in consecutive images, thereby improving the smoothing effect of key points in consecutive images and adapting to scenarios where the stability requirements of key points in consecutive images are high.

[0038] The image key point processing method provided in this application embodiment is based on artificial intelligence (AI). AI is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0039] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0040] In the embodiments of this application, the main artificial intelligence software technologies involved include the aforementioned computer vision technologies. Computer vision is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing, detecting, and measuring targets, and further performs image processing to make the computer-processed images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to establish artificial intelligence systems capable of extracting information from images or multidimensional data. For example, it may involve image processing, virtual reality, augmented reality, and other technologies within computer vision (CV) technology, as well as common keypoint recognition technologies.

[0041] The image key point processing method provided in this application can be applied to image key point processing devices with data processing capabilities, such as terminal devices and servers. The terminal device can be a smartphone, computer, tablet, laptop, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, vehicle terminal, etc.; the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services, but is not limited to these. The terminal device and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein. The image key point processing method provided in this application can be applied to various scenarios, including but not limited to video, cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0042] To facilitate understanding of the technical solution of this application, the processing method of key points in images provided in the embodiments of this application will be introduced below in conjunction with actual application scenarios.

[0043] See Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of a method for processing key points in an image, as provided in an embodiment of this application. Figure 1 The application scenario shown includes a terminal device 100, which serves as a processing device for key points in an image and loads and runs a video application that implements AR effects.

[0044] During the operation of a video application that implements AR effects, the terminal device 100 can acquire continuous images of the video, specifically the (N-1)th frame and the Nth frame. To ensure the aesthetic appeal and realism of the AR presentation in the (N-1)th and Nth frames, any key point in the (N-1)th frame is used as a target key point. Therefore, the stability of the target key point in the (N-1)th and Nth frames is particularly important.

[0045] Based on this, the terminal device 100 outputs the first predicted position information of the target key point in the N-1th frame image and the second predicted position information of the target key point in the Nth frame image through a deep learning model; in order to ensure the stability of the target key point in the N-1th frame image and the Nth frame image, the terminal device 100 needs to perform temporal smoothing on the second predicted position information based on the first predicted position information in the following manner.

[0046] First, the terminal device 100 needs to acquire the first region image to which the target key point belongs in the (N-1)th frame image, where N is an integer greater than or equal to 2; correspondingly, the terminal device 100 also needs to acquire the second region image to which the target key point belongs in the Nth frame image, and the position of the second region image in the Nth frame image corresponds to the position of the first region image in the (N-1)th frame image.

[0047] Secondly, the terminal device 100 can perform optical flow processing on the target key points in the first region image and the second region image to determine the predicted position offset information of the target key points; that is, the terminal device 100 obtains the predicted position offset information of the target key points in the (N-1)th frame image and the Nth frame image.

[0048] Then, the terminal device 100 can use the predicted position offset information to compensate the first predicted position information of the target key point in the (N-1)th frame image to obtain the compensated predicted position information of the target key point; the compensated predicted position information is closer to the second predicted position information than the first predicted position information.

[0049] Finally, the terminal device 100 performs temporal smoothing based on the compensated predicted location information and the second predicted location information of the target key points in the Nth frame image to obtain the smoothed predicted location information of the target key points in the Nth frame image. That is, the terminal device 100 uses the compensated predicted location information, which is closer to the second predicted location information, to perform temporal smoothing on the second predicted location information instead of the first predicted location information, reducing the bias of the temporal smoothing result and thus making the smoothed predicted location information more accurate.

[0050] As can be seen, the terminal device 100 can compensate for the first predicted position information of the target key points in the (N-1)th frame image by using the predicted position offset information of the target key points in the first and second region images. This results in compensated predicted position information that is closer to the second predicted position information of the target key points in the Nth frame image. This compensated predicted position information is then used to perform temporal smoothing on the second predicted position information, reducing the bias of the temporal smoothing result. Based on this, this method can effectively avoid delays or jitter in key points in continuous images, thereby improving the smoothing effect of key points in continuous images and adapting to scenarios where the stability requirements of key points in continuous images are high.

[0051] Next, the processing method for key points in images provided in this application embodiment will be specifically described using a terminal device as the processing device for key points in an image.

[0052] See Figure 2 This figure is a flowchart of a method for processing key points in an image according to an embodiment of this application. Figure 2 As shown, the processing method for key points in this image is executed by the terminal device and includes the following steps:

[0053] S201: Obtain the first region image to which the target key point belongs in the (N-1)th frame image, where N is an integer greater than or equal to 2.

[0054] In related technologies, when outputting the predicted location information of keypoints in an image using a deep learning model, to ensure the stability of keypoints in consecutive output images, temporal smoothing is typically performed on the predicted location information of keypoints in the current frame image using the predicted location information of keypoints in the previous frame image. However, the predicted location information of keypoints in the current frame image is shifted relative to the predicted location information of keypoints in the previous frame image. If this shift is small, the predicted location information of keypoints in the previous frame image will have a larger weight in the aforementioned temporal smoothing method, favoring the predicted location information of keypoints in the previous frame image after temporal smoothing, resulting in a delay in keypoints in consecutive images. Conversely, if the shift is large, the predicted location information of keypoints in the previous frame image will have a smaller weight, favoring the predicted location information of keypoints in the current frame image after temporal smoothing, resulting in jitter in keypoints in consecutive images. In other words, it is difficult to adjust the weights reasonably to solve the problem of delay or jitter in keypoints in consecutive images.

[0055] In this embodiment of the application, in order to solve the problem that the above-mentioned temporal smoothing method causes delays or jitter in key points in continuous images, it is considered to compensate the predicted position information of key points in the previous frame image based on the predicted position information of key points in the previous frame image and the predicted position information of key points in the current frame image. This makes the compensated predicted position information of key points in the previous frame image closer to the predicted position information of key points in the current frame image. In this way, the predicted position information of key points in the current frame image can be temporally smoothed by the compensated predicted position information of key points in the previous frame image, reducing the bias of the temporal smoothing result and avoiding delays or jitter in key points in continuous images.

[0056] The prerequisite for compensating for the predicted position information of key points in the previous frame image is to analyze the optical flow of key points between the region image to which the key point belongs in the previous frame image and the region image to which the key point belongs in the current frame image, so as to obtain the predicted position offset information of the key point. Therefore, in this embodiment, if the previous frame image is represented by the (N-1)th frame image and the current frame image is represented by the Nth frame image, and any key point in the (N-1)th frame image is taken as the target key point, then it is first necessary to obtain the first region image to which the target key point belongs from the (N-1)th frame image.

[0057] In a specific implementation of S201, based on obtaining the first predicted position information of the target key point in the (N-1)th frame image, it is necessary to determine the first region image to which the target key point belongs in the (N-1)th frame image using the first predicted position information of the target key point, and then obtain the first region image. Therefore, this application provides a possible implementation method, and S201 may include, for example, obtaining the first region image from the (N-1)th frame image based on the first predicted position information.

[0058] S202: Obtain the second region image to which the target key point belongs in the Nth frame image; the position of the second region image in the Nth frame image corresponds to the position of the first region image in the (N-1)th frame image.

[0059] In this embodiment of the application, after obtaining the first region image to which the target key point belongs in the (N-1)th frame image in S201, corresponding to the position of the first region image in the (N-1)th frame image, it is also necessary to obtain the second region image to which the target key point belongs from the Nth frame image according to the position.

[0060] Corresponding to the implementation of S201 described above, in the specific implementation of S202, in order to ensure that the position of the second region image in the Nth frame corresponds to the position of the first region image in the (N-1)th frame, it is still necessary to determine the second region image to which the target key point belongs in the Nth frame image based on the first predicted position information of the target key point in the (N-1)th frame image, and then obtain the second region image. Therefore, this application provides a possible implementation, and S202 may include, for example, obtaining the second region image from the Nth frame image based on the first predicted position information.

[0061] As an example, such as Figure 3 The diagram shows a first region image and a second region image. The first region image is obtained from the (N-1)th frame of the face image based on the first predicted position information of the mouth keypoints in the (N-1)th frame; the second region image is obtained from the Nth frame of the face image based on the first predicted position information of the mouth keypoints in the (N-1)th frame.

[0062] S203: Perform optical flow processing on the target key points in the first region image and the second region image to determine the predicted position offset information of the target key points.

[0063] In this embodiment of the application, after acquiring the first region image and the second region image in S201-S202, it is necessary to analyze the optical flow of the target key point between the first region image and the second region image to determine the predicted position offset information of the target key point in the first region image and the second region image; the predicted position offset information can be used to compensate the first predicted position information of the target key point in the (N-1)th frame image, so that the compensated first predicted position information is closer to the second predicted position information of the target key point in the Nth frame image.

[0064] In a specific implementation of S203, based on obtaining the target key points in the second region image from the target key points in the first region image after the time interval from the (N-1)th frame image to the Nth frame image, firstly, the optical flow of the target key points between the first and second region images can be analyzed based on optical flow method. That is, optical flow processing is performed on the target key points in the first and second region images to calculate the optical flow vector representing the instantaneous velocity of the target key points. Then, the predicted position offset information of the target key points in the first and second region images can be calculated using the optical flow vector representing the instantaneous velocity of the target key points. Therefore, this application provides a possible implementation method, and S203 may include, for example, the following S2031-S2032:

[0065] S2031: Perform optical flow processing on the target key points in the first region image and the second region image to obtain the optical flow vector of the target key points; the optical flow vector represents the instantaneous velocity of the target key points.

[0066] In the specific implementation of S2031, firstly, it can be based on the two basic assumptions of the optical flow method: the brightness value, that is, the pixel gray value, of a pixel remains constant over time, and the change of time will not cause drastic changes in position; that is, optical flow processing is performed on the target key points in the first region image and the second region image. Based on the fact that the pixel gray value of the target key points in the first region image and the second region image remains constant, and the position of the target key points in the first region image and the second region image will not change drastically, the first relationship between the optical flow vector of the target key points and the image gradient based on the target key points can be obtained.

[0067] Then, an additional assumption is introduced: the surrounding pixels of a pixel have similar motion to that pixel; that is, based on the fact that both the first region image and the second region image include the target key point and the surrounding pixels of the target key point, the fact that the surrounding pixels of the target key point have similar motion to the target key point is equivalent to obtaining a second relationship between the optical flow vector of the target key point and the image gradient based on the surrounding pixels.

[0068] Finally, regarding the first and second relations, since the image gradients based on the target keypoints and the image gradients based on the surrounding pixels can be obtained from the second region image, solving the first and second relations yields the optical flow vector representing the target keypoint. Therefore, this application provides a possible implementation where both the first and second region images include the target keypoint and the surrounding pixels; correspondingly, S2031 may include, for example, the following S1-S3:

[0069] S1: Perform optical flow processing on the target key points in the first region image and the second region image to determine the first relationship between the optical flow vector and the image gradient based on the target key points.

[0070] S2: Based on the first relation, determine the second relation between the optical flow vector and the image gradient based on the surrounding pixels.

[0071] Wherein, the surrounding pixels of the target keypoint can be m×m pixels around the target keypoint, where m is an integer greater than or equal to 3, and the m×m pixels can be, for example, 3×3 pixels or 5×5 pixels; considering the adjacent pixels of the target keypoint, that is, the 3×3 pixels around the target keypoint have more similar motion to the target keypoint, and based on the first relationship, a second relationship between the optical flow vector and the image gradient based on the adjacent pixels is determined. The number of second relationships is minimized, which can greatly reduce the computational load of subsequent solutions. Therefore, this application provides a possible implementation method, in which the surrounding pixels include the adjacent pixels of the target keypoint; S2 may include, for example, determining the second relationship between the optical flow vector and the image gradient based on the adjacent pixels based on the first relationship.

[0072] S3: Solve the first and second relations to obtain the optical flow vector.

[0073] Generally, the (N-1)th frame and the Nth frame in a video are two-dimensional images. Therefore, the first region image and the second region image are also two-dimensional images. Correspondingly, the optical flow vector of the target keypoint can be a two-dimensional vector representing the instantaneous velocity of the target keypoint, that is, a first instantaneous velocity component in the x-axis direction and a second instantaneous velocity component in the y-axis direction. Therefore, this application provides a possible implementation where the optical flow vector includes the first instantaneous velocity component in the x-axis direction and the second instantaneous velocity component in the y-axis direction of the target keypoint.

[0074] As an example, based on the Lucas Kanade optical flow method (LK optical flow method), assuming that the target keypoint I(x,y,t) in the first region image has a time interval dt between the (N-1)th frame and the Nth frame image, and the target keypoint I(x+dx,y+dy,t+dt) in the first region image, and assuming that the pixel grayscale values ​​of the target keypoints in the first and second region images remain constant, we can obtain:

[0075] I(x,y,t)=I(x+dx,y+dy,t+dt)

[0076] Based on the premise that the positions of the target key points in the first and second region images do not change drastically, applying a Taylor expansion to the right side of the above equation yields:

[0077]

[0078] Where ε represents a second-order infinitesimal, which can be neglected, and after simplification, we obtain the following formula:

[0079]

[0080] make We can obtain:

[0081] I x u+I y v+I t =0

[0082] Among them, I x It is the image gradient along the x-axis, I y It is the image gradient along the y-axis, I t It is the gradient along time, (u,v) is the optical flow vector of the target key point, u is the first instantaneous velocity component of the target key point in the x-axis direction, and v is the second instantaneous velocity component of the target key point in the y-axis direction.

[0083] In the above formula, u and v are both unknowns. Since a single formula cannot solve for two unknowns, based on the adjacent pixels of the target keypoint, that is, the 3×3 pixels around the target keypoint have similar motion to the target keypoint, we can obtain:

[0084]

[0085] Solving for (u,v) using the least squares method yields:

[0086]

[0087] Of course, in addition to the LK optical flow method mentioned above, other optical flow methods such as Fast Optical Flow using DenseInverse Search (DIS optical flow method) can also be used; however, the LK optical flow method is faster and more effective in calculating optical loss compared to other optical flow methods.

[0088] S2032: Determine the predicted position offset information based on the optical flow vector.

[0089] When the time interval between the (N-1)th frame and the Nth frame is very small, the optical flow vector of the target key point obtained is equivalent to the predicted position offset information of the target key point in the first region image and the second region image.

[0090] S204: Use the predicted position offset information to compensate for the first predicted position information of the target key points in the (N-1)th frame image, and obtain the compensated predicted position information of the target key points.

[0091] In this embodiment, after determining the predicted position offset information of the target key point in S203, the predicted position offset information can be used to compensate for the first predicted position information of the target key point in the (N-1)th frame image, resulting in the compensated first predicted position information, which serves as the compensated predicted position information of the target key point. The compensated predicted position information of the target key point is closer to the second predicted position information of the target key point in the Nth frame image than the first predicted position information of the target key point in the (N-1)th frame image.

[0092] S205: Perform temporal smoothing based on the compensated predicted location information and the second predicted location information of the target key points in the Nth frame image to obtain the smoothed predicted location information of the target key points in the Nth frame image.

[0093] In this embodiment, after obtaining the compensated predicted position information of the target key point in S204, the compensated predicted position information is used to replace the first predicted position information of the target key point in the (N-1)th frame image, and the second predicted position information of the target key point in the Nth frame image is temporally smoothed. Based on the fact that the compensated predicted position information of the target key point is closer to the second predicted position information of the target key point in the Nth frame image, the tendency of the temporal smoothing result can be reduced, so as to effectively avoid the key point from being delayed or jittered in the continuous images.

[0094] In the specific implementation of S205, firstly, it is necessary to determine the first weight corresponding to the compensated predicted position information of the target key point, and the second weight corresponding to the second predicted position information of the target key point in the Nth frame image, in order to achieve temporal smoothing; the first weight is determined by the first predicted position information of the target key point in the (N-1)th frame image, the second predicted position information of the target key point in the Nth frame image, and the predicted position offset information of the target key point, and the second weight is determined by the first weight. Therefore, this application provides a possible implementation, and S205 may include, for example, the following S2051-S2053:

[0095] S2051: Based on the first predicted location information, the second predicted location information, and the predicted location bias information, determine the first weight corresponding to the compensated predicted location information.

[0096] Specifically, in step S2051, firstly, the difference between the second predicted position information and the first predicted position information is calculated to obtain the predicted position difference information of the target key point; then, based on the predicted position difference information and the predicted position offset information of the target key point, a first weight corresponding to the compensated predicted position information of the target key point is calculated in combination with preset parameters. Therefore, this application provides a possible implementation method, and S2051 may include, for example, the following S4-S5:

[0097] S4: Based on the first predicted location information and the second predicted location information, obtain the predicted location difference information of the target key points.

[0098] S5: Determine the first weight based on the predicted position difference information, the predicted position offset information, and the preset parameters.

[0099] S2052: Based on the first weight, determine the second weight corresponding to the second predicted location information.

[0100] In practical applications, the sum of the first weight and the second weight is 1.

[0101] S2053: Perform temporal smoothing based on the first weight, compensated predicted location information, second weight, and second predicted location information to obtain smoothed predicted location information.

[0102] In practical applications, the formula used by S205 for timing smoothing is as follows:

[0103] P′ T =w(ΔP+Δuv)×(P) T-1 +Δuv)+[1-w(ΔP+Δuv)]×P T

[0104]

[0105] Among them, P T-1 Let P represent the first predicted position information of the target key point in the (N-1)th frame image, and let Δuv represent the predicted position offset information of the target key point. T-1 +Δuv represents the compensated predicted location information of the target key point, and w(ΔP+Δuv) represents the first weight; P T The second predicted location information of the target key point in the Nth frame image is represented by 1-w(ΔP+Δuv), which represents the second weight.

[0106] Furthermore, for video test sets including continuous images, the image key point processing method provided in this application embodiment, compared with the temporal smoothing method of key points in related technologies, calculates the accuracy index—Normalization Mean Error (NME) and the stability index—Mean Landmark Stability Error (MLSE) based on the smoothed predicted location information and the actual location information of the key points, as shown in the table below:

[0107] Table NME and MLSE values

[0108] method NME MLSE Temporal smoothing methods for key points in images in related technologies 0.0259 0.00800 The method for processing key points in images provided in this application embodiment is as follows: 0.0256 0.00788

[0109] In the table, the image key point processing method provided in this application embodiment, compared with the temporal smoothing method of image key points in related technologies, reduces the calculated NME and MLSE values. This indicates that the image key point processing method provided in this application embodiment, compared with the temporal smoothing method of image key points in related technologies, can effectively avoid delays or jitters in key points in continuous images, thereby improving the smoothing effect of key points in continuous images.

[0110] The image key point processing method provided in the above embodiments involves: obtaining a first region image to which the target key point belongs from the (N-1)th frame image, where N is an integer greater than or equal to 2; obtaining a second region image to which the target key point belongs from the Nth frame image, the position of the second region image in the Nth frame image corresponding to the position of the first region image in the (N-1)th frame image; analyzing the optical flow of the target key point between the first and second region images to determine the predicted position offset information of the target key point; compensating the first predicted position information of the target key point in the (N-1)th frame image with the predicted position offset information to obtain compensated predicted position information of the target key point; and using the compensated predicted position information to temporally smooth the second predicted position information of the target key point in the Nth frame image to obtain smoothed predicted position information of the target key point in the Nth frame image. It can be seen that by compensating the first predicted position information of the target key point in the (N-1)th frame image with the predicted position offset information of the target key point in the first and second region images, compensated predicted position information is obtained that is closer to the second predicted position information of the target key point in the Nth frame image, and this compensated predicted position information is used to temporally smooth the second predicted position information, reducing the bias of the temporal smoothing result. Based on this, the method can effectively avoid delays or jitters in key points in continuous images, thereby improving the smoothness of key points in continuous images and adapting to scenarios where the stability requirements of key points in continuous images are high.

[0111] Next, as Figure 4 The diagram illustrates a specific process for processing key points in an image in a particular application scenario. Using a terminal device as the processing device for key points in an image, the method for processing key points in an image provided in this application embodiment is described in detail.

[0112] First, for the face image of frame N-1, the first predicted position information of the mouth key points is obtained. Based on the first predicted position information, the first mouth region image is obtained from the face image of frame N-1. In addition, based on the first predicted position information, the second mouth region image is also obtained from the face image of frame N.

[0113] Secondly, optical flow processing is performed on the key points of the mouth in the first and second mouth region images based on the LK optical flow method to obtain the optical flow vectors of the key points of the mouth; the predicted position offset information of the key points of the mouth is determined based on the optical flow vectors.

[0114] Then, the predicted position offset information of the mouth key points is used to compensate the first predicted position information of the mouth key points in the N-1th frame face image to obtain the compensated predicted position information of the mouth key points.

[0115] Finally, temporal smoothing is performed based on the compensated predicted location information of the mouth key points and the second predicted location information of the mouth key points in the Nth frame face image to obtain the smoothed predicted location information of the mouth key points in the Nth frame face image.

[0116] The image key point processing method provided in this application is applicable to scenarios where the stability of key points in continuous images is highly critical, such as scenarios where AR technology is applied, such as video editing applications, short video applications, and video call applications running on terminal devices. Figure 5 The illustration shows a scene of applying makeup to continuous face images in a video. It uses the image key point processing method provided in the embodiments of this application to ensure the stability of facial key points in continuous face images.

[0117] In addition to the image key point processing method provided in the above embodiments, this application also provides an image key point processing apparatus.

[0118] See Figure 6 , Figure 6 This is a schematic diagram of an image key point processing device provided in an embodiment of this application. Figure 6 As shown, the image key point processing device 600 includes an acquisition unit 601, a determination unit 602, a compensation unit 603, and a smoothing unit 604;

[0119] The acquisition unit 601 is used to acquire the first region image to which the target key points belong in the (N-1)th frame image, where N is an integer greater than or equal to 2;

[0120] The acquisition unit 601 is also used to acquire the second region image to which the target key point belongs in the Nth frame image; the position of the second region image in the Nth frame image corresponds to the position of the first region image in the (N-1)th frame image;

[0121] The determining unit 602 is used to perform optical flow processing on target key points in the first region image and the second region image to determine the predicted position offset information of the target key points.

[0122] The compensation unit 603 is used to compensate the first predicted position information of the target key point in the N-1th frame image using the predicted position offset information, so as to obtain the compensated predicted position information of the target key point.

[0123] The smoothing unit 604 is used to perform temporal smoothing based on the compensated predicted position information and the second predicted position information of the target key points in the Nth frame image, so as to obtain the smoothed predicted position information of the target key points in the Nth frame image.

[0124] As one possible implementation, the smoothing unit 604 is used for:

[0125] Based on the first predicted location information, the second predicted location information, and the predicted location bias information, determine the first weight corresponding to the compensated predicted location information;

[0126] Based on the first weight, determine the second weight corresponding to the second predicted location information;

[0127] Temporal smoothing is performed based on the first weight, compensated predicted location information, second weight, and second predicted location information to obtain smoothed predicted location information.

[0128] As one possible implementation, the smoothing unit 604 is used for:

[0129] Based on the first and second predicted location information, the predicted location difference information of the target key points is obtained.

[0130] The first weight is determined based on the predicted position difference information, the predicted position bias information, and preset parameters.

[0131] As one possible implementation, unit 602 is used for:

[0132] Optical flow processing is performed on the target key points in the first region image and the second region image to obtain the optical flow vector of the target key points; the optical flow vector represents the instantaneous velocity of the target key points;

[0133] Predicted position offset information is determined based on optical flow vectors.

[0134] As one possible implementation, both the first region image and the second region image include target key points and surrounding pixels of the target key points; the determining unit 602 is used for:

[0135] Optical flow processing is performed on target key points in the first region image and the second region image to determine a first relationship between the optical flow vector and the image gradient based on the target key points;

[0136] Based on the first relation, a second relation is determined between the optical flow vector and the image gradient based on the surrounding pixels;

[0137] Solve the first and second relations to obtain the optical flow vector.

[0138] As one possible implementation, surrounding pixels include adjacent pixels of the target keypoint; the determining unit 602 is used for:

[0139] Based on the first relation, a second relation is determined between the optical flow vector and the image gradient based on neighboring pixels.

[0140] As one possible implementation, the optical flow vector includes a first instantaneous velocity component of the target key point in the x-axis direction and a second instantaneous velocity component in the y-axis direction.

[0141] As one possible implementation, the acquisition unit 601 is used for:

[0142] Based on the first predicted location information, the first region image is obtained from the (N-1)th frame image;

[0143] The acquisition unit 601 is also used for:

[0144] Based on the first predicted location information, the second region image is obtained from the Nth frame image.

[0145] The image key point processing apparatus provided in the above embodiments obtains a first region image to which the target key point belongs from the (N-1)th frame image, where N is an integer greater than or equal to 2; obtains a second region image to which the target key point belongs from the Nth frame image, the position of the second region image in the Nth frame image corresponding to the position of the first region image in the (N-1)th frame image; analyzes the optical flow of the target key point between the first and second region images to determine the predicted position offset information of the target key point; compensates the first predicted position information of the target key point in the (N-1)th frame image with the predicted position offset information to obtain compensated predicted position information of the target key point; and uses the compensated predicted position information to perform temporal smoothing on the second predicted position information of the target key point in the Nth frame image to obtain smoothed predicted position information of the target key point in the Nth frame image. It can be seen that by using the predicted position offset information of the target key point in the first and second region images to compensate the first predicted position information of the target key point in the (N-1)th frame image, compensated predicted position information that is closer to the second predicted position information of the target key point in the Nth frame image is obtained, and temporal smoothing is performed on the second predicted position information accordingly, reducing the bias of the temporal smoothing result. Based on this, the method can effectively avoid delays or jitters in key points in continuous images, thereby improving the smoothness of key points in continuous images and adapting to scenarios where the stability requirements of key points in continuous images are high.

[0146] This application also provides a processing device for key points in an image. The computer device provided in this application will be described below from the perspective of hardware implementation.

[0147] In addition to the image key point processing method described above, this application embodiment also provides a terminal device for image key point processing, so that the above image key point processing method can be implemented and applied in practice.

[0148] See Figure 7 , Figure 7This is a schematic diagram of a terminal device provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiment of this application. The terminal device can be any terminal device including mobile phones, tablets, etc.; taking a mobile phone as an example:

[0149] Figure 7 This diagram illustrates a partial structural representation of a mobile phone related to the terminal device provided in this embodiment. (Reference) Figure 7 The mobile phone includes components such as a radio frequency (RF) circuit 710, a memory 720, an input unit 730, a display unit 740, a sensor 750, an audio circuit 760, a Wi-Fi module 770, a processor 780, and a power supply 790. Those skilled in the art will understand that... Figure 7 The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0150] The following is combined Figure 7 A detailed introduction to each component of a mobile phone:

[0151] RF circuit 710 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with processor 780; additionally, it transmits uplink data to the base station. Typically, RF circuit 710 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), and a duplexer. Furthermore, RF circuit 710 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LNE), email, and Short Message Service (SMS).

[0152] The memory 720 can be used to store software programs and modules. The processor 780 runs the software programs and modules stored in the memory 720 to realize various functions and data processing of the mobile phone. The memory 720 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 720 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0153] The input unit 730 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 730 may include a touch panel 731 and other input devices 732. The touch panel 731, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 731), and drive the corresponding connected devices according to a pre-set program. Optionally, the touch panel 731 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 780, and can also receive and execute commands sent by the processor 780. In addition, the touch panel 731 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 731, the input unit 730 may also include other input devices 732. Specifically, other input devices 732 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0154] The display unit 740 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 740 may include a display panel 741, which may optionally be configured as a Liquid Crystal Display (LCD), Organic Light Emitting Diode (OLED), or similar display panel 741. Further, a touch panel 731 may cover the display panel 741. When the touch panel 731 detects a touch operation on or near it, it transmits the information to the processor 780 to determine the type of touch event. Subsequently, the processor 780 provides corresponding visual output on the display panel 741 based on the type of touch event. Although in Figure 7 In this embodiment, the touch panel 731 and the display panel 741 are two separate components to realize the input and output functions of the mobile phone. However, in some embodiments, the touch panel 731 and the display panel 741 can be integrated to realize the input and output functions of the mobile phone.

[0155] The mobile phone may also include at least one sensor 750, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 741 according to the ambient light level, and the proximity sensor can turn off the display panel 741 and / or backlight when the phone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, taps), etc. Other sensors that may be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0156] Audio circuit 760, speaker 761, and microphone 762 provide an audio interface between the user and the mobile phone. Audio circuit 760 converts received audio data into electrical signals and transmits them to speaker 761, where speaker 761 converts them into sound signals for output. On the other hand, microphone 762 converts collected sound signals into electrical signals, which are received by audio circuit 760, converted into audio data, and then processed by processor 780 before being transmitted via RF circuit 710 to, for example, another mobile phone, or the audio data can be output to memory 720 for further processing.

[0157] WiFi is a short-range wireless transmission technology. Through the WiFi module 770, mobile phones can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 7The WiFi module 770 is shown, but it is understood that it is not an essential component of a mobile phone and can be omitted as needed without changing the essence of the invention.

[0158] The processor 780 is the control center of the mobile phone, connecting various parts of the phone through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 720, and by calling data stored in the memory 720, thereby controlling the phone as a whole. Optionally, the processor 780 may include one or more processing units; preferably, the processor 780 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 780.

[0159] The mobile phone also includes a power supply 790 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 780 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0160] Although not shown, mobile phones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0161] In this embodiment of the application, the memory 720 included in the mobile phone can store program code and transmit the program code to the processor.

[0162] The processor 780 included in the mobile phone can execute the following steps according to the instructions in the program code:

[0163] Get the first region image to which the target key points belong in the (N-1)th frame image, where N is an integer greater than or equal to 2;

[0164] Obtain the second region image to which the target key point belongs in the Nth frame image; the position of the second region image in the Nth frame image corresponds to the position of the first region image in the (N-1)th frame image;

[0165] Optical flow processing is performed on the target key points in the first region image and the second region image to determine the predicted position offset information of the target key points;

[0166] The first predicted position information of the target key points in the N-1th frame image is compensated by the predicted position offset information to obtain the compensated predicted position information of the target key points.

[0167] Temporal smoothing is performed based on the compensated predicted location information and the second predicted location information of the target key points in the Nth frame image to obtain the smoothed predicted location information of the target key points in the Nth frame image.

[0168] See Figure 8 , Figure 8 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 800 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 822 (e.g., one or more processors) and memory 832, and one or more storage media 830 (e.g., one or more mass storage devices) for storing application programs 842 or data 844. The memory 832 and storage media 830 can be temporary or persistent storage. The program stored in the storage media 830 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the server. Furthermore, the CPU 822 may be configured to communicate with the storage media 830 and execute the series of instruction operations in the storage media 830 on the server 800.

[0169] Server 800 may also include one or more power supplies 826, one or more wired or wireless network interfaces 850, one or more input / output interfaces 858, and / or one or more operating systems 841, such as Windows Server. NM Mac OS X NM Unix NM Linux NM FreeBSD NM etc.

[0170] The steps performed by the server in the above embodiments can be based on this Figure 8 The server structure shown.

[0171] For example, CPU 822 is used to perform the following steps:

[0172] Get the first region image to which the target key points belong in the (N-1)th frame image, where N is an integer greater than or equal to 2;

[0173] Obtain the second region image to which the target key point belongs in the Nth frame image; the position of the second region image in the Nth frame image corresponds to the position of the first region image in the (N-1)th frame image;

[0174] Optical flow processing is performed on the target key points in the first region image and the second region image to determine the predicted position offset information of the target key points;

[0175] The first predicted position information of the target key points in the N-1th frame image is compensated by the predicted position offset information to obtain the compensated predicted position information of the target key points.

[0176] Temporal smoothing is performed based on the compensated predicted location information and the second predicted location information of the target key points in the Nth frame image to obtain the smoothed predicted location information of the target key points in the Nth frame image.

[0177] This application also provides a computer-readable storage medium for storing a computer program for executing the image key point processing method provided in the above embodiments.

[0178] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of an image keypoint processing apparatus reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the image keypoint processing apparatus to perform the image keypoint processing methods provided in the various optional implementations of the above aspects.

[0179] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium can be at least one of the following media: read-only memory (ROM), RAM, magnetic disk, or optical disk, etc., and other media capable of storing program code.

[0180] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0181] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for processing key points in an image, characterized in that, The method includes: Get the first region image to which the target key points belong in the (N-1)th frame image, where N is an integer greater than or equal to 2; Obtain the second region image to which the target key point belongs in the Nth frame image; the position of the second region image in the Nth frame image corresponds to the position of the first region image in the (N-1)th frame image; Optical flow processing is performed on the target key points in the first region image and the second region image to obtain the optical flow vector of the target key points; the optical flow vector represents the instantaneous velocity of the target key points; The predicted position offset information of the target key point is determined based on the optical flow vector, and the predicted position offset information is used to compensate for the first predicted position information of the target key point in the (N-1)th frame image; The first predicted position information of the target key point in the (N-1)th frame image is compensated using the predicted position offset information to obtain the compensated predicted position information of the target key point. The distance between the compensated predicted position information and the second predicted position information of the target key point in the Nth frame image is less than the distance between the first predicted position information and the second predicted position information. Based on the first predicted location information, the second predicted location information, and the predicted location bias information, a first weight corresponding to the compensated predicted location information is determined; Based on the first weight, determine the second weight corresponding to the second predicted location information; Temporal smoothing is performed based on the first weight, the compensated predicted position information, the second weight, and the second predicted position information to obtain the smoothed predicted position information of the target key point in the Nth frame image.

2. The method according to claim 1, characterized in that, The step of determining the first weight corresponding to the compensated predicted position information based on the first predicted position information, the second predicted position information, and the predicted position bias information includes: Based on the first predicted location information and the second predicted location information, the predicted location difference information of the target key point is obtained; The first weight is determined based on the predicted position difference information, the predicted position bias information, and preset parameters.

3. The method according to claim 1, characterized in that, Both the first region image and the second region image include the target key point and the surrounding pixels of the target key point; the step of performing optical flow processing on the target key point in the first region image and the second region image to obtain the optical flow vector of the target key point includes: Optical flow processing is performed on the target key points in the first region image and the second region image to determine a first relationship between the optical flow vector and the image gradient based on the target key points; Based on the first relationship, a second relationship is determined between the optical flow vector and the image gradient based on the surrounding pixels; Solve the first and second relations to obtain the optical flow vector.

4. The method according to claim 3, characterized in that, The surrounding pixels include the adjacent pixels of the target key point; determining the second relationship between the optical flow vector and the image gradient based on the surrounding pixels, based on the first relationship, includes: Based on the first relationship, a second relationship is determined between the optical flow vector and the image gradient based on the adjacent pixels.

5. The method according to any one of claims 1-4, characterized in that, The optical flow vector includes a first instantaneous velocity component of the target key point in the x-axis direction and a second instantaneous velocity component in the y-axis direction.

6. The method according to any one of claims 1-4, characterized in that, The step of obtaining the first region image to which the target key points belong in the (N-1)th frame image includes: Based on the first predicted location information, the first region image is obtained from the (N-1)th frame image; The step of obtaining the second region image to which the target key point belongs in the Nth frame image includes: Based on the first predicted location information, the second region image is obtained from the Nth frame image.

7. A processing apparatus for key points in an image, characterized in that, The device includes: an acquisition unit, a determination unit, a compensation unit, and a smoothing unit; The acquisition unit is used to acquire the first region image to which the target key point belongs in the (N-1)th frame image, where N is an integer greater than or equal to 2; The acquisition unit is further configured to acquire a second region image to which the target key point belongs in the Nth frame image; the position of the second region image in the Nth frame image corresponds to the position of the first region image in the (N-1)th frame image; The determining unit is configured to perform optical flow processing on the target key points in the first region image and the second region image to obtain the optical flow vector of the target key points; the optical flow vector represents the instantaneous velocity of the target key points; and determine the predicted position offset information of the target key points based on the optical flow vector, the predicted position offset information being used to compensate for the first predicted position information of the target key points in the (N-1)th frame image. The compensation unit is used to compensate the first predicted position information of the target key point in the (N-1)th frame image using the predicted position offset information, so as to obtain the compensated predicted position information of the target key point. The smoothing unit is used to perform temporal smoothing based on the compensated predicted position information and the second predicted position information of the target key point in the Nth frame image to obtain the smoothed predicted position information of the target key point in the Nth frame image. The distance between the compensated predicted position information and the second predicted position information is less than the distance between the first predicted position information and the second predicted position information. The smoothing unit is used for: Based on the first predicted location information, the second predicted location information, and the predicted location bias information, a first weight corresponding to the compensated predicted location information is determined. Based on the first weight, determine the second weight corresponding to the second predicted location information; Based on the first weight, the compensated predicted location information, the second weight, and the second predicted location information, temporal smoothing is performed to obtain the smoothed predicted location information.

8. The apparatus according to claim 7, characterized in that, The smoothing unit is used for: Based on the first predicted location information and the second predicted location information, the predicted location difference information of the target key point is obtained; The first weight is determined based on the predicted position difference information, the predicted position bias information, and preset parameters.

9. The apparatus according to claim 7, characterized in that, Both the first and second region images include target key points and surrounding pixels of the target key points; the determining unit is configured to: Optical flow processing is performed on the target key points in the first region image and the second region image to determine a first relationship between the optical flow vector and the image gradient based on the target key points; Based on the first relationship, a second relationship is determined between the optical flow vector and the image gradient based on the surrounding pixels; Solve the first and second relations to obtain the optical flow vector.

10. The apparatus according to claim 9, characterized in that, The surrounding pixels include adjacent pixels of the target key point; the determining unit is used for: Based on the first relationship, a second relationship is determined between the optical flow vector and the image gradient based on the adjacent pixels.

11. The apparatus according to any one of claims 7-10, characterized in that, The optical flow vector includes a first instantaneous velocity component of the target key point in the x-axis direction and a second instantaneous velocity component in the y-axis direction.

12. The apparatus according to any one of claims 7-10, characterized in that, The acquisition unit is used for: Based on the first predicted location information, the first region image is obtained from the (N-1)th frame image; The acquisition unit is further configured to: Based on the first predicted location information, the second region image is obtained from the Nth frame image.

13. A computer device, characterized in that, The computer device includes a processor and memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the image key point processing method according to any one of claims 1-6 according to the instructions in the program code.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the image key point processing method according to any one of claims 1-6.

15. A computer program product, characterized in that, The computer program product includes a computer program or instructions; when the computer program or instructions are executed by a processor, the method for processing key points in an image as described in any one of claims 1-6 is performed.

Images