Data processing method and apparatus, and electronic device and computer-readable medium

Abstract

Provided in the embodiments of the present disclosure are a data processing method and apparatus, and an electronic device and a computer-readable medium. The method comprises: first, acquiring a key point prediction result for an image to be processed, wherein the key point prediction result comprises position description data of a plurality of key points; and then with reference to position description data of some key points described in the key point prediction result, adjusting position description data of the remaining key points described in the key point prediction result.

Description

Data processing method, device, electronic device, and computer-readable medium

[0001] This application claims priority to Chinese Patent Application No. 202310808970.3 filed on July 3, 2023. The contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this application. Technical Field

[0002] The present disclosure relates to a data processing method, device, electronic device, and computer-readable medium. Background Art

[0003] Key point prediction is a computer vision task, and key point prediction is used to locate certain key points in an image data (for example, the key point of the corner of the eye, the key point of the finger, etc.).

[0004] Summary of the Invention

[0005] The technical solutions provided by this disclosure are as follows:

[0006] The present disclosure provides a data processing method, the method comprising:

[0007] Acquire key point prediction results of the image to be processed, wherein the key point prediction results include position description data of multiple key points; the multiple key points include points to be adjusted and reference points corresponding to the points to be adjusted;

[0008] The position description data of the point to be adjusted in the key point prediction result is adjusted according to the position description data of the reference point.

[0009] In a possible implementation manner, the adjustment process of the position description data of the point to be adjusted includes:

[0010] Determining an inter-point offset between the point to be adjusted and the reference point based on the position description data of the reference point and the position description data of the point to be adjusted;

[0011] Correcting the inter-point offset using a historical offset between the point to be adjusted and the reference point to obtain a corrected offset; the historical offset is determined based on a historical image corresponding to the image to be processed;

[0012] The position description data of the point to be adjusted in the key point prediction result is updated according to the corrected offset and the position description data of the reference point.

[0013] In one possible implementation manner, before updating the position description data of the point to be adjusted in the key point prediction result based on the corrected offset and the position description data of the reference point, the method further includes:

[0014] Correcting the position description data of the reference point using the historical position data of the reference point to obtain corrected position data of the reference point; the historical position data is determined based on a historical image corresponding to the image to be processed;

[0015] The updating of the position description data of the point to be adjusted in the key point prediction result according to the corrected offset and the position description data of the reference point includes:

[0016] The position description data of the point to be adjusted in the key point prediction result is updated using the corrected offset and the corrected position data of the reference point.

[0017] In a possible implementation manner, after obtaining the key point prediction result of the image to be processed, the method further includes:

[0018] Using the historical position data of the reference point, adjusting the position description data of the reference point in the key point prediction result; the historical position data is determined based on the historical image corresponding to the image to be processed;

[0019] The adjusted key point prediction result includes the adjusted position description data of the reference point and the adjusted position description data of the point to be adjusted.

[0020] In a possible implementation manner, the image to be processed and the historical image belong to the same image sequence;

[0021] The time sequence corresponding to the historical image in the image sequence is earlier than the time sequence corresponding to the image to be processed in the image sequence.

[0022] In a possible implementation manner, before adjusting the position description data of the point to be adjusted in the key point prediction result based on the position description data of the reference point, the method further includes:

[0023] Dividing the key point prediction results to obtain at least one group, the group including position description data of at least one key point, the key points in the same group corresponding to the same structural division area, and the key points in different groups corresponding to different structural division areas;

[0024] For any of the groups, the points to be adjusted and the reference points corresponding to the points to be adjusted are determined according to the key point pairs having a structural constraint relationship in the group.

[0025] In one possible implementation, if the multiple key points all belong to a face, then the at least one structural division area corresponding to the face includes one or more of an eye description area, an eyebrow description area, a nose description area, a mouth description area, and a facial contour description area.

[0026] In one possible implementation, the point to be adjusted corresponds to at least two reference points;

[0027] The adjusting the position description data of the point to be adjusted in the key point prediction result based on the position description data of the reference point includes:

[0028] The position description data of the point to be adjusted in the key point prediction result is adjusted according to the position description data of each reference point and the weight corresponding to each reference point.

[0029] In one possible implementation, for any reference point corresponding to the point to be adjusted, the weight corresponding to the reference point is determined based on the distance between the reference point and the point to be adjusted, and the distance is determined based on the position description data of the reference point and the position description data of the point to be adjusted.

[0030] The present disclosure provides a data processing device, comprising:

[0031] An acquisition unit is configured to acquire a key point prediction result of the image to be processed, wherein the key point prediction result includes position description data of a plurality of key points; the plurality of key points include points to be adjusted and reference points corresponding to the points to be adjusted;

[0032] The adjusting unit is configured to adjust the position description data of the point to be adjusted in the key point prediction result according to the position description data of the reference point.

[0033] The present disclosure provides an electronic device, the device comprising: a processor and a memory;

[0034] The memory is configured to store instructions or computer programs;

[0035] The processor is configured to execute the instructions or computer programs in the memory so that the electronic device performs the data processing method provided by the present disclosure.

[0036] The present disclosure provides a computer-readable medium, wherein the computer-readable medium stores instructions or computer programs. When the instructions or computer programs are executed on a device, the device executes the data processing method provided by the present disclosure.

[0037] The present disclosure provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the data processing method provided by the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0039] FIG1 is a flow chart of a data processing method provided by an embodiment of the present disclosure;

[0040] FIG2 is a schematic structural diagram of a data processing device provided by an embodiment of the present disclosure;

[0041] FIG3 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] Research has found that for some key point prediction methods, due to the limitations of the key point prediction methods in certain aspects (such as prediction accuracy, uncertainty of annotation information, etc.), the prediction results determined by the key point prediction methods will fluctuate within a certain range, thereby causing visual jitter.

[0043] The study also found that for key points, the visual recognition of different key points is different. Specifically, some key points (for example, key points located at the corners of the eyes, etc.) are visually very obvious, so that these key points with obvious visual features have higher visual recognition, so that the prediction results determined by the upper key point prediction method for these key points with obvious visual features are relatively stable; however, other key points have no obvious visual features, and they only serve as transitions between certain key points (for example, key points located at the two corners of the left eye, etc.), so that these key points without obvious visual features have lower visual recognition, so that the prediction results determined by the upper key point prediction method for these key points without obvious visual features are prone to fluctuations, which in turn leads to visual jitter.

[0044] Further research has found that for key points, there are relatively stable structural constraints between some key points (for example, the relative offset between certain key points remains unchanged, etc.). Therefore, in order to better improve the prediction effect, the prediction results shown in the previous paragraph can be corrected based on this structural constraint.

[0045] Based on the findings in the previous paragraph, in order to better improve the prediction effect, the present disclosure provides a data processing method, which includes: for an image to be processed (for example, any frame of video image in a certain video data), obtaining the key point prediction result of the image to be processed, so that the key point prediction result includes position description data of multiple key points, so that when it is determined that the multiple key points include a reference point and a point to be adjusted with a structural constraint relationship, the position description data of the point to be adjusted is adjusted according to the position description data of the reference point, so that the adjusted position description data can better describe the position of the point to be adjusted, thereby effectively avoiding the visual jitter phenomenon caused by fluctuations in the prediction results for the point to be adjusted, thereby helping to improve the stability of key point position determination.

[0046] In addition, the present disclosure does not limit the execution subject of the above data processing method. For example, the data processing method provided in the embodiment of the present disclosure can be applied to a device with data processing function such as a terminal device or a server. For another example, the data processing method provided in the embodiment of the present disclosure can also be implemented with the help of a data communication process between different devices (for example, a terminal device and a server, two terminal devices, or two servers). Among them, the terminal device can be a smart phone, a computer, a personal digital assistant (PDA) or a tablet computer. The server can be a stand-alone server, a cluster server or a cloud server.

[0047] In order to enable those skilled in the art to better understand the solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0048] To better understand the technical solutions provided by the present disclosure, the data processing method provided by the present disclosure is described below with reference to some figures. As shown in Figure 1, the data processing method provided by the embodiment of the present disclosure includes the following S1-S2. Figure 1 is a flow chart of a data processing method provided by the embodiment of the present disclosure.

[0049] S1: Obtain key point prediction results of the image to be processed, where the key point prediction results include position description data of multiple key points; the multiple key points include points to be adjusted and reference points corresponding to the points to be adjusted.

[0050] The image to be processed refers to image data that needs to be processed for key point determination. The present disclosure does not limit the image to be processed. For example, it can be image data that needs to be processed for key point determination in any application field. For another example, in some image sequence-related application scenarios (for example, video-related application scenarios), the image to be processed can refer to any image data in the image sequence (for example, any frame of video image in a video data). It can be seen that in one possible implementation, the image to be processed refers to image data extracted from an image sequence.

[0051] The key point prediction results of the image to be processed are used to describe the positional distribution of key points corresponding to the target object in the image to be processed. The target object refers to the object depicted by the image to be processed; this disclosure is not limited to this target object. For example, the target object can be a face. In another example, the target object can be the body of a living creature. In another example, the target object can be an object.

[0052] In addition, for the key point prediction results of the above-mentioned image to be processed, the key point prediction results can include position description data of multiple key points, so that the key point prediction results can describe the predicted positions of these key points in the image to be processed. The position description data of the key points is used to describe the predicted positions of the key points in the image to be processed; and the present disclosure does not limit the implementation method of the position description data of the key points. For example, it can be implemented using any existing or future data that can represent the position of a key point (such as position coordinates, etc.).

[0053] In addition, in some application scenarios, the key point prediction results of the above-mentioned image to be processed may include not only the position description data of multiple key points, but also the point identification data corresponding to the multiple key points, so that the key point prediction results can not only indicate the predicted positions of these key points in the image to be processed, but also indicate which key point each key point belongs to corresponding to the above-mentioned target object. Among them, the point identification data corresponding to the key point is used to uniquely identify the key point, so that the point identification data corresponding to the key point can indicate which key point the key point belongs to corresponding to the target object (for example, a key point with a serial number of 2). It should be noted that the present disclosure does not limit the implementation method of the point identification data. For example, it can be implemented using any existing or future identification information (for example, serial numbers, etc.) that can distinguish different key points corresponding to the target object.

[0054] In addition, the present disclosure does not limit the method for obtaining the key point prediction results of the image to be processed. For example, it can be specifically: after obtaining the image to be processed, the key point prediction processing is performed on the image to be processed to obtain the key point prediction results of the image to be processed, so that the key point prediction results can describe the key point distribution presented in the image to be processed. It should be noted that the present disclosure does not limit the implementation method of the key point prediction processing. For example, it can be implemented using any existing or future method that can perform key point detection processing on an image data (for example, a pre-built machine learning model with key point detection function, etc.).

[0055] The point to be adjusted refers to a key point without obvious visual features (that is, a key point with low recognition), so that the point to be adjusted can indicate the key point that needs to be adjusted for prediction; and the present disclosure does not limit the process of determining the point to be adjusted. For example, when the key point prediction results above include prediction results of N key points (for example, position description data + point identification data), if the prediction result of the nth key point meets the preset adjustment condition, then the nth key point can be regarded as the point to be adjusted. Wherein, n is a positive integer, n≤N. The preset adjustment condition is used to describe the conditions reached by the key point without obvious visual features; and the preset adjustment condition can be set according to the actual application scenario. In addition, the present disclosure does not limit the preset adjustment condition. For example, the preset adjustment condition can specifically be: the position described by the position description data of the nth key point is within a preset adjustment area; the preset adjustment area refers to an area that is determined in advance for the above image to be processed and needs to be adjusted for prediction position. For example, the preset adjustment condition may also be: the point identification data corresponding to the nth key point belongs to a pre-set non-anchor point identification set; the non-anchor point identification set is used to record the identification of key points without obvious visual features; and the non-anchor point identification set can be set in advance based on the application scenario.

[0056] The reference point corresponding to the point to be adjusted refers to the key point required to be referenced when performing the predicted position adjustment process for the point to be adjusted; and the reference point has obvious visual features (that is, has high recognition). In addition, the present disclosure does not limit the determination process of the reference point. For example, it can be specifically: searching for a key point that has a corresponding relationship with the point to be adjusted from a pre-constructed mapping relationship as the reference point corresponding to the point to be adjusted. Among them, the mapping relationship is used to record the structural constraint relationship between some key points without obvious visual features and some key points with obvious visual features, so that the mapping relationship can represent one or more key points with obvious visual features (for example, reference points) corresponding to each key point without obvious visual features (for example, each point to be adjusted). It can be seen that in one possible embodiment, the reference point corresponding to the point to be adjusted can include at least one key point. That is, in some possible embodiments, the point to be adjusted corresponds to one reference point. In other possible embodiments, the point to be adjusted corresponds to at least two reference points.

[0057] In fact, for some key points corresponding to the target object (e.g., face) mentioned above, these key points may be concentrated in different areas (e.g., eye area, eyebrow area, etc.), and there may be a structural constraint relationship between the key points in the same area, but there is no structural constraint relationship between the key points in different areas. Therefore, in order to better improve the structural constraint effect, the present disclosure also provides a possible implementation method of the process of determining the above-mentioned points to be adjusted and their corresponding reference points, which may specifically include the following steps 11-12.

[0058] Step 11: Divide the key point prediction results of the above image to be processed to obtain at least one group, which includes the position description data of at least one key point. The structural division areas corresponding to the key points in the same group are the same, and the structural division areas corresponding to the key points in different groups are different.

[0059] Among them, the division processing is used to divide the position description data of the key points with structural associations in the key point prediction results into the same group, so that one group can be used to record the position description data of the key points appearing in a certain structural division area, so that different groups can represent the distribution of key points presented in different structural division areas. The structural division area refers to the area in the image to be processed above that is used to aggregate and display multiple key points with structural associations. It should be noted that the structural association relationship can be set in advance based on the actual application scenario.

[0060] Based on the content in the previous paragraph, it can be seen that the multiple groups obtained by division have the following characteristics: each group can include the location description data of at least one key point; the structural division areas corresponding to the key points in the same group are the same (that is, for any group, all key points in the group belong to the same structural division area); the structural division areas corresponding to the key points in different groups are different.

[0061] In addition, the embodiments of the present disclosure do not limit the implementation methods of the above-mentioned structural division areas. For example, when the key point prediction results of the above-mentioned image to be processed are used to describe some key points corresponding to the target object (for example, the body of an animal), at least one structural division area corresponding to the target object may include at least one part description area of ​​the target object (for example, a head description area, an arm description area, a leg description area, etc.).

[0062] Based on the content of the above paragraph, it can be seen that in one possible implementation, when the target object is a face, the key point prediction result of the image to be processed can be used to describe some key points corresponding to the face (that is, the multiple key points described by the key point prediction result all belong to the face), and the at least one structural division area corresponding to the face can include one or more of the eye description area, the eyebrow description area, the nose description area, the mouth description area, and the facial contour description area. It can be seen that the multiple groups determined based on the at least one structural division area corresponding to the face can respectively include one or more of the group corresponding to the eye description area, the group corresponding to the eyebrow description area, the group corresponding to the nose description area, the group corresponding to the mouth description area, and the group corresponding to the facial contour description area. Among them, the group corresponding to the eye description area is used to record the distribution of key points presented in the eye description area, so that the key points in the group corresponding to the eye description area are all located within the eye description area; the group corresponding to the eyebrow description area is used to record the distribution of key points presented in the eyebrow description area, so that the key points in the group corresponding to the eyebrow description area are all located within the eyebrow description area; the group corresponding to the nose description area is used to record the distribution of key points presented in the nose description area, so that the key points in the group corresponding to the nose description area are all located within the nose description area; the group corresponding to the mouth description area is used to record the distribution of key points presented in the mouth description area, so that the key points in the group corresponding to the mouth description area are all located within the mouth description area; the group corresponding to the facial contour description area is used to record the distribution of key points presented in the facial contour description area, so that the key points in the group corresponding to the facial contour description area are all located within the facial contour description area.

[0063] Based on the relevant content of step 11 above, it can be known that for the above image to be processed, after obtaining the key point prediction result of the image to be processed, the key point prediction result can be divided and processed to obtain at least one group, so that each group includes the location description data of at least one key point, and different groups are used to describe the distribution of key points in different structural division areas, so that key point pairs with structural association relationships can be determined based on the grouping later.

[0064] Step 12: For any group, determine the point to be adjusted and the reference point corresponding to the point to be adjusted based on the key point pairs with structural constraint relationships in the group.

[0065] In the present disclosure, for the j-th key point pair with a structural constraint relationship in the m-th group, the key point in the key point pair that does not meet the preset reference condition can be determined as the point to be adjusted, and the key point in the key point pair that meets the preset reference condition can be determined as the reference point corresponding to the point to be adjusted, so that the predicted position adjustment processing for one or two key points in the key point pair can be completed with the help of the adjustment process shown below (for example, the adjustment process shown in S2 below or the adjustment process shown in steps 52-53 below). Wherein, m is a positive integer, m≤the number of groups in the above "at least one group"; j is a positive integer, j≤the number of key point pairs with a structural constraint relationship in the m-th group. The preset reference condition refers to the condition met by the key point that can be used as a reference point; and the preset reference condition can be set in advance according to the application scenario. For example, the preset reference condition can be: a key point with obvious visual characteristics that exists in a key point pair with a structural constraint relationship (for example, a key point that exists in the key point pair with a structural constraint relationship and whose point identification data belongs to a preset anchor point identification set, etc.); the anchor point identification set is used to record the identification of the key point with obvious visual characteristics; and the anchor point identification set can be set in advance according to the application scenario.

[0066] Based on the content of the above paragraph, it can be seen that in some possible implementations, the mth group above may include the point to be adjusted and a reference point corresponding to the point to be adjusted, so that the position adjustment processing of the point to be adjusted can be performed subsequently based on the position information of this reference point. In addition, in other possible implementations, the mth group above may include the point to be adjusted and multiple reference points corresponding to the point to be adjusted (for example, 2 reference points, or 3 reference points, etc.), so that the position adjustment processing of the point to be adjusted can be performed subsequently based on the position information of these reference points and the weights corresponding to these reference points.

[0067] Based on the relevant contents of steps 11 to 12 above, it can be known that in one possible implementation mode, after obtaining the key point prediction results of the image to be processed, the key point prediction results can be divided and processed to obtain at least one group (for example, a group corresponding to the eye description area, a group corresponding to the eyebrow description area, a group corresponding to the nose description area, a group corresponding to the mouth description area, and a group corresponding to the facial contour description area, etc.), so that different groups are used to represent the distribution of key points in different structural areas; and then the key point pairs with structural constraint relationships in each group are used to determine the points to be adjusted and the reference points with corresponding relationships, so that the predicted positions of some or all key points in the key point prediction results can be adjusted based on the points to be adjusted and the reference points with corresponding relationships. In this way, the independence between different structural areas can be effectively ensured, thereby ensuring that each structural area has sufficient deformation ability, and thus effectively avoiding the mutual influence between different structural areas, which is conducive to improving the structural constraint effect.

[0068] Based on the relevant content of S1 above, it can be known that in some application scenarios, for the image to be processed (for example, any frame of video image in a certain video data), the key point prediction result of the image to be processed can be obtained first, so that the key point prediction result can describe the distribution of key points presented in the image to be processed; then, the key point prediction result is divided into at least one group, so that different groups are used to describe the distribution of key points in different structural division areas; then, using the key point pairs with structural constraint relationships in each group, some corresponding points to be adjusted and reference points are determined, so that the predicted position adjustment processing of some or all key points in the key point prediction result can be realized based on these corresponding points to be adjusted and reference points.

[0069] S2: Adjust the position description data of the point to be adjusted in the key point prediction result of the image to be processed based on the position description data of the reference point.

[0070] It should be noted that the present disclosure does not limit the implementation method of the adjustment process of the position description data of the point to be adjusted involved in S2 above. For example, in some possible implementation methods, when a point to be adjusted corresponds to one reference point, S2 can specifically be: directly adjusting the position description data of the point to be adjusted in the key point prediction result of the image to be processed based on the position description data of the reference point. For example, in other possible implementation methods, when a point to be adjusted corresponds to at least two reference points, S2 can specifically be: adjusting the position description data of the point to be adjusted in the key point prediction result based on the position description data of each reference point corresponding to the point to be adjusted and the weight corresponding to each reference point. The weight corresponding to the reference point is used to characterize the degree of influence of the reference point in the process of adjusting the position of the point to be adjusted; and the present disclosure does not limit the method of obtaining the weight corresponding to the reference point. For example, it can be set in advance based on the application scenario. For example, the weight corresponding to the reference point can be determined based on the distance between the reference point and the point to be adjusted, and the distance is determined based on the position description data of the reference point and the position description data of the point to be adjusted, so that the distance can represent the relative distance between the position described by the position description data of the reference point and the position described by the position description data of the point to be adjusted.

[0071] In order to facilitate understanding of the above content, the following is an explanation based on some situations.

[0072] Case 1: In some application scenarios (for example, when the target object is an object), the relative offsets between the key point pairs with structural constraints may remain unchanged (for example, the relative offsets between all key points in the handlebar description area of ​​a bicycle usually do not change over time).

[0073] Based on the above situation 1, the present disclosure provides a possible implementation method of the adjustment process of the position description data of the point to be adjusted above. Under this implementation method, for the point to be adjusted and the reference point having a corresponding relationship, the adjustment process of the position description data of the point to be adjusted can specifically include the following steps 21-22.

[0074] Step 21: Determine the inter-point offset between the point to be adjusted and the reference point based on the position description data of the reference point and the position description data of the point to be adjusted.

[0075] The inter-point offset refers to the relative offset between the predicted position of the point to be adjusted and the predicted position of the reference point, so that the inter-point offset can represent the predicted offset of the point to be adjusted relative to the reference point.

[0076] It should be noted that the present disclosure does not limit the implementation method of the offset. For example, when the position description data of a key point is implemented using position coordinates, the offset can be implemented using the distance between the two position coordinates, so that the offset can represent the relative distance between the two key points. For another example, when the position description data of a key point is implemented using position coordinates, the offset can also be implemented using the coordinate difference between the two position coordinates, so that the offset can represent the relative offset of the two key points on different coordinate axes.

[0077] In addition, the present disclosure does not limit the implementation method of the above step 21. For example, when a point to be adjusted corresponds to K reference points, and K is a positive integer (for example, K = 1, or 2, or 3, ...), the step 21 can be specifically: based on the position description data of the kth reference point corresponding to the point to be adjusted and the position description data of the point to be adjusted, determine the inter-point offset between the point to be adjusted and the kth reference point, where k is a positive integer, k≤K.

[0078] Step 22: Determine offset difference characterization data using the standard offset between the point to be adjusted and the reference point, and the inter-point offset between the point to be adjusted and the reference point.

[0079] The standard offset between the point to be adjusted and the reference point refers to an offset pre-set for the point to be adjusted and the reference point, so that the standard offset can represent the positional offset between the point to be adjusted and the reference point under standard conditions.

[0080] The offset difference characterization data is used to represent the difference between the above standard offset and the above inter-point offset; and the present disclosure does not limit the determination process of the offset difference characterization data. For example, it can be specifically: the standard offset and the inter-point offset are subtracted to obtain the offset difference characterization data.

[0081] In addition, the present disclosure does not limit the implementation method of the above step 22. For example, when a point to be adjusted corresponds to K reference points, and K is a positive integer (for example, K = 1, or 2, or 3, ...), the step 22 can be specifically: using the standard offset between the point to be adjusted and the kth reference point corresponding to the point to be adjusted, and the inter-point offset between the point to be adjusted and the kth reference point corresponding to the point to be adjusted, determine the kth offset difference characterization data corresponding to the point to be adjusted, where k is a positive integer, k≤K.

[0082] Step 23: Based on the above offset difference representation data, update the position description data of the point to be adjusted in the key point prediction result of the above image to be processed.

[0083] It should be noted that the present disclosure does not limit the implementation method of the above step 23. For example, when a point to be adjusted corresponds to 1 reference point, and the above offset difference characterization data refers to the difference between the above standard offset and the above point offset, the step 23 can be specifically as follows: first, the position description data of the above point to be adjusted and the offset difference characterization data are summed to obtain the adjusted position description data of the point to be adjusted, so that the relative offset between the adjusted position description data of the point to be adjusted and the position description data of the above reference point reaches the relative offset required under the standard situation; then, the adjusted position description data of the point to be adjusted is used to replace the position description data of the point to be adjusted in the key point prediction result of the above image to be processed to obtain an updated key point prediction result, so that the updated key point prediction result includes the adjusted position description data of the point to be adjusted, thereby making the updated key point prediction result have better key point position determination stability.

[0084] For example, when a point to be adjusted corresponds to K reference points, and the first offset difference characterization data corresponding to the point to be adjusted, the second offset difference characterization data corresponding to the point to be adjusted, ... (and so on), and the K-th offset difference characterization data corresponding to the point to be adjusted are obtained using step 22 above, the above step 23 can be specifically as follows: first perform weighted averaging processing on the first offset difference characterization data corresponding to the point to be adjusted, the second offset difference characterization data corresponding to the point to be adjusted, ... (and so on), and the K-th offset difference characterization data corresponding to the point to be adjusted to obtain the final offset difference characterization data, wherein, for the weighted averaging processing, the weight corresponding to the k-th offset difference characterization data corresponding to the point to be adjusted is the weight corresponding to the k-th reference point, k is a positive integer, k≤K. Then, the final offset difference characterization data is added to the position description data of the point to be adjusted to obtain the adjusted position description data of the point to be adjusted; and then the adjusted position description data of the point to be adjusted is used to replace the position description data of the point to be adjusted in the key point prediction result of the above processed image to obtain an updated key point prediction result, so that the updated key point prediction result includes the adjusted position description data of the point to be adjusted.

[0085] Based on the relevant contents of steps 21 to 23 above, it can be known that in one possible implementation mode, for any pair of corresponding points to be adjusted and reference points, the position description data of the point to be adjusted can be adjusted based on the position description data of the reference point and the standard offset between the point to be adjusted and the reference point, so that the relative offset between the adjusted position description data of the point to be adjusted and the position description data of the reference point reaches the relative offset described by the standard offset.

[0086] Case 2: In some application scenarios, the relative offset between some key point pairs with structural constraint relationships may not change significantly in a short period of time, so that the relative offset between the key pairs described by some adjacent image data in the image sequence is almost unchanged.

[0087] Based on the above-mentioned situation 2, the present disclosure provides a possible implementation method of the adjustment process of the position description data of the above-mentioned point to be adjusted. Under this implementation method, when the above-mentioned image to be processed refers to image data extracted from an image sequence, for the point to be adjusted and the reference point having a corresponding relationship, the adjustment process of the position description data of the point to be adjusted can specifically include the following steps 31 to 33.

[0088] Step 31: Determine the inter-point offset between the point to be adjusted and the reference point based on the position description data of the reference point and the position description data of the point to be adjusted.

[0089] It should be noted that for the relevant content of step 31, please refer to the relevant content of step 21 above. For the sake of brevity, it will not be repeated here.

[0090] Step 32: Using the historical offset between the point to be adjusted and the reference point, the offset between the above points is corrected to obtain a corrected offset; the historical offset is determined based on the historical image corresponding to the image to be processed.

[0091] Among them, the historical offset between the point to be adjusted and the reference point is used to describe the relative offset between the point to be adjusted and the reference point within the historical time period. It should be noted that the present disclosure does not limit the historical time period. For example, the historical time period may include the time point corresponding to the previous frame image corresponding to the image to be processed; and the previous frame image and the image to be processed belong to the same image sequence, the time sequence corresponding to the previous frame image in the image sequence is adjacent to the time sequence corresponding to the image to be processed in the image sequence, and the time sequence corresponding to the previous frame image in the image sequence is earlier than the time sequence corresponding to the image to be processed in the image sequence. For another example, the historical time period may include the time points corresponding to the previous few frames of images corresponding to the image to be processed.

[0092] In addition, the historical offset between the above-mentioned point to be adjusted and the reference point is determined based on the historical image corresponding to the image to be processed. The historical image is used to describe the position distribution of multiple key points in the above-mentioned image to be processed within the historical time period; and the present disclosure does not limit the implementation method of the historical image. For example, when the image to be processed refers to image data extracted from an image sequence (for example, video data), the historical image corresponding to the image to be processed may include one or more image data existing in the image sequence and having a temporal sequence earlier than that of the image to be processed. It can be seen that in one possible implementation method, when the image to be processed refers to the t-th frame video image in a video data, the historical image corresponding to the image to be processed may include one or more of the t-1-th frame video image, the t-2-th frame video image, ... (and so on), and the 1st frame video image in the video data. Wherein, t is a positive integer.

[0093] Based on the above content, it can be seen that for the historical image corresponding to the image to be processed above, the historical image and the image to be processed belong to the same image sequence; and the corresponding time sequence of the historical image in the image sequence is earlier than the corresponding time sequence of the image to be processed in the image sequence.

[0094] In addition, the present disclosure does not limit the number of images in the historical images corresponding to the image to be processed in the above paragraph.

[0095] In addition, the present disclosure does not limit the process of determining the historical offset between the above-mentioned point to be adjusted and the reference point. For example, the process of determining the historical offset may include: first performing key point prediction processing on the historical image corresponding to the image to be processed to obtain the key point prediction result of the historical image, so that the key point prediction result can represent the predicted position distribution of the above-mentioned multiple key points in the historical time period; then extracting the position prediction result of the above-mentioned reference point and the position prediction result of the above-mentioned point to be adjusted from the key point prediction result of the historical image; then, the relative offset between the position prediction result of the point to be adjusted and the position prediction result of the reference point is regarded as the historical offset between the point to be adjusted and the reference point (for example, the following ). The position prediction result of the reference point refers to the position information predicted for the reference point within the historical time period; the position prediction result of the point to be adjusted refers to the position information predicted for the point to be adjusted within the historical time period.

[0096] For another example, the process of determining the historical offset between the above-mentioned point to be adjusted and the reference point may include: first performing key point prediction processing on the previous frame image corresponding to the image to be processed (for example, the t-1th frame video image) to obtain the key point prediction result of the previous frame image, so that the key point prediction result can represent the position distribution of the above multiple key points predicted in the previous frame image; then extracting the position prediction result of the above reference point and the position prediction result of the above-mentioned point to be adjusted from the key point prediction result of the previous frame image; secondly, calculating the relative offset between the position prediction result of the point to be adjusted and the position prediction result of the reference point, so that the relative offset can represent the offset of the point to be adjusted relative to the reference point prediction in the previous frame image; then, using the historical offset corresponding to the relative offset, performing correction processing on the relative offset to obtain the "historical offset between the point to be adjusted and the reference point" (for example, the following ). The historical offset corresponding to the relative offset is determined based on the historical image corresponding to the previous frame (eg, the t-2th frame of video image).

[0097] Based on the above two paragraphs, it can be seen that in a possible implementation manner, when the above-mentioned image to be processed refers to the t-th frame video image in a video data, and the historical image corresponding to the image to be processed includes the t-1-th frame video image in the video data, the determination process of the "historical offset between the point to be adjusted and the reference point" in the above step 32 includes: first using the data processing method provided by the present disclosure (for example, S1-S2 or the following steps 51-53), a key point position determination process is performed on the t-1-th frame video image to obtain position description information of multiple key points in the t-1-th frame video image, so that these position description information can represent the positions of these key points in the t-1-th frame video image; then extracting the position description information of the above-mentioned reference point and the position description information of the above-mentioned point to be adjusted from these position description information; finally, determining the "historical offset between the point to be adjusted and the reference point" based on the relative offset between the position description information of the point to be adjusted and the position description information of the reference point (and its corresponding historical offset).

[0098] The corrected offset refers to the result obtained by correcting the "inter-point offset between the point to be adjusted and the reference point" mentioned above; and the present disclosure does not limit the determination process of the corrected offset. For example, it can be implemented using the following formulas (1)-(2), or using the following formulas (1) and (3).

[0099] Where, Represents the position description data of the i-th point to be adjusted in the t-th video image (for example, the predicted position coordinates); The position description data of the reference point corresponding to the i-th point to be adjusted (for example, the predicted position coordinates); represents an inter-point offset between the i-th point to be adjusted and a reference point corresponding to the i-th point to be adjusted, determined based on the t-th video image; represents the inter-point offset between the i-th point to be adjusted and the reference point corresponding to the i-th point to be adjusted, determined based on the t-1-th video image; represents the corrected offset between the i-th point to be adjusted and the reference point corresponding to the i-th point to be adjusted, determined based on the t-th video image; represents the corrected offset between the i-th point to be adjusted and the reference point corresponding to the i-th point to be adjusted, determined based on the t-1-th video image; μ represents a preset smoothing coefficient.

[0100] Based on the relevant content of step 32 above, it can be known that in some application scenarios, for the current frame (for example, the image to be processed above or the t-th frame of the video image above), an inter-point offset (for example, ) can then be used to determine the offset based on the historical frame (e.g., similar to etc. based on the offset between points determined by the historical frame or similar The offset between points determined based on the current frame is smoothed to obtain the offset between the point to be adjusted and the reference point corresponding to the point to be adjusted.

[0101] In addition, the present disclosure does not limit the implementation method of the above step 32. For example, when a point to be adjusted corresponds to K reference points, and K is a positive integer (for example, K = 1, or 2, or 3, ...), the step 32 can specifically be: using the historical offset between the point to be adjusted and the kth reference point corresponding to the point to be adjusted, the point-to-point offset between the point to be adjusted and the kth reference point is corrected to obtain the kth corrected offset corresponding to the point to be adjusted, where k is a positive integer, k≤K.

[0102] Step 33: Based on the corrected offset and the position description data of the reference point, the position description data of the point to be adjusted in the key point prediction result of the image to be processed is updated.

[0103] It should be noted that the present disclosure does not limit the implementation method of the above step 33. For example, when a point to be adjusted corresponds to 1 reference point, the step 33 can be specifically as follows: summing the above corrected offset and the position description data of the above reference point (as shown in formula (4) below) to obtain the adjusted position description data of the above point to be adjusted; and then using the adjusted position description data of the point to be adjusted to replace the position description data of the point to be adjusted in the key point prediction result of the above image to be processed, so that the updated key point prediction result includes the adjusted position description data of the point to be adjusted.

[0104] Where, Represents the adjusted position description data of the i-th point to be adjusted in the t-th video image; represents the corrected offset between the i-th point to be adjusted and the reference point corresponding to the i-th point to be adjusted, determined based on the t-th video image; The position description data (eg, the predicted position coordinates) of the reference point corresponding to the i-th point to be adjusted.

[0105] For another example, when a point to be adjusted corresponds to K reference points, the above step 33 may specifically be: first, summing the kth corrected offset corresponding to the point to be adjusted and the position description data of the kth reference point corresponding to the point to be adjusted to obtain the kth adjusted position description data corresponding to the point to be adjusted, where k is a positive integer and k≤K; then, performing weighted averaging on the first adjusted position description data corresponding to the point to be adjusted, the second adjusted position description data corresponding to the point to be adjusted, ... (and so on), and the Kth adjusted position description data corresponding to the point to be adjusted to obtain the final adjusted position description data of the point to be adjusted, wherein, for the weighted averaging, the weight corresponding to the kth adjusted position description data corresponding to the point to be adjusted is the weight corresponding to the kth reference point, where k is a positive integer and k≤K; and then using the final adjusted position description data of the point to be adjusted to replace the position description data of the point to be adjusted in the key point prediction result of the image to be processed, so that the updated key point prediction result includes the final adjusted position description data of the point to be adjusted.

[0106] Based on the relevant contents of steps 31 to 33 above, it can be known that, under a possible implementation mode, for any pair of corresponding points to be adjusted and reference points, the relative offset between the position description data of the point to be adjusted and the position description data of the reference point can be calculated first; then, based on the historical offset between the two points, the relative offset is smoothed to obtain the corrected offset, so that the corrected offset can better represent the relative offset of the two points in the above image to be processed; finally, the sum of the corrected offset and the position description data of the reference point is used as the adjusted position description data of the point to be adjusted, so that the adjusted position description data can better represent the position of the point to be adjusted in the image to be processed.

[0107] Case 3: In some application scenarios, in order to better improve the stability of key point position determination, the key point information determined for the above historical image data can also be used to constrain the position description data of the above reference point, so that the position description data of the reference point can better represent the position of the reference point in the image to be processed, so that the position description data of the above point to be adjusted can be better adjusted based on the position description data of the reference point.

[0108] Based on the above situation 3, the present disclosure provides a possible implementation method of the adjustment process of the position description data of the above-mentioned point to be adjusted. Under this implementation method, when the above-mentioned image to be processed refers to image data extracted from an image sequence, for the point to be adjusted and the reference point having a corresponding relationship, the adjustment process of the position description data of the point to be adjusted can specifically include the following steps 41 to 44.

[0109] Step 41: Determine the inter-point offset between the point to be adjusted and the reference point based on the position description data of the reference point and the position description data of the point to be adjusted.

[0110] It should be noted that for the relevant content of step 41, please refer to the relevant content of step 21 above. For the sake of brevity, it will not be repeated here.

[0111] Step 42: Using the historical offset between the point to be adjusted and the reference point, the offset between the above points is corrected to obtain a corrected offset; the historical offset is determined based on the historical image corresponding to the image to be processed.

[0112] It should be noted that for the relevant content of step 42, please refer to the relevant content of step 32 above. For the sake of brevity, it will not be repeated here.

[0113] Step 43: Using the historical position data of the reference point, the position description data of the reference point is corrected to obtain the corrected position data of the reference point; the historical position data is determined based on the historical image corresponding to the image to be processed.

[0114] The historical location data of the reference point is used to describe the location of the reference point within a historical time period; and the historical location data of the reference point is determined based on the historical image corresponding to the image to be processed. It should be noted that the relevant content of the historical image corresponding to the image to be processed is mentioned above.

[0115] In addition, the present disclosure does not limit the process of determining the historical position data of the above reference point. For example, it can be specifically as follows: first, key point prediction processing is performed on the historical image corresponding to the image to be processed to obtain the key point prediction result of the historical image, so that the key point prediction result can represent the predicted position distribution of the above multiple key points in the historical time period; then, the position prediction result of the above reference point extracted from the key point prediction result of the historical image is used as the historical position data of the reference point, so that the historical position data can represent the position information predicted for the reference point in the historical time period.

[0116] For another example, when the historical image corresponding to the image to be processed (e.g., the t-th frame of video image) includes the previous frame of image corresponding to the image to be processed (e.g., the t-1-th frame of video image), the process of determining the historical position data of the reference point includes: first performing key point prediction processing on the previous frame of image to obtain a key point prediction result for the previous frame of image, so that the key point prediction result can represent the position distribution of the multiple key points predicted in the previous frame of image; then extracting the position prediction result of the reference point from the key point prediction result of the previous frame of image, so that the position prediction result can represent the position predicted for the reference point when the data processing was performed on the previous frame of image; then, using the historical position data corresponding to the position prediction result of the reference point, correcting the position prediction result of the reference point to obtain the historical position data of the reference point, so that the historical position data of the reference point can better represent the position of the reference point within the historical time period. The historical position data corresponding to the position prediction result of the reference point is determined based on the historical image corresponding to the previous frame of image (e.g., the t-2-th frame of video image).

[0117] Based on the above two paragraphs, it can be seen that in some application scenarios, for an image sequence (such as video data), when data processing is being performed on the t-th image data in the image sequence, key point prediction processing can be first performed on the t-th image data to obtain the key point prediction result of the t-th image data, so that the key point prediction result includes the position description data of some reference points; then, for any reference point, the historical position data of the reference point can be used to correct the position description data of the reference point to obtain the corrected position data of the reference point, so that the corrected position data of the reference point can better represent the position of the reference point in the t-th image data. The historical position data of the reference point can be the position description data predicted for the reference point when data processing is performed on the t-1-th image data in the image sequence; or the historical position data of the reference point can also be the corrected position data obtained by correcting the position description data predicted for the reference point when data processing is performed on the t-1-th image data in the image sequence.

[0118] It should be noted that the present disclosure does not limit the implementation method of the "correction processing" in the above step 43. For example, it can be implemented by adopting any existing or future method that can perform smoothing processing on multiple data.

[0119] In addition, the present disclosure does not limit the above step 43. For example, when a point to be adjusted corresponds to K reference points, and K is a positive integer (for example, K = 1, or 2, or 3, ...), the step 43 can be specifically: using the historical position data of the kth reference point corresponding to the point to be adjusted, the position description data of the kth reference point is corrected to obtain the corrected position data of the kth reference point, where k is a positive integer, k≤K.

[0120] Based on the relevant content of step 43 above, it can be known that for the image to be processed, after obtaining the key point prediction result of the image to be processed, the position description data of the reference point can be first extracted from the key point prediction result; then, using the historical position data of the reference point, the position description data of the reference point is corrected (for example, smoothed) to obtain the corrected position data of the reference point, so that the corrected position data can better represent the position of the reference point in the image to be processed, which is conducive to improving the stability of the position determination of the reference point.

[0121] It should be noted that the present disclosure does not limit the execution time of the above step 43. It only needs to ensure that the execution time of the step 43 is earlier than the execution time of the following step 44 and later than the execution time of the above step S1.

[0122] Step 44: using the corrected offset and the corrected position data of the reference point, update the position description data of the point to be adjusted in the key point prediction result of the image to be processed.

[0123] It should be noted that the present disclosure does not limit the implementation method of the above step 33. For example, when a point to be adjusted corresponds to one reference point, the step 44 can be specifically as follows: summing the above corrected offset and the above corrected position data of the reference point (as shown in formula (5) below) to obtain the adjusted position description data of the above point to be adjusted; and then using the adjusted position description data of the point to be adjusted to replace the position description data of the point to be adjusted in the key point prediction result of the above image to be processed, so that the updated key point prediction result includes the adjusted position description data of the point to be adjusted.

[0124] Where, Represents the adjusted position description data of the i-th point to be adjusted in the t-th video image; represents the corrected offset between the i-th point to be adjusted and the reference point corresponding to the i-th point to be adjusted, determined based on the t-th video image; Represents the corrected position data (eg, predicted position coordinates) of the reference point corresponding to the i-th point to be adjusted.

[0125] For another example, when a point to be adjusted corresponds to K reference points, the above step 44 may specifically be: summing the kth corrected offset corresponding to the point to be adjusted and the corrected position data of the kth reference point corresponding to the point to be adjusted to obtain the kth adjusted position description data corresponding to the point to be adjusted, where k is a positive integer and k≤K; then, performing weighted averaging on the first adjusted position description data corresponding to the point to be adjusted, the second adjusted position description data corresponding to the point to be adjusted, ... (and so on), and the Kth adjusted position description data corresponding to the point to be adjusted to obtain the final adjusted position description data of the point to be adjusted, wherein, for the weighted averaging, the weight corresponding to the kth adjusted position description data corresponding to the point to be adjusted is the weight corresponding to the kth reference point, where k is a positive integer and k≤K; and replacing the position description data of the point to be adjusted in the key point prediction result of the above image to be processed with the final adjusted position description data of the point to be adjusted, so that the updated key point prediction result includes the final adjusted position description data of the point to be adjusted.

[0126] Based on the relevant contents of steps 41 to 44 above, it can be known that in one possible implementation mode, for any pair of corresponding points to be adjusted and reference points, the relative offset between the position description data of the point to be adjusted and the position description data of the reference point can be calculated first; then, based on the historical offset between the two points, the relative offset is smoothed to obtain a corrected offset, so that the corrected offset can better represent the relative offset of the two points in the above image to be processed, so that after the position description data of the reference point is corrected using the historical position data of the reference point to obtain the corrected position data of the reference point, the sum of the corrected offset and the corrected position data of the reference point is used as the adjusted position description data of the point to be adjusted, so that the adjusted position description data can better represent the position of the point to be adjusted in the image to be processed.

[0127] Based on the relevant contents of S1 to S2 above, it can be seen that for the data processing method provided by the embodiment of the present disclosure, the key point prediction results of the image to be processed (for example, any frame of video image in a certain video data) are obtained so that the key point prediction results include position description data of multiple key points, so that when it is determined that the multiple key points include reference points and points to be adjusted with structural constraint relationships, the position description data of the points to be adjusted are adjusted based on the position description data of the reference points, so that the adjusted position description data can better describe the position of the point to be adjusted, thereby effectively avoiding the visual jitter phenomenon caused by fluctuations in the prediction results for the points to be adjusted, thereby helping to improve the stability of key point position determination.

[0128] In addition, based on the relevant contents of S1 to S2 above, it can be seen that in some application scenarios, in order to better improve the stability of key point position determination, after obtaining the key point prediction results, the position description data of some points to be adjusted in the key point prediction results can be adjusted so that the adjusted key point prediction results at least include the adjusted position description data of these points to be adjusted (for example, the adjusted key point prediction results can include the unabridged position description data of all reference points, and the adjusted position description data of all points to be adjusted, etc.).

[0129] In fact, in other application scenarios, to further improve the stability of key point position determination, after obtaining the key point prediction results, not only can the predicted positions of some to-be-adjusted points in the key point prediction results be adjusted, but also the predicted positions of some reference points in the key point prediction results can be adjusted with reference to historical information. This allows the adjusted key point prediction results to include the adjusted position description data for these to-be-adjusted points and the adjusted position description data for these reference points. Based on this, the present disclosure also provides a possible implementation of the above data processing method, which may include steps 51-53 below.

[0130] Step 51: Obtain key point prediction results of the image to be processed, where the key point prediction results include position description data of multiple key points; the multiple key points include points to be adjusted and reference points corresponding to the points to be adjusted.

[0131] It should be noted that, for the relevant content of step 51, please refer to the relevant content of S1 above, and for the sake of brevity, it will not be repeated here.

[0132] Step 52: adjusting the position description data of the point to be adjusted in the key point prediction result of the image to be processed based on the position description data of the reference point, so that the adjusted key point prediction result includes the adjusted position description data of the point to be adjusted.

[0133] It should be noted that, for the relevant content of step 52, please refer to the relevant content of S2 above, and for the sake of brevity, it will not be repeated here.

[0134] Step 53: Using the historical position data of the reference point above, adjust the position description data of the reference point in the key point prediction result of the image to be processed above, so that the adjusted key point prediction result includes the adjusted position description data of the reference point; the historical position data is determined based on the historical image corresponding to the image to be processed.

[0135] It should be noted that the present disclosure does not limit the implementation method of the adjustment process in step 53 above. For example, it can be implemented using the correction processing process shown in step 43 above. For the sake of brevity, it will not be repeated here.

[0136] It should also be noted that the present disclosure does not limit the execution time of the above step 53, as long as the execution time of the above step 53 is ensured to be later than the execution time of the above step 51.

[0137] Based on the relevant contents of steps 51 to 53 above, it can be known that, in a possible implementation mode, for any image data in a certain image sequence (for example, the t-th image data), key point prediction processing can be first performed on the image data to obtain the key point prediction result of the image data, so that the key point prediction result includes position description data of multiple key points; the position description data of these key points are then grouped to obtain groups corresponding to each structural area; then, for the group corresponding to each structural area, the points to be adjusted and the reference points with corresponding relationships in the group are determined; subsequently, for each reference point, the historical position data of the reference point is used to smooth the position description data of the reference point to obtain the smoothed position description data of the reference point (for example, the corrected position data of the reference point above), so that the smoothed position description data of the reference point can better represent the reference point in the image data. The position of the point to be adjusted; and, for each point to be adjusted, first calculate the relative offset between the point to be adjusted and the reference point corresponding to the point to be adjusted; then use the historical offset between the point to be adjusted and the reference point corresponding to the point to be adjusted to smooth the relative offset to obtain a smoothed offset (for example, the offset after correction mentioned above); then, based on the smoothed position description data of the reference point and the smoothed offset, calculate the adjusted position description data of the point to be adjusted, so that the adjusted position description data of the point to be adjusted can better represent the position of the point to be adjusted in the image data; finally, output the smoothed position description data of all reference points and the adjusted position description data of all points to be adjusted as the key point information determined for the image data, so that the key point information can be displayed to the user later or other data processing can be performed based on the key point information, thereby effectively improving the stability of key point position determination.

[0138] Based on the relevant content of the data processing method provided in this disclosure, it can be seen that the data processing method has at least one of the following advantages:

[0139] ① The present invention selects key points with strong recognition (that is, key points with obvious visual features) and uses the positions of these key points with strong recognition as a benchmark to adjust the positions of other key points with weaker recognition, which is conducive to obtaining more accurate and stable key point information.

[0140] ② The present invention adjusts the positions of key points with weaker recognition based on the structural constraint relationship between key points, so that the key point information finally obtained can maintain better structure, which is conducive to obtaining better visual stability.

[0141] ③ The present disclosure maintains the independence between various structural regions through grouping, so that each structural region has sufficient deformation ability and will not affect each other, which is conducive to obtaining better visual stability.

[0142] Based on the data processing method provided in the embodiments of the present disclosure, the embodiments of the present disclosure also provide a data processing device, which will be explained and illustrated below in conjunction with Figure 2. Figure 2 is a schematic diagram of the structure of a data processing device provided in the embodiments of the present disclosure. It should be noted that for the technical details of the data processing device provided in the embodiments of the present disclosure, please refer to the relevant content of the data processing method above.

[0143] As shown in FIG2 , the data processing device 200 provided in an embodiment of the present disclosure includes:

[0144] An acquisition unit 201 is configured to acquire key point prediction results of an image to be processed, wherein the key point prediction results include position description data of a plurality of key points; the plurality of key points include points to be adjusted and reference points corresponding to the points to be adjusted;

[0145] The adjusting unit 202 is configured to adjust the position description data of the point to be adjusted in the key point prediction result according to the position description data of the reference point.

[0146] In a possible implementation manner, the adjusting unit 202 includes:

[0147] a data determination subunit, configured to determine an inter-point offset between the point to be adjusted and the reference point based on the position description data of the reference point and the position description data of the point to be adjusted;

[0148] a first correction subunit, configured to correct the inter-point offset using a historical offset between the point to be adjusted and the reference point to obtain a corrected offset; the historical offset being determined based on a historical image corresponding to the image to be processed;

[0149] The data updating subunit is configured to update the position description data of the point to be adjusted in the key point prediction result according to the corrected offset and the position description data of the reference point.

[0150] In a possible implementation manner, the adjusting unit 202 further includes:

[0151] a second correction subunit, configured to correct the position description data of the reference point using the historical position data of the reference point to obtain corrected position data of the reference point; the historical position data being determined based on a historical image corresponding to the image to be processed;

[0152] The data updating subunit is specifically configured to update the position description data of the point to be adjusted in the key point prediction result by using the corrected offset and the corrected position data of the reference point.

[0153] In one possible implementation, the adjustment unit 202 is also used to: use the historical position data of the reference point to adjust the position description data of the reference point in the key point prediction result; the historical position data is determined based on the historical image corresponding to the image to be processed; the adjusted key point prediction result includes the adjusted position description data of the reference point and the adjusted position description data of the point to be adjusted.

[0154] In a possible implementation manner, the image to be processed and the historical image belong to the same image sequence; the time sequence corresponding to the historical image in the image sequence is earlier than the time sequence corresponding to the image to be processed in the image sequence.

[0155] In a possible implementation manner, the data processing device 200 further includes:

[0156] a partitioning unit, configured to partition the key point prediction result to obtain at least one group, wherein the group includes position description data of at least one key point, the key points in the same group correspond to the same structural partition area, and the key points in different groups correspond to different structural partition areas;

[0157] The determining unit is configured to determine, for any of the groups, the points to be adjusted and the reference points corresponding to the points to be adjusted according to the key point pairs having a structural constraint relationship in the group.

[0158] In one possible implementation, if the multiple key points all belong to a face, then the at least one structural division area corresponding to the face includes one or more of an eye description area, an eyebrow description area, a nose description area, a mouth description area, and a facial contour description area.

[0159] In one possible implementation, the point to be adjusted corresponds to at least two reference points;

[0160] The adjusting unit 202 is specifically configured to adjust the position description data of the to-be-adjusted point in the key point prediction result according to the position description data of each reference point and the weight corresponding to each reference point.

[0161] In one possible implementation, for any reference point corresponding to the point to be adjusted, the weight corresponding to the reference point is determined based on the distance between the reference point and the point to be adjusted, and the distance is determined based on the position description data of the reference point and the position description data of the point to be adjusted.

[0162] Based on the relevant content of the above-mentioned data processing device 200, it can be known that for the data processing device 200 provided by the embodiment of the present disclosure, the key point prediction result of the image to be processed (for example, any frame of video image in a certain video data) is first obtained, so that the key point prediction result includes position description data of multiple key points; then, with reference to the position description data of some key points (that is, reference points) described by the key point prediction result, the position description data of another part of the key points (that is, the points to be adjusted) described by the key point prediction result is adjusted, so that the adjusted position description data can better describe the position of the other part of the key points, thereby effectively avoiding the visual jitter phenomenon caused by fluctuations in the prediction results for the other part of the key points, thereby helping to improve the stability of key point position determination.

[0163] In addition, an embodiment of the present disclosure also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes any implementation of the data processing method provided by the embodiment of the present disclosure.

[0164] Referring to FIG3 , a schematic diagram of the structure of an electronic device 300 suitable for implementing embodiments of the present disclosure is shown. Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG3 is merely an example and should not limit the functionality or scope of use of embodiments of the present disclosure.

[0165] As shown in Figure 3, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. Various programs and data required for the operation of the electronic device 300 are also stored in the RAM 303. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0166] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although FIG3 shows the electronic device 300 with various devices, it should be understood that not all of the devices shown are required to be implemented or present. More or fewer devices may alternatively be implemented or present.

[0167] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0168] The electronic device provided by the embodiment of the present disclosure and the method provided by the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0169] The embodiments of the present disclosure further provide a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes any implementation of the data processing method provided in the embodiments of the present disclosure.

[0170] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0171] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0172] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0173] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device can perform the method.

[0174] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0175] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0176] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit / module does not, in some cases, limit the unit itself.

[0177] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0178] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0179] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems or devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0180] It should be understood that in the present disclosure, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0181] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0182] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0183] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present disclosure. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments shown herein, but is intended to be construed in the widest manner consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, comprising: Acquire a key point prediction result of the image to be processed, wherein the key point prediction result includes position description data of a plurality of key points; The multiple key points include points to be adjusted and reference points corresponding to the points to be adjusted; The position description data of the point to be adjusted in the key point prediction result is adjusted according to the position description data of the reference point.

2. The method according to claim 1, wherein: The adjustment process of the position description data of the point to be adjusted includes: Determine the inter-point offset between the point to be adjusted and the reference point according to the position description data of the reference point and the position description data of the point to be adjusted; Using the historical offset between the point to be adjusted and the reference point, the offset between the points is corrected to obtain a corrected offset; the historical offset is determined based on a historical image corresponding to the image to be processed; The position description data of the point to be adjusted in the key point prediction result is updated according to the corrected offset and the position description data of the reference point.

3. The method according to claim 2, wherein: Before updating the position description data of the point to be adjusted in the key point prediction result according to the corrected offset and the position description data of the reference point, the method further includes: Using the historical position data of the reference point, the position description data of the reference point is corrected to obtain the corrected position data of the reference point; the historical position data is determined based on the historical image corresponding to the image to be processed; The updating of the position description data of the point to be adjusted in the key point prediction result according to the corrected offset and the position description data of the reference point comprises: The position description data of the point to be adjusted in the key point prediction result is updated by using the corrected offset and the corrected position data of the reference point.

4. The method according to claim 1, wherein: After obtaining the key point prediction result of the image to be processed, the method further includes: Using the historical position data of the reference point, adjust the reference point in the key point prediction result The location description data of the point; the historical location data is determined according to the historical image corresponding to the image to be processed; The adjusted key point prediction result includes the adjusted position description data of the reference point and the adjusted position description data of the point to be adjusted.

5. The method according to any one of claims 2 to 4, wherein: The image to be processed and the historical image belong to the same image sequence; The time sequence corresponding to the historical image in the image sequence is earlier than the time sequence corresponding to the image to be processed in the image sequence.

6. The method according to claim 1, wherein: Before adjusting the position description data of the point to be adjusted in the key point prediction result according to the position description data of the reference point, the method further includes: The key point prediction result is divided into at least one group, wherein the group includes the position description data of at least one key point, the key points in the same group correspond to the same structural division area, and the key points in different groups correspond to different structural division areas; For any of the groups, the points to be adjusted and the reference points corresponding to the points to be adjusted are determined according to the key point pairs having a structural constraint relationship in the group.

7. The method according to claim 6, wherein: If the multiple key points all belong to a face, then the at least one structural division area corresponding to the face includes one or more of an eye description area, an eyebrow description area, a nose description area, a mouth description area, and a facial contour description area.

8. The method according to claim 1, wherein: The points to be adjusted correspond to at least two reference points; The step of adjusting the position description data of the point to be adjusted in the key point prediction result according to the position description data of the reference point includes: According to the position description data of each reference point and the weight corresponding to each reference point, the position description data of the point to be adjusted in the key point prediction result is adjusted.

9. The method according to claim 8, wherein: For any reference point corresponding to the point to be adjusted, the weight corresponding to the reference point is determined based on the distance between the reference point and the point to be adjusted, and the distance is determined based on the position description data of the reference point and the position description data of the point to be adjusted.

10. A data processing device, comprising: An acquisition unit is configured to acquire a key point prediction result of the image to be processed, wherein the key point prediction result includes position description data of a plurality of key points; The multiple key points include points to be adjusted and reference points corresponding to the points to be adjusted; The adjustment unit is configured to adjust the position description data of the point to be adjusted in the key point prediction result according to the position description data of the reference point.

11. An electronic device, comprising: processor and memory; wherein, The memory is configured to store instructions or computer programs; The processor is configured to execute the instructions or computer programs in the memory so that the electronic device performs the method according to any one of claims 1 to 9.

12. A computer-readable medium, wherein: The computer-readable medium stores instructions or computer programs, and when the instructions or computer programs are executed on a device, the device executes the method according to any one of claims 1 to 9.

Images