Data processing method and device, electronic equipment and computer readable medium

Abstract

The invention discloses a data processing method and device, electronic equipment and a computer readable medium, and the method comprises the steps: firstly obtaining a key point prediction sequence of an image sequence and a key point tag sequence corresponding to the key point prediction sequence, so as to enable the key point prediction sequence to comprise a prediction track of at least one key point, the key point label sequence comprises label tracks of the key points; then, for any key point, determining trajectory similarity characterization data between the predicted trajectory of the key point and the label trajectory of the key point, so that the trajectory similarity characterization data can represent the similarity between the predicted trajectory of the key point and the label trajectory of the key point in the trajectory length and / or trajectory direction; and finally, according to the trajectory similarity representation data corresponding to all the key points, determining key point prediction performance corresponding to the key point prediction sequence, so that the key point prediction performance can better represent prediction performance presented when key point prediction processing is performed on one image sequence.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, electronic device, and computer-readable medium. Background Art

[0002] Key point prediction is a computer vision task, and key point prediction is used to locate certain key points (such as the key point of the corner of the eye, etc.) in an image data or an image sequence (such as video data, etc.).

[0003] For some application scenarios (such as training a machine learning model with key point prediction function), after performing key point prediction processing on an image sequence (such as video data) to obtain a key point prediction sequence, it may be necessary to further evaluate the status of the key point prediction sequence in certain aspects (such as stability, etc.). Summary of the invention

[0004] The present application provides a data processing method, device, electronic device, and computer-readable medium.

[0005] In order to achieve the above objectives, the technical solutions provided by this application are as follows:

[0006] The present application provides a data processing method, the method comprising:

[0007] Acquire a key point prediction sequence of an image sequence and a key point label sequence corresponding to the image sequence; the key point prediction sequence includes a prediction trajectory of at least one key point; the key point label sequence includes a label trajectory of the at least one key point;

[0008] For any of the key points, determining trajectory similarity characterization data between the predicted trajectory of the key point and the label trajectory of the key point; the trajectory similarity characterization data is used to indicate the degree of similarity between the predicted trajectory of the key point and the label trajectory of the key point in terms of trajectory length and / or trajectory direction;

[0009] The key point prediction performance corresponding to the key point prediction sequence is determined according to the trajectory similarity characterization data.

[0010] In a possible implementation manner, the image sequence includes a first image and a second image, a time sequence corresponding to the first image in the image sequence is earlier than a time sequence corresponding to the second image in the image sequence, and a time sequence corresponding to the first image in the image sequence is adjacent to a time sequence corresponding to the second image in the image sequence;

[0011] The predicted trajectory of the key point includes the predicted position corresponding to the key point in the first image and the predicted position corresponding to the key point in the second image;

[0012] The label track of the key point includes the label position corresponding to the key point in the first image and the label position corresponding to the key point in the second image;

[0013] The process of determining the trajectory similarity characterization data includes:

[0014] Calculating the distance between the predicted position corresponding to the key point in the second image and the predicted position corresponding to the key point in the first image to obtain a predicted length of the trajectory segment;

[0015] Calculate the distance between the label position corresponding to the key point in the second image and the label position corresponding to the key point in the first image to obtain the trajectory segment label length;

[0016] Determining length similarity representation data according to a difference between the predicted length of the trajectory segment and the label length of the trajectory segment;

[0017] The trajectory similarity characterization data is determined according to the length similarity characterization data.

[0018] In a possible implementation manner, the image sequence includes a first image and a second image, a time sequence corresponding to the first image in the image sequence is earlier than a time sequence corresponding to the second image in the image sequence, and a time sequence corresponding to the first image in the image sequence is adjacent to a time sequence corresponding to the second image in the image sequence;

[0019] The predicted trajectory of the key point includes the predicted position corresponding to the key point in the first image and the predicted position corresponding to the key point in the second image;

[0020] The label track of the key point includes the label position corresponding to the key point in the first image and the label position corresponding to the key point in the second image;

[0021] The process of determining the trajectory similarity characterization data includes:

[0022] Determining trajectory segment prediction direction representation data according to direction representation data between the predicted position corresponding to the key point in the second image and the predicted position corresponding to the key point in the first image;

[0023] Determine the trajectory segment label direction representation data according to the direction representation data between the label position corresponding to the key point in the second image and the label position corresponding to the key point in the first image;

[0024] Determining the direction similarity representation data according to the trajectory segment prediction direction representation data and the trajectory segment label direction representation data;

[0025] The trajectory similarity characterization data is determined according to the direction similarity characterization data.

[0026] In one possible implementation, the trajectory similarity representation data includes length similarity representation data and / or direction similarity representation data; the length similarity representation data is used to indicate the degree of similarity between the predicted trajectory of the key point and the label trajectory of the key point in trajectory length; the direction similarity representation data is used to indicate the degree of similarity between the predicted trajectory of the key point and the label trajectory of the key point in trajectory direction.

[0027] In a possible implementation manner, determining the key point prediction performance corresponding to the key point prediction sequence according to the trajectory similarity representation data includes:

[0028] Determining prediction stability characterization data corresponding to the key point prediction sequence according to the trajectory similarity characterization data;

[0029] The key point prediction performance corresponding to the key point prediction sequence is determined according to the prediction stability characterization data corresponding to the key point prediction sequence.

[0030] In a possible implementation manner, before determining the key point prediction performance corresponding to the key point prediction sequence according to the prediction stability characterization data corresponding to the key point prediction sequence, the method further includes:

[0031] Determining prediction accuracy characterization data corresponding to the key point prediction sequence according to the key point prediction sequence and the key point label sequence;

[0032] The determining, according to the prediction stability characterization data corresponding to the key point prediction sequence, the key point prediction performance corresponding to the key point prediction sequence comprises:

[0033] The key point prediction performance corresponding to the key point prediction sequence is determined according to the prediction stability characterization data corresponding to the key point prediction sequence and the prediction accuracy characterization data corresponding to the key point prediction sequence.

[0034] In a possible implementation manner, the step of acquiring a key point prediction sequence of an image sequence includes:

[0035] Performing key point prediction processing on the image sequence using a key point prediction model to obtain a key point prediction sequence of the image sequence;

[0036] After determining the key point prediction performance corresponding to the key point prediction sequence, the method further includes:

[0037] The key point prediction model is updated according to the key point prediction performance corresponding to the key point prediction sequence.

[0038] The present application provides a data processing device, comprising:

[0039] A data acquisition unit, configured to acquire a key point prediction sequence of an image sequence and a key point label sequence corresponding to the image sequence; the key point prediction sequence includes a prediction trajectory of at least one key point; the key point label sequence includes a label trajectory of the at least one key point;

[0040] A first determining unit is used to determine, for any of the key points, trajectory similarity characterization data between the predicted trajectory of the key point and the label trajectory of the key point; the trajectory similarity characterization data is used to indicate the degree of similarity between the predicted trajectory of the key point and the label trajectory of the key point in terms of trajectory length and / or trajectory direction;

[0041] The second determining unit is used to determine the key point prediction performance corresponding to the key point prediction sequence according to the trajectory similarity characterization data.

[0042] The present application provides an electronic device, the device comprising: a processor and a memory;

[0043] The memory is used to store instructions or computer programs;

[0044] The processor is used to execute the instructions or computer programs in the memory so that the electronic device executes the data processing method provided in the present application.

[0045] The present application provides a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes the data processing method provided by the present application.

[0046] The present application provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program codes for executing the data processing method provided by the present application.

[0047] Compared with the prior art, this application has at least the following advantages:

[0048] In the technical solution provided by the present application, for an image sequence, a key point prediction sequence of the image sequence and a key point label sequence corresponding to the image sequence are first obtained, so that the key point prediction sequence includes a predicted trajectory of at least one key point, and the key point label sequence includes a label trajectory of at least one key point; then, for any key point, the trajectory similarity characterization data between the predicted trajectory of the key point and the label trajectory of the key point is determined, so that the trajectory similarity characterization data can represent the degree of similarity between the predicted trajectory of the key point and the label trajectory of the key point in terms of trajectory length and / or trajectory direction, so that the trajectory similarity characterization data can be more better represent the degree of jitter of the predicted trajectory of the key point relative to the label trajectory of the key point; finally, determine the key point prediction performance corresponding to the key point prediction sequence based on the trajectory similarity characterization data corresponding to all key points, so that the key point prediction performance can at least better represent the degree of jitter of the predicted trajectory of each key point relative to its corresponding label trajectory, so that the key point prediction performance can at least better represent the prediction stability presented when the key point prediction sequence is predicted for the image sequence, and further the key point prediction performance can better represent the prediction performance presented when key point prediction processing is performed on an image sequence. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0051] Figure 2 A schematic diagram of a predicted trajectory and a label trajectory with a corresponding relationship provided in an embodiment of the present application;

[0052] Figure 3 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0053] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] Through research, it is found that for a key point, under the condition of jitter (that is, unstable condition), the predicted position of the key point will usually be distributed around the label position of the key point, and if the jitter is weaker (for example, there is almost no jitter), the predicted position of the key point is closer to the label position of the key point, so that the trajectory formed by the predicted position corresponding to the key point in multiple image data and the trajectory formed by the label position corresponding to the key point in multiple image data are more similar (for example, the trajectory lengths are closer, the trajectory directions are more similar, etc.); if the jitter is stronger (that is, more unstable), the predicted position of the key point deviates from the label position of the key point, so that the difference between the trajectory formed by the predicted position corresponding to the key point in multiple image data and the trajectory formed by the label position corresponding to the key point in multiple image data is greater (for example, the trajectory length difference is larger, the trajectory direction is more dissimilar, etc.).

[0055] Based on the above research findings, in order to better evaluate the prediction performance (for example, stability, etc.) presented when performing key point prediction processing on an image sequence, the present application provides a data processing method, which includes: first obtaining a key point prediction sequence of an image sequence and a key point label sequence corresponding to the image sequence, so that the key point prediction sequence includes a prediction trajectory of at least one key point, and the key point label sequence includes a label trajectory of at least one key point; then, for any key point, determining the trajectory similarity characterization data between the prediction trajectory of the key point and the label trajectory of the key point, so that the trajectory similarity characterization data can represent the difference between the prediction trajectory of the key point and the label trajectory of the key point in trajectory length and / or trajectory direction. The degree of similarity presented, so that the trajectory similarity characterization data can better represent the degree of jitter of the predicted trajectory of the key point relative to the label trajectory of the key point; finally, based on the trajectory similarity characterization data corresponding to all key points, the key point prediction performance corresponding to the key point prediction sequence is determined, so that the key point prediction performance can at least better represent the degree of jitter of the predicted trajectory of each key point relative to its corresponding label trajectory, so that the key point prediction performance can at least better represent the prediction stability presented when the key point prediction sequence is predicted for the image sequence, and then the key point prediction performance can better represent the prediction performance presented when key point prediction processing is performed on an image sequence.

[0056] In addition, the present application does not limit the execution subject of the above data processing method. For example, the data processing method provided in the embodiment of the present application can be applied to a device with a data processing function such as a terminal device or a server. For another example, the data processing method provided in the embodiment of the present application can also be implemented by means of a data communication process between different devices (for example, a terminal device and a server, two terminal devices, or two servers). Among them, the terminal device can be a smart phone, a computer, a personal digital assistant (PDA) or a tablet computer. The server can be a stand-alone server, a cluster server or a cloud server.

[0057] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] In order to better understand the technical solution provided by this application, the data processing method provided by this application is described below with reference to some drawings. Figure 1 As shown, the data processing method provided in the embodiment of the present application includes the following S1-S3. Figure 1 A flowchart of a data processing method provided in an embodiment of the present application.

[0059] S1: Obtain a key point prediction sequence of an image sequence and a key point label sequence corresponding to the image sequence; the key point prediction sequence includes a prediction trajectory of at least one key point; the key point label sequence includes a label trajectory of at least one key point.

[0060] The image sequence is used to record a plurality of image data arranged in a certain order (eg, time sequence); and the present application does not limit the implementation method of the image sequence, for example, it can be implemented using video data.

[0061] The key point prediction sequence of an image sequence is used to describe the changes in the predicted positions of the key points in the image sequence; and the present application does not limit the key point prediction sequence. For example, when the image sequence is the sequence of {1st image data, 2nd image data, ... (and so on), Nth image data}, the key point prediction sequence of the image sequence may include the key point prediction results of the 1st image data, the key point prediction results of the 2nd image data, ... (and so on), and the key point prediction results of the Nth image data. Among them, the key point prediction results of the nth image data are used to describe the predicted distribution of the key points in the nth image data; and the present application does not limit the key point prediction results of the nth image data. For example, the key point prediction results of the nth image data may include the predicted positions of multiple key points, and the predicted positions of the key points are used to describe the predicted positions of the key points in the nth image data. n is a positive integer, n≤N, N is a positive integer, and N represents the number of image data in the image sequence.

[0062] In fact, for the key point prediction sequence of the above image sequence, the key point prediction sequence can not only represent the key point distribution according to the image data dimension (for example, the key point prediction sequence includes the key point prediction results of each image data in the image sequence), but also represent the key point distribution according to the key point dimension (for example, the key point prediction sequence includes the predicted position changes of each key point in the image sequence).

[0063] Based on the content of the previous paragraph, it can be known that in a possible implementation, when the above image sequence is used to describe at least one key point, the key point prediction sequence of the image sequence may include the predicted trajectory of the at least one key point, so that the key point prediction sequence can represent the predicted position changes of each key point in the image sequence. Among them, for any key point, the predicted trajectory of the key point is used to describe the predicted position changes of the key point in the image sequence; and the present application does not limit the predicted trajectory of the key point. For example, when the image sequence is the sequence of {1st image data, 2nd image data, ..., Nth image data}, the predicted trajectory of the key point may include the predicted position corresponding to the key point under the 1st image data (for example, Figure 2 x0 shown), the predicted position of the key point under the second image data (for example, Figure 2 x1 shown), the predicted position of the key point under the third image data (for example, Figure 2 x2 as shown), ... (and so on), the predicted position corresponding to the key point in the Nth image data (for example, Figure 2The predicted position corresponding to the key point in the nth image data is used to describe the predicted position of the key point in the nth image data. n is a positive integer, n≤N, N is a positive integer, and N represents the number of image data in the image sequence.

[0064] It should be noted that for Figure 2 For example, x0→x1→x2→x3 represents the predicted trajectory of a key point, x0 represents the predicted position of the key point in the first image data, x1 represents the predicted position of the key point in the second image data, x2 represents the predicted position of the key point in the third image data, and x3 represents the predicted position of the key point in the fourth image data; g0→g1→g2→g3 represents the label trajectory of the key point, g0 represents the label position of the key point in the first image data, g1 represents the label position of the key point in the second image data, g2 represents the label position of the key point in the third image data, and g3 represents the label position of the key point in the fourth image data.

[0065] It should also be noted that the present application does not limit the method of representing the position (for example, the "label position" above or the "predicted position" above). For example, it can be implemented using any existing or future position representation method (for example, coordinates, etc.).

[0066] In addition, the present application does not limit the method for obtaining the above-mentioned key point prediction sequence. For example, it can be implemented by any existing or future method that can perform key point prediction processing on an image sequence (for example, with the help of a pre-built machine learning model with key point prediction function, etc.).

[0067] Based on the content of the above paragraph, it can be known that in one possible implementation, the process of acquiring the above key point prediction sequence can be specifically as follows: after acquiring the image sequence, performing key point prediction processing on the image sequence to obtain the key point prediction sequence of the image sequence. In another possible implementation, the process of acquiring the above key point prediction sequence can be specifically as follows: after acquiring the image sequence, performing key point prediction processing on the image sequence using a key point prediction model to obtain the key point prediction sequence of the image sequence. The key point prediction model is used to perform key point prediction processing on the input data of the key point prediction model; and the present application does not limit the key point prediction model.

[0068] The key point label sequence corresponding to the image sequence is used to describe the actual position changes of the key points in the image sequence; and the present application does not limit the key point label sequence. For example, when the image sequence is the sequence of {1st image data, 2nd image data, ..., Nth image data}, the key point label sequence corresponding to the image sequence may include the key point label result of the 1st image data, the key point label result of the 2nd image data, ... (and so on), and the key point label result of the Nth image data. Among them, the key point label result of the nth image data is used to describe the actual distribution of the key points in the nth image data; and the present application does not limit the key point label result of the nth image data. For example, the key point label result of the nth image data may include the label positions of multiple key points, and the label position of the key point is used to describe the actual position of the key point in the nth image data. n is a positive integer, n≤N, N is a positive integer, and N represents the number of image data in the image sequence.

[0069] For another example, in a possible implementation, when the above image sequence is used to describe at least one key point, the key point label sequence corresponding to the image sequence may include at least one key point label track. Wherein, for any key point, the label track of the key point is used to describe the actual position change of the key point in the above image sequence; and the present application does not limit the label track of the key point. For example, when the image sequence is the sequence of {1st image data, 2nd image data, ..., Nth image data}, the label track of the key point may include the label position corresponding to the key point under the 1st image data (for example, Figure 2 g0 shown), the label position corresponding to the key point in the second image data (for example, Figure 2 g1 shown), the label position corresponding to the key point in the third image data (for example, Figure 2 g2 shown), ... (and so on), the label position corresponding to the key point in the Nth image data (for example, Figure 2 g3 shown). The label position corresponding to the key point in the nth image data is used to describe the actual position of the key point in the nth image data. n is a positive integer, n≤N, N is a positive integer, and N represents the number of image data in the image sequence.

[0070] In addition, the present application does not limit the method for determining the key point label sequence corresponding to the above image sequence. For example, it can be implemented by manual annotation.

[0071] Based on the relevant content of S1 above, it can be known that for an image sequence, the key point prediction sequence of the image sequence and the key point label sequence corresponding to the image sequence are obtained, so that the key point prediction sequence can describe the predicted position changes of the key points in the image sequence (for example, the predicted trajectory of some key points, etc.), and the key point label sequence is used to describe the actual position changes of the key points in the above image sequence (for example, the label trajectory of some key points, etc.), so that the performance (for example, stability, etc.) presented when performing key point prediction processing on the image sequence can be determined based on the differences between the key point prediction sequence and the key point label sequence in certain aspects (for example, trajectory length, etc.).

[0072] S2: For any key point, determine the trajectory similarity characterization data between the predicted trajectory of the key point and the label trajectory of the key point, where the trajectory similarity characterization data is used to represent the degree of similarity between the predicted trajectory of the key point and the label trajectory of the key point in terms of trajectory length and / or trajectory direction.

[0073] The trajectory similarity characterization data between the predicted trajectory of the mth key point and the label trajectory of the mth key point is used to indicate the degree of similarity between the predicted trajectory of the mth key point and the label trajectory of the mth key point, and in particular, can indicate the degree of similarity between the predicted trajectory of the mth key point and the label trajectory of the mth key point in terms of trajectory length and / or trajectory direction. m is a positive integer, m≤the number of key points in the above “at least one key point” (e.g., the number of all key points in the above image sequence, etc.).

[0074] In addition, the present application does not limit the determination process of the above trajectory similarity characterization data. For ease of understanding, some situations are described below.

[0075] Case 1: After research, it is found that for two trajectories, the more similar the two trajectories are, the closer their trajectory lengths should be. Therefore, in some application scenarios, the similarity between the two trajectories can be determined by comparing their trajectory lengths.

[0076] Based on the above situation 1, the present application provides a possible implementation method of the process of determining the above trajectory similarity characterization data. For ease of understanding, the following is an explanation taking the process of determining the above "trajectory similarity characterization data between the predicted trajectory of the mth key point and the label trajectory of the mth key point" as an example.

[0077] As an example, when the above image sequence includes a first image (for example, the nth image data) and a second image (for example, the n+1th image data), the timing corresponding to the first image in the image sequence is earlier than the timing corresponding to the second image in the image sequence, the timing corresponding to the first image in the image sequence is adjacent to the timing corresponding to the second image in the image sequence, the predicted trajectory of the mth key point above includes the predicted position corresponding to the mth key point in the first image and the predicted position corresponding to the mth key point in the second image, and the label trajectory of the mth key point includes the label position corresponding to the mth key point in the first image and the label position corresponding to the mth key point in the second image, the above determination process of "the trajectory similarity characterization data between the predicted trajectory of the mth key point and the label trajectory of the mth key point" may include the following steps 11-14.

[0078] Step 11: Calculate the distance between the predicted position corresponding to the mth key point in the second image and the predicted position corresponding to the mth key point in the first image, and obtain the predicted length of the trajectory segment of the mth key point between the second image and the first image.

[0079] The first image is used to represent the image data at the front position in any pair of adjacent image data in the above image sequence; the second image is used to represent the image data at the back position in the pair of adjacent image data.

[0080] Based on the above content, it can be known that for the above image sequence, when the image sequence is the sequence of {1st image data, 2nd image data, ..., Nth image data}, if the above first image is the 1st image data, then the above second image is the 2nd image data; if the first image is the 2nd image data, then the second image is the 3rd image data; ... (and so on); if the first image is the N-1th image data, then the second image is the Nth image data. It can be seen that in a possible implementation, if the first image is the nth image data in the image sequence, then the second image is the n+1th image data in the image sequence. Wherein, n is a positive integer, n+1≤N.

[0081] The above “predicted position corresponding to the mth key point in the first image” refers to the position information predicted for the mth key point using the first image, so that the “predicted position corresponding to the mth key point in the first image” can represent the predicted position of the mth key point in the first image.

[0082] The above “predicted position corresponding to the mth key point in the second image” refers to the position information predicted for the mth key point using the second image, so that the “predicted position corresponding to the mth key point in the second image” can represent the predicted position of the mth key point in the second image.

[0083] The above “the distance between the predicted position corresponding to the mth key point in the second image and the predicted position corresponding to the mth key point in the first image” is used to describe the trajectory length between the predicted position corresponding to the mth key point in the second image and the predicted position corresponding to the mth key point in the first image; and the embodiment of the present application does not limit the calculation process of the “distance between the predicted position corresponding to the mth key point in the second image and the predicted position corresponding to the mth key point in the first image”, for example, it can be implemented using any existing or future method that can calculate the distance between two points.

[0084] The above “predicted length of the trajectory segment of the m-th key point between the second image and the first image” is used to describe the predicted trajectory length of the m-th key point between the second image and the first image.

[0085] Based on the relevant content of step 11 above, it can be known that in some application scenarios, if the above image sequence is the sequence of {1st image data, 2nd image data, ..., Nth image data}, then the predicted trajectory of the mth key point above may include the predicted position corresponding to the mth key point under the 1st image data, the predicted position corresponding to the mth key point under the 2nd image data, ... (and so on), the predicted position corresponding to the mth key point under the Nth image data, and the distance between the predicted position corresponding to the mth key point under the 2nd image data and the predicted position corresponding to the mth key point under the 1st image data can be determined as the distance between the mth key point in the 2nd image data and the 1st image data. ; the distance between the predicted position corresponding to the m-th key point under the third image data and the predicted position corresponding to the m-th key point under the second image data is determined as the predicted length of the trajectory segment of the m-th key point between the third image data and the second image data; ... (and so on); the distance between the predicted position corresponding to the m-th key point under the N-th image data and the predicted position corresponding to the m-th key point under the N-1-th image data is determined as the predicted length of the trajectory segment of the m-th key point between the N-th image data and the N-1-th image data, so that the predicted length of the trajectory segment presented by the m-th key point in any pair of adjacent image data in the image sequence can be obtained.

[0086] Step 12: Calculate the distance between the label position corresponding to the mth key point in the second image and the label position corresponding to the mth key point in the first image, and obtain the label length of the trajectory segment of the mth key point between the second image and the first image.

[0087] The above “label position corresponding to the mth key point in the first image” refers to the position information pre-marked for the mth key point in the first image, so that the “label position corresponding to the mth key point in the first image” can indicate the actual position of the mth key point in the first image.

[0088] The above “the label position corresponding to the mth key point in the second image” refers to the position information pre-marked for the mth key point in the second image, so that the “label position corresponding to the mth key point in the second image” can indicate the actual position of the mth key point in the second image.

[0089] The above “the distance between the label position corresponding to the mth key point in the second image and the label position corresponding to the mth key point in the first image” is used to describe the trajectory length between the label position corresponding to the mth key point in the second image and the label position corresponding to the mth key point in the first image; and the embodiment of the present application does not limit the calculation process of the “distance between the label position corresponding to the mth key point in the second image and the label position corresponding to the mth key point in the first image”. For example, the implementation method of the calculation process is similar to the implementation method of the above “the distance between the predicted position corresponding to the mth key point in the second image and the predicted position corresponding to the mth key point in the first image”. For the sake of brevity, it will not be repeated here.

[0090] The above “the length of the trajectory segment label of the m-th key point between the second image and the first image” is used to describe the actual trajectory length of the m-th key point between the second image and the first image.

[0091] Based on the relevant content of step 12 above, it can be known that in some application scenarios, if the above image sequence is the sequence of {1st image data, 2nd image data, ..., Nth image data}, then the label trajectory of the mth key point above may include the label position corresponding to the mth key point under the 1st image data, the label position corresponding to the mth key point under the 2nd image data, ... (and so on), the label position corresponding to the mth key point under the Nth image data, and the distance between the label position corresponding to the mth key point under the 2nd image data and the label position corresponding to the mth key point under the 1st image data can be determined as the trajectory of the mth key point between the 2nd image data and the 1st image data. Segment label length; the distance between the label position corresponding to the mth key point in the third image data and the label position corresponding to the mth key point in the second image data is determined as the trajectory segment label length of the mth key point between the third image data and the second image data; ... (and so on); the distance between the label position corresponding to the mth key point in the Nth image data and the label position corresponding to the mth key point in the N-1th image data is determined as the trajectory segment label length of the mth key point between the Nth image data and the N-1th image data, so that the label length (that is, the actual length) of the trajectory segment presented by the mth key point in any pair of adjacent image data in the image sequence can be obtained.

[0092] Step 13: Determine the length similarity representation data between the predicted trajectory of the m-th key point and the labeled trajectory of the m-th key point based on the difference between the predicted length of the corresponding trajectory segment and the label length of the trajectory segment, so that the length similarity representation data can represent the degree of similarity between the predicted trajectory of the m-th key point and the labeled trajectory of the m-th key point in terms of trajectory length.

[0093] The above “corresponding predicted length of trajectory segment and trajectory segment label length” are used to describe the predicted length and actual length of the trajectory segment presented by the same key point in the same pair of adjacent image data. For example, the “corresponding predicted length of trajectory segment and trajectory segment label length” may include: the above “the predicted length of trajectory segment of the mth key point between the n+1th image data and the nth image data” and the above “the trajectory segment label length of the mth key point between the n+1th image data and the nth image data”; wherein n is a positive integer, n+1≤N.

[0094] In addition, for the above “trajectory segment prediction length and trajectory segment label length having a corresponding relationship”, the corresponding relationship is predetermined, for example, the corresponding relationship may specifically include: the corresponding relationship between the above “the trajectory segment prediction length of the mth key point between the n+1th image data and the nth image data” and the above “the trajectory segment label length of the mth key point between the n+1th image data and the nth image data”. Where n is a positive integer, n+1≤N.

[0095] In addition, the present application does not limit the implementation method of the above step 13. For example, it can be implemented using the following formula (1).

[0096]

[0097] Where E represents the length similarity between the predicted trajectory of a key point and the label trajectory of the key point; x i Indicates the predicted position of the key point under the i+1th image data; x i-1 Indicates the predicted position of the key point under the i-th image data; ∥x i -x i-1 ||2 represents the predicted length of the trajectory segment of the key point between the i+1th image data and the ith image data; g i Indicates the label position corresponding to the key point in the i+1th image data; g i-1 Indicates the label position corresponding to the key point in the i-th image data; ∥g i -g i-1 ||2 represents the predicted length of the trajectory segment of the key point between the i+1th image data and the ith image data; N represents the number of image data in the above image sequence.

[0098] Step 14: Determine the trajectory similarity representation data between the predicted trajectory of the mth key point and the label trajectory of the mth key point based on the length similarity representation data between the predicted trajectory of the mth key point and the label trajectory of the mth key point.

[0099] It should be noted that the present application does not limit the implementation method of the above step 14. For example, it can specifically be: directly determine the length similarity characterization data between the predicted trajectory of the mth key point and the label trajectory of the mth key point as the trajectory similarity characterization data between the predicted trajectory of the mth key point and the label trajectory of the mth key point, so that the trajectory similarity characterization data can at least represent the degree of similarity between the predicted trajectory of the mth key point and the label trajectory of the mth key point in terms of trajectory length.

[0100] Based on the relevant contents of steps 11 to 14 above, it can be known that for any key point described in the above image sequence (for example, the mth key point above), if the image sequence includes a first image and a second image, the predicted trajectory of the key point includes the predicted position corresponding to the key point in the first image and the predicted position corresponding to the key point in the second image, and the label trajectory of the key point includes the label position corresponding to the key point in the first image and the label position corresponding to the key point in the second image, then the process of determining the trajectory similarity representation data between the predicted trajectory of the key point and the label trajectory of the key point can be specifically as follows: first, calculate the corresponding position of the key point in the second image; The predicted length of the trajectory segment is obtained by calculating the distance between the predicted position and the predicted position corresponding to the key point in the first image, and the distance between the label position corresponding to the key point in the second image and the label position corresponding to the key point in the first image is calculated to obtain the label length of the trajectory segment; then, according to the difference between the predicted length of the trajectory segment and the label length of the trajectory segment, the length similarity representation data is determined; finally, according to the length similarity representation data, the trajectory similarity representation data between the predicted trajectory of the key point and the label trajectory of the key point is determined, so that the trajectory similarity representation data can at least represent the similarity between the predicted trajectory of the key point and the label trajectory of the key point in terms of trajectory length.

[0101] Case 2: After research, it is found that for two trajectories, the more similar the two trajectories are, the more similar the changing trends of the trajectory directions of the two trajectories should be. Therefore, in some application scenarios, the similarity between the two trajectories can be determined by comparing the trajectory directions of the two trajectories.

[0102] Based on the above situation 2, the present application provides a possible implementation method of the process of determining the above trajectory similarity characterization data. For ease of understanding, the following is an explanation taking the process of determining the above "trajectory similarity characterization data between the predicted trajectory of the mth key point and the label trajectory of the mth key point" as an example.

[0103] As an example, when the above image sequence includes a first image (for example, the nth image data) and a second image (for example, the n+1th image data), the timing corresponding to the first image in the image sequence is earlier than the timing corresponding to the second image in the image sequence, the timing corresponding to the first image in the image sequence is adjacent to the timing corresponding to the second image in the image sequence, the predicted trajectory of the mth key point includes the predicted position corresponding to the mth key point in the first image and the predicted position corresponding to the mth key point in the second image, and the label trajectory of the mth key point includes the label position corresponding to the mth key point in the first image and the label position corresponding to the mth key point in the second image, the above determination process of "the trajectory similarity characterization data between the predicted trajectory of the mth key point and the label trajectory of the mth key point" may include the following steps 21-24.

[0104] Step 21: Determine the predicted direction representation data of the trajectory segment of the mth key point between the second image and the first image based on the direction representation data between the predicted position corresponding to the mth key point in the second image and the predicted position corresponding to the mth key point in the first image.

[0105] The above “directional characterization data between the predicted position corresponding to the mth key point in the second image and the predicted position corresponding to the mth key point in the first image” is used to describe the trajectory direction between the predicted position corresponding to the mth key point in the second image and the predicted position corresponding to the mth key point in the first image.

[0106] In addition, the present application does not limit the implementation method of the above "the direction representation data between the predicted position corresponding to the mth key point in the second image and the predicted position corresponding to the mth key point in the first image", for example, it can be implemented by any existing or future method that can represent the direction between two positions (for example, position vector, unit vector, straight line slope, or straight line inclination angle, etc.). For ease of understanding, the following five examples are used for explanation.

[0107] Example 1, the above step 21 can be specifically as follows: subtract the predicted position corresponding to the mth key point in the first image from the predicted position corresponding to the mth key point in the second image to obtain a position vector, and regard the position vector as the above "the directional representation data between the predicted position corresponding to the mth key point in the second image and the predicted position corresponding to the mth key point in the first image".

[0108] Example 2, the above step 21 can be specifically as follows: first, subtract the predicted position corresponding to the mth key point in the first image from the predicted position corresponding to the mth key point in the second image to obtain a position vector; then, using the position vector, determine a unit vector so that the unit vector can describe the direction described by the position vector; and then regard the unit vector as the above "the direction representation data between the predicted position corresponding to the mth key point in the second image and the predicted position corresponding to the mth key point in the first image".

[0109] Example 3, the above step 21 can be specifically as follows: perform a straight line fitting on the predicted position corresponding to the mth key point in the second image and the predicted position corresponding to the mth key point in the first image to obtain the slope of the straight line, and regard the slope of the straight line as the above "the directional characterization data between the predicted position corresponding to the mth key point in the second image and the predicted position corresponding to the mth key point in the first image".

[0110] Example 4, the above step 21 can be specifically as follows: first, perform a straight line fitting on the predicted position corresponding to the mth key point in the second image and the predicted position corresponding to the mth key point in the first image to obtain the slope of the straight line; then convert the slope of the straight line into an inclination angle of the straight line, and regard the inclination angle of the straight line as the above "the direction characterization data between the predicted position corresponding to the mth key point in the second image and the predicted position corresponding to the mth key point in the first image".

[0111] Example 5, the above step 21 can be specifically as follows: based on the predicted position corresponding to the mth key point in the second image and the predicted position corresponding to the mth key point in the first image, calculate the straight line inclination angle to obtain the straight line inclination angle, and regard the straight line inclination angle as the above "the direction characterization data between the predicted position corresponding to the mth key point in the second image and the predicted position corresponding to the mth key point in the first image".

[0112] The above “the predicted direction representation data of the trajectory segment of the m-th key point between the second image and the first image” is used to describe the trajectory direction predicted by the m-th key point between the second image and the first image.

[0113] Based on the relevant content of step 21 above, it can be known that in some application scenarios, if the above image sequence is the sequence of {1st image data, 2nd image data, ..., Nth image data}, then the predicted trajectory of the mth key point above may include the predicted position corresponding to the mth key point under the 1st image data, the predicted position corresponding to the mth key point under the 2nd image data, ... (and so on), the predicted position corresponding to the mth key point under the Nth image data, and the direction representation data between the predicted position corresponding to the mth key point under the 2nd image data and the predicted position corresponding to the mth key point under the 1st image data can be determined as the position of the mth key point between the 2nd image data and the 1st image data. The predicted direction of the trajectory segment of the m-th key point is determined by the direction representation data between the predicted position corresponding to the m-th key point under the third image data and the predicted position corresponding to the m-th key point under the second image data as the predicted direction of the trajectory segment of the m-th key point between the third image data and the second image data; ... (and so on); the predicted direction representation data between the predicted position corresponding to the m-th key point under the N-th image data and the predicted position corresponding to the m-th key point under the N-1-th image data are determined as the predicted direction of the trajectory segment of the m-th key point between the N-th image data and the N-1-th image data, so that the predicted direction of the trajectory segment presented by the m-th key point in any pair of adjacent image data in the image sequence can be obtained.

[0114] Step 22: Determine the label direction representation data of the trajectory segment of the mth key point between the second image and the first image based on the direction representation data between the label position corresponding to the mth key point in the second image and the label position corresponding to the mth key point in the first image.

[0115] The above “directional representation data between the label position corresponding to the mth key point in the second image and the label position corresponding to the mth key point in the first image” is used to describe the trajectory direction between the label position corresponding to the mth key point in the second image and the label position corresponding to the mth key point in the first image.

[0116] In addition, the implementation method of the above "the directional representation data between the label position corresponding to the m-th key point in the second image and the label position corresponding to the m-th key point in the first image" is similar to the implementation method of the above "the directional representation data between the predicted position corresponding to the m-th key point in the second image and the predicted position corresponding to the m-th key point in the first image", and for the sake of brevity, it will not be repeated here.

[0117] The above “the trajectory segment label direction characterization data of the m-th key point between the second image and the first image” is used to describe the actual trajectory direction of the m-th key point between the second image and the first image.

[0118] Based on the relevant content of step 22 above, it can be known that in some application scenarios, if the above image sequence is the sequence of {1st image data, 2nd image data, ..., Nth image data}, then the label trajectory of the mth key point above may include the label position corresponding to the mth key point under the 1st image data, the label position corresponding to the mth key point under the 2nd image data, ... (and so on), the label position corresponding to the mth key point under the Nth image data, and the direction representation data between the label position corresponding to the mth key point under the 2nd image data and the label position corresponding to the mth key point under the 1st image data can be determined as the trajectory segment label of the mth key point between the 2nd image data and the 1st image data. label direction; determine the direction representation data between the label position corresponding to the m-th key point under the third image data and the label position corresponding to the m-th key point under the second image data as the label direction of the trajectory segment of the m-th key point between the third image data and the second image data; ... (and so on); determine the direction representation data between the label position corresponding to the m-th key point under the N-th image data and the label position corresponding to the m-th key point under the N-1-th image data as the label direction of the trajectory segment of the m-th key point between the N-th image data and the N-1-th image data, so that the label direction (that is, the actual direction) of the trajectory segment presented by the m-th key point in any pair of adjacent image data in the image sequence can be obtained.

[0119] Step 23: Determine the directional similarity characterization data between the predicted trajectory of the m-th key point and the labeled trajectory of the m-th key point based on the predicted directional characterization data of the above trajectory segment and the labeled directional characterization data of the above trajectory segment, so that the directional similarity characterization data can represent the degree of similarity between the predicted trajectory of the m-th key point and the labeled trajectory of the m-th key point in the trajectory direction.

[0120] It should be noted that the present application does not limit the implementation method of step 23. For example, it can be specifically as follows: first, based on the predicted direction of the trajectory segment of the mth key point between the nth image data and the n+1th image data, and the label direction of the trajectory segment of the mth key point between the nth image data and the n+1th image data, determine the direction difference representation data of the mth key point between the nth image data and the n+1th image data, so that the direction difference representation data can represent the direction deviation of the predicted direction of the trajectory segment relative to the label direction of the trajectory segment, n is a positive integer, n+1≤N; then calculate the average value of all the direction difference representation data corresponding to the mth key point as the direction similarity representation data between the predicted trajectory of the mth key point and the label trajectory of the mth key point.

[0121] It should also be noted that the present application does not limit the calculation method of the "direction difference characterization data" in the above paragraph. For example, if the predicted direction of the above trajectory segment and the label direction of the above trajectory segment are both represented by vectors (for example, position vectors or unit vectors), then the "direction difference characterization data" can be determined with the help of the inner product calculation formula. For another example, if the predicted direction of the above trajectory segment and the label direction of the above trajectory segment are both represented by the slope of a straight line, then the "direction difference characterization data" can be directly implemented with the help of a difference calculation formula or a calculation formula for the angle between the slopes of two straight lines. For another example, if the predicted direction of the above trajectory segment and the label direction of the above trajectory segment are both represented by the inclination angle of a straight line, then the "direction difference characterization data" can be directly implemented with the help of a difference calculation formula.

[0122] Step 24: Determine trajectory similarity characterization data between the predicted trajectory of the mth key point and the label trajectory of the mth key point based on the directional similarity characterization data between the predicted trajectory of the mth key point and the label trajectory of the mth key point.

[0123] It should be noted that the present application does not limit the implementation method of the above step 24. For example, it can specifically be: directly determine the directional similarity characterization data between the predicted trajectory of the mth key point and the label trajectory of the mth key point as the trajectory similarity characterization data between the predicted trajectory of the mth key point and the label trajectory of the mth key point, so that the trajectory similarity characterization data can at least represent the degree of similarity between the predicted trajectory of the mth key point and the label trajectory of the mth key point in the trajectory direction.

[0124] Based on the relevant contents of steps 21 to 24 above, it can be known that for any key point described in the above image sequence (for example, the mth key point above), if the image sequence includes a first image and a second image, the predicted trajectory of the key point includes the predicted position corresponding to the key point in the first image and the predicted position corresponding to the key point in the second image, and the label trajectory of the key point includes the label position corresponding to the key point in the first image and the label position corresponding to the key point in the second image, then the process of determining the trajectory similarity representation data between the predicted trajectory of the key point and the label trajectory of the key point can be specifically as follows: first, based on the predicted position corresponding to the key point in the second image and the predicted position of the key point in the second image, the predicted trajectory of the key point is determined. The predicted direction characterization data of the trajectory segment is determined based on the direction characterization data between the corresponding predicted positions under the first image, and the trajectory segment label direction characterization data is determined based on the direction characterization data between the label position corresponding to the key point under the second image and the label position corresponding to the key point under the first image; the direction similarity characterization data is then determined based on the predicted direction characterization data of the trajectory segment and the direction characterization data of the trajectory segment label; finally, the trajectory similarity characterization data between the predicted trajectory of the key point and the label trajectory of the key point is determined based on the direction similarity characterization data, so that the trajectory similarity characterization data can at least represent the degree of similarity between the predicted trajectory of the key point and the label trajectory of the key point in the trajectory direction.

[0125] Case 3: In some application scenarios, in order to better improve the effect of determining the trajectory similarity, the similarity between the two trajectories can be determined by comparing the trajectory directions and trajectory lengths of the two trajectories.

[0126] Based on the above situation 3, the present application provides a possible implementation of the process of determining the above trajectory similarity characterization data, which may specifically include the following steps 31 and 32.

[0127] Step 31: Determine, based on the predicted trajectory of the mth key point and the label trajectory of the mth key point, length similarity characterization data between the predicted trajectory of the mth key point and the label trajectory of the mth key point, and direction similarity characterization data between the predicted trajectory of the mth key point and the label trajectory of the mth key point. m is a positive integer, m≤the number of key points in the above “at least one key point”.

[0128] Step 32: Determine the trajectory similarity characterization data between the predicted trajectory of the mth key point and the label trajectory of the mth key point based on the length similarity characterization data between the predicted trajectory of the mth key point and the label trajectory of the mth key point, and the direction similarity characterization data between the predicted trajectory of the mth key point and the label trajectory of the mth key point, so that the trajectory similarity characterization data can represent the degree of similarity between the predicted trajectory of the mth key point and the label trajectory of the mth key point in terms of trajectory length and trajectory direction.

[0129] It should be noted that the present application does not limit the implementation method of step 32. For example, step 32 may specifically be: combining the length similarity characterization data between the predicted trajectory of the mth key point and the label trajectory of the mth key point, and the direction similarity characterization data between the predicted trajectory of the mth key point and the label trajectory of the mth key point, to obtain the trajectory similarity characterization data between the predicted trajectory of the mth key point and the label trajectory of the mth key point, so that the trajectory similarity characterization data includes the length similarity characterization data and the direction similarity characterization data.

[0130] For example, the above step 32 can be specifically as follows: weighted summation of the length similarity representation data between the predicted trajectory of the mth key point and the label trajectory of the mth key point, and the direction similarity representation data between the predicted trajectory of the mth key point and the label trajectory of the mth key point, to obtain the trajectory similarity representation data between the predicted trajectory of the mth key point and the label trajectory of the mth key point.

[0131] Based on the relevant contents of steps 31 to 32 above, it can be known that for any key point described in the above image sequence (for example, the mth key point above), the length similarity characterization data and the direction similarity characterization data can be first determined based on the predicted trajectory of the key point and the label trajectory of the key point; and then, based on the length similarity characterization data and the direction similarity characterization data, the trajectory similarity characterization data between the predicted trajectory of the key point and the label trajectory of the key point can be determined, so that the trajectory similarity characterization data can at least represent the degree of similarity between the predicted trajectory of the key point and the label trajectory of the key point in terms of trajectory length and trajectory direction.

[0132] Based on the relevant content of "trajectory similarity representation data" above, it can be known that in a possible implementation manner, for any key point described in the above image sequence (for example, the mth key point above), the trajectory similarity representation data between the predicted trajectory of the key point and the label trajectory of the key point includes length similarity representation data and / or direction similarity representation data; the length similarity representation data is used to indicate the degree of similarity between the predicted trajectory of the key point and the label trajectory of the key point in trajectory length; the direction similarity representation data is used to indicate the degree of similarity between the predicted trajectory of the key point and the label trajectory of the key point in trajectory direction.

[0133] Based on the relevant content of S2 above, it can be known that for any key point described in the above image sequence, after obtaining the predicted trajectory of the key point and the label trajectory of the key point, the trajectory similarity characterization data between the predicted trajectory of the key point and the label trajectory of the key point can be calculated, so that the trajectory similarity characterization data can represent the degree of similarity between the predicted trajectory of the key point and the label trajectory of the key point in terms of trajectory length and / or trajectory direction, so that the performance (for example, stability, etc.) presented when performing key point prediction processing on the image sequence can be determined based on the trajectory similarity characterization data.

[0134] S3: Determine the key point prediction performance corresponding to the key point prediction sequence based on the trajectory similarity representation data.

[0135] The key point prediction performance corresponding to the key point prediction sequence is used to describe the performance (eg, stability, etc.) presented when the key point prediction sequence is obtained by predicting the above image sequence.

[0136] In addition, the present application does not limit the implementation method of S3 above. For example, it can specifically be: directly performing some statistical analysis (for example, average value, maximum value, median value, etc.) on the trajectory similarity representation data corresponding to all key points to obtain the key point prediction performance corresponding to the key point prediction sequence above.

[0137] It is found through research that the similarity between the predicted trajectory and the label trajectory can be used to measure the stability of the key point prediction result. Based on this, the present application also provides a possible implementation of S3 above, which may specifically include the following steps 41-42.

[0138] Step 41: Determine the prediction stability characterization data corresponding to the above key point prediction sequence based on the above trajectory similarity characterization data, so that the prediction stability characterization data is used to represent the prediction stability presented when the key point prediction sequence is predicted for the above image sequence.

[0139] It should be noted that the present application does not limit the implementation method of the above step 41. For example, the step 41 can specifically be: collecting the trajectory similarity characterization data corresponding to all key points to obtain the predicted stability characterization data corresponding to the above key point prediction sequence, so that the predicted stability characterization data includes the trajectory similarity characterization data corresponding to all key points.

[0140] For example, the above step 41 can also be: directly determine the average value of the trajectory similarity characterization data corresponding to all key points as the prediction stability characterization data corresponding to the above key point prediction sequence, so that the prediction stability characterization data is used to represent the stability presented when the key point prediction sequence is predicted for the above image sequence.

[0141] Step 42: Determine the key point prediction performance corresponding to the key point prediction sequence according to the prediction stability characterization data corresponding to the key point prediction sequence.

[0142] It should be noted that the present application does not limit the implementation method of the above step 42. For example, when the predicted stability characterization data corresponding to the above key point prediction sequence is a data value (for example, the above "average value between the trajectory similarity characterization data corresponding to all key points", etc.), the step 42 can specifically be: directly determining the predicted stability characterization data as the key point prediction performance corresponding to the key point prediction sequence.

[0143] For example, when the predicted stability characterization data corresponding to the above key point prediction sequence is a data set (for example, the predicted stability characterization data includes trajectory similarity characterization data corresponding to all key points), the above step 42 can specifically be: determining the average value between all elements in the data set as the predicted stability characterization data corresponding to the key point prediction sequence.

[0144] Through research, it is found that in some application scenarios, for an image sequence, when evaluating the key point prediction effect for the image sequence, not only stability but also accuracy can be considered. Based on this, the present application also provides a possible implementation of the key point prediction performance determination process corresponding to the key point prediction sequence above (that is, step 42 above), which can specifically include the following steps 421-422.

[0145] Step 421: Determine the prediction accuracy characterization data corresponding to the key point prediction sequence according to the above image sequence and the key point label sequence corresponding to the image sequence, so that the prediction accuracy characterization data is used to represent the prediction accuracy presented when the key point prediction sequence is predicted for the image sequence.

[0146] It should be noted that the present application does not limit the implementation of the above step 421. For example, it can be implemented by any existing or future method that can calculate the prediction accuracy with the help of prediction information and standard information (for example, a method of calculating the similarity between the prediction information and the standard information). For ease of understanding, two examples are provided below for explanation.

[0147] Assume that the above image sequence is the sequence of {1st image data, 2nd image data, ..., Nth image data}; the key point prediction sequence of the image sequence includes the prediction position corresponding to the mth key point under the 1st image data, the prediction position corresponding to the mth key point under the 2nd image data, ... (and so on), and the prediction position corresponding to the mth key point under the Nth image data, m is a positive integer, m≤the number of key points in the above "at least one key point"; the key point label sequence corresponding to the image sequence includes the label position corresponding to the mth key point under the 1st image data, the label position corresponding to the mth key point under the 2nd image data, ... (and so on), and the label position corresponding to the mth key point under the Nth image data, m is a positive integer, m≤the number of key points in the above "at least one key point".

[0148] Example 1. Based on the above assumptions, it can be known that the above step 421 can be specifically as follows: first, calculate the distance between the predicted position corresponding to the mth key point in the nth image data and the label position corresponding to the mth key point in the nth image data, and obtain the position prediction loss corresponding to the mth key point in the nth image data, where n is a positive integer, n≤N, and m is a positive integer, m≤the number of key points in the above "at least one key point"; then, the position prediction losses corresponding to all key points in all image data are averaged (or summed) to obtain the prediction accuracy characterization data corresponding to the above key point prediction sequence.

[0149] Example 2. Based on the above assumptions, it can be known that the above step 421 can be specifically as follows: first, calculate the distance between the predicted position corresponding to the mth key point in the nth image data and the label position corresponding to the mth key point in the nth image data, and obtain the position prediction loss corresponding to the mth key point in the nth image data, where n is a positive integer, n≤N, and m is a positive integer, m≤the number of key points in the above "at least one key point"; then, collect the position prediction losses corresponding to all key points in all image data to obtain the prediction accuracy characterization data corresponding to the above key point prediction sequence.

[0150] It should be noted that the present application does not limit the execution time of the above step 421. It only needs to ensure that the execution time of the step 421 is later than the execution time of the above S1, and ensure that the execution time of the step 421 is earlier than the execution time of the following step 422.

[0151] Step 422: Determine the key point prediction performance corresponding to the key point prediction sequence according to the prediction stability characterization data corresponding to the above key point prediction sequence and the prediction accuracy characterization data corresponding to the key point prediction sequence.

[0152] It should be noted that the present application does not limit the implementation method of the above step 422. For example, when the prediction stability characterization data corresponding to the above key point prediction sequence and the prediction accuracy characterization data corresponding to the key point prediction sequence are both the same data value, the step 422 can specifically be: directly performing weighted sum processing on the prediction stability characterization data corresponding to the key point prediction sequence and the prediction accuracy characterization data corresponding to the key point prediction sequence to obtain the key point prediction performance corresponding to the key point prediction sequence.

[0153] For example, when the prediction stability characterization data corresponding to the above key point prediction sequence and the prediction accuracy characterization data corresponding to the key point prediction sequence are both a data set, the above step 422 can be specifically: first calculate the average values ​​between all elements in the two data sets respectively; then perform weighted sum processing on the two average values ​​to obtain the key point prediction performance corresponding to the key point prediction sequence.

[0154] For example, when the prediction stability characterization data corresponding to the key point prediction sequence above and the prediction accuracy characterization data corresponding to the key point prediction sequence are both a data set, the above step 422 can be specifically as follows: first, the average value between the position prediction loss corresponding to the mth key point under the n+1th image data and the position prediction loss corresponding to the mth key point under the nth image data is determined as the weighted weight corresponding to the trajectory segment prediction information of the mth key point between the nth image data and the n+1th image data (for example, the trajectory segment prediction length and / or the trajectory segment prediction direction), n is a positive integer, n≤N, m is a positive integer, m≤the number of key points in the above "at least one key point"; then, according to the weighted weight, these trajectory segment prediction information are weighted and summed to obtain the key point prediction performance corresponding to the key point prediction sequence.

[0155] Based on the relevant contents of steps 421 to 422 above, it can be known that in some application scenarios, for an image sequence, after obtaining the key point prediction sequence of the image sequence and the key point label sequence corresponding to the image sequence, it is necessary not only to analyze the prediction stability presented when the key point prediction sequence is predicted for the image sequence based on the key point prediction sequence and the key point label sequence, but also to analyze the prediction accuracy presented when the key point prediction sequence is predicted for the image sequence based on the key point prediction sequence and the key point label sequence, so that the key point prediction performance corresponding to the key point prediction sequence can be determined with the help of these two analysis results, so that the key point prediction performance can better represent the prediction effect presented when the key point prediction sequence is predicted for the image sequence.

[0156] Based on the relevant contents of steps 41 to 42 above, it can be known that in some application scenarios, for an image sequence, after obtaining the key point prediction sequence of the image sequence and the key point label sequence corresponding to the image sequence, the trajectory similarity characterization data between the predicted trajectory of each key point described by the image sequence and the label trajectory of the key point can be determined based on the key point prediction sequence and the key point label sequence; then, based on these trajectory similarity characterization data, the prediction stability characterization data corresponding to the key point prediction sequence is calculated, so that the prediction stability characterization data is used to represent the prediction stability presented when the key point prediction sequence is predicted for the image sequence; finally, based on the prediction stability characterization data (and the prediction accuracy characterization data corresponding to the key point prediction sequence), the key point prediction performance corresponding to the key point prediction sequence is determined, so that the key point prediction performance can represent the state achieved under at least one evaluation indicator (for example, stability + accuracy) when the key point prediction sequence is predicted for the image sequence, so that the key point prediction performance can better represent the prediction effect presented when the key point prediction sequence is predicted for the image sequence.

[0157] Based on the relevant contents of S1 to S3 above, it can be known that for the data processing method provided in the embodiment of the present application, the key point prediction sequence of the image sequence and the key point label sequence corresponding to the image sequence are first obtained, so that the key point prediction sequence includes the predicted trajectory of at least one key point, and the key point label sequence includes the label trajectory of at least one key point; then, for any key point, the trajectory similarity characterization data between the predicted trajectory of the key point and the label trajectory of the key point is determined, so that the trajectory similarity characterization data can represent the degree of similarity between the predicted trajectory of the key point and the label trajectory of the key point in terms of trajectory length and / or trajectory direction, so that the trajectory similarity The characterization data can better express the degree of jitter of the predicted trajectory of the key point relative to the label trajectory of the key point; finally, based on the trajectory similarity characterization data corresponding to all key points, the key point prediction performance corresponding to the key point prediction sequence is determined, so that the key point prediction performance can at least better express the degree of jitter of the predicted trajectory of each key point relative to its corresponding label trajectory, so that the key point prediction performance can at least better express the prediction stability presented when the key point prediction sequence is predicted for the image sequence, and then the key point prediction performance can better express the prediction performance presented when key point prediction processing is performed on an image sequence.

[0158] In addition, for the data processing method provided by the present application, the present application does not limit the application scenario of the data processing method. For example, it can be applied to any application scenario that requires evaluation and processing of a key point prediction sequence of an image sequence (for example, model training and other scenarios). For better understanding, an application scenario is used as an example for explanation below.

[0159] As an example, in some application scenarios (eg, model training scenarios), the data processing method provided in the present application may include the following steps 51 to 55.

[0160] Step 51: Acquire an image sequence and a key point label sequence corresponding to the image sequence; the image sequence is used to describe at least one key point, and the key point label sequence includes a label trajectory of the at least one key point.

[0161] It should be noted that the present application does not limit the method for obtaining the image sequence in step 51. For example, it can be specifically: randomly extracting an image sequence from the training data set as the image sequence required for the current round of training process. Among them, the training data set refers to the data set required to be based on when training a machine learning model; and the learning goal of the machine learning model in the training process is: to perform key point prediction processing on an image sequence. In addition, the present application does not limit the implementation method of the training data set. For example, the training data set can include multiple image sequences and key point label sequences corresponding to each image sequence.

[0162] It should also be noted that, for the relevant content of the "key point label sequence corresponding to the image sequence" in step 51 above, please refer to S1 above. For the sake of brevity, it will not be repeated here. In addition, the present application does not limit the method of obtaining the "key point label sequence corresponding to the image sequence" in step 51. For example, it can be specifically: after randomly extracting an image sequence from the training data set, extract the key point label sequence corresponding to the image sequence from the training data set as the key point label sequence required to be used in the current round of training process.

[0163] Step 52: perform key point prediction processing on the above image sequence using the key point prediction model to obtain a key point prediction sequence of the image sequence; the key point prediction sequence includes a prediction trajectory of at least one key point in the above.

[0164] Among them, the key point prediction model is used to perform key point prediction processing on the input data of the key point prediction model; and the present application does not limit the key point prediction model. For example, it can be implemented using any existing or future machine learning model that requires learning and training in key point prediction.

[0165] In addition, for the relevant contents of “the key point prediction sequence of the image sequence” in the above step 52, please refer to the above S1, which will not be repeated here for the sake of brevity.

[0166] Step 53: For any key point, determine the trajectory similarity characterization data between the predicted trajectory of the key point and the label trajectory of the key point, where the trajectory similarity characterization data is used to represent the degree of similarity between the predicted trajectory of the key point and the label trajectory of the key point in terms of trajectory length and / or trajectory direction.

[0167] It should be noted that the relevant content of step 53 can be found in S2 above, and for the sake of brevity, it will not be repeated here.

[0168] Step 54: Determine the key point prediction performance corresponding to the key point prediction sequence based on the above trajectory similarity representation data.

[0169] It should be noted that the relevant content of step 54 can be found in S3 above, and for the sake of brevity, it will not be repeated here.

[0170] Step 55: Update the key point prediction model according to the key point prediction performance corresponding to the key point prediction sequence above, and return to continue to execute step 51 above and its subsequent steps until the preset stop condition is reached.

[0171] It should be noted that the present application does not limit the implementation method of the updating process in step 55 above. For example, it can specifically be: first use the key point prediction performance corresponding to the key point prediction sequence above to determine the model loss of the key point prediction model; then update the key point prediction model based on the model loss to obtain an updated key point prediction model, so that the updated key point prediction model has better key point prediction performance, so that the above step 51 and its subsequent steps can be continued to be executed based on the updated key point prediction model to realize the next round of training process for the key point prediction model.

[0172] The preset stop condition mentioned above refers to the condition that needs to be achieved at the end of the training process for the key point prediction model; and the preset stop condition can be set in advance according to the application scenario. For example, the preset stop condition may specifically include: the model loss of the key point prediction model is lower than a preset first threshold. For another example, the preset stop condition may specifically include: the rate of change of the model loss of the key point prediction model is lower than a preset second threshold. For another example, the preset stop condition may specifically include: the number of updates of the key point prediction model is higher than a preset third threshold.

[0173] Based on the above content, it can be known that for the current round of training process for the key point prediction model, if it is determined that the preset stop condition is reached, it can be determined that the prediction performance of the key point prediction model is relatively good, so the training process for the key point prediction model can be directly stopped; however, if it is determined that the preset stop condition is still not reached, it can be determined that the prediction performance of the key point prediction model still needs to be improved, so the key point prediction model can be updated using the key point prediction performance corresponding to the key point prediction sequence determined in the current round of training, so that the updated key point prediction model has better key point prediction performance, so that the above step 51 and its subsequent steps can be continued based on the updated key point prediction model to achieve the next round of training process for the key point prediction model. In this way, the purpose of conducting at least one round of training for the key point prediction model can be achieved.

[0174] Based on the relevant contents of steps 51 to 55 above, it can be known that for each round of training process for the key point prediction model, the key point prediction model can be used to perform key point prediction processing on the image sequence to obtain the key point prediction sequence of the image sequence; then, based on the key point prediction sequence of the image sequence and the key point label sequence corresponding to the image sequence, the key point prediction performance corresponding to the key point prediction sequence is determined; finally, based on the key point prediction performance, the key point prediction model is updated so that the updated key point prediction model has better key point prediction performance. Among them, because the key point prediction performance can better represent the prediction effect presented when the key point prediction sequence is predicted for the image sequence, so that the key point prediction performance can better represent the prediction performance of the key point prediction model, so that the key point prediction performance can better guide the update process of the key point prediction model, and then the key point prediction model obtained by training based on the key point prediction performance has better prediction performance (for example, prediction stability + prediction accuracy), which is conducive to improving the model training effect for the key point prediction model.

[0175] Based on the data processing method provided in the embodiment of the present application, the embodiment of the present application also provides a data processing device. Figure 3 Explain and illustrate. Figure 3 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. It should be noted that for the technical details of the data processing device provided in an embodiment of the present application, please refer to the relevant content of the data processing method above.

[0176] like Figure 3 As shown, the data processing device 300 provided in the embodiment of the present application includes:

[0177] A data acquisition unit 301 is used to acquire a key point prediction sequence of an image sequence and a key point label sequence corresponding to the image sequence; the key point prediction sequence includes a prediction trajectory of at least one key point; the key point label sequence includes a label trajectory of the at least one key point;

[0178] A first determining unit 302 is used to determine, for any of the key points, trajectory similarity characterization data between the predicted trajectory of the key point and the label trajectory of the key point; the trajectory similarity characterization data is used to indicate the degree of similarity between the predicted trajectory of the key point and the label trajectory of the key point in terms of trajectory length and / or trajectory direction;

[0179] The second determining unit 303 is used to determine the key point prediction performance corresponding to the key point prediction sequence according to the trajectory similarity characterization data.

[0180] In a possible implementation manner, the image sequence includes a first image and a second image, a time sequence corresponding to the first image in the image sequence is earlier than a time sequence corresponding to the second image in the image sequence, and a time sequence corresponding to the first image in the image sequence is adjacent to a time sequence corresponding to the second image in the image sequence;

[0181] The predicted trajectory of the key point includes the predicted position corresponding to the key point in the first image and the predicted position corresponding to the key point in the second image;

[0182] The label track of the key point includes the label position corresponding to the key point in the first image and the label position corresponding to the key point in the second image;

[0183] The first determination unit 302 is specifically used to: calculate the distance between the predicted position corresponding to the key point in the second image and the predicted position corresponding to the key point in the first image to obtain the predicted length of the trajectory segment; calculate the distance between the label position corresponding to the key point in the second image and the label position corresponding to the key point in the first image to obtain the label length of the trajectory segment; determine the length similarity representation data according to the difference between the predicted length of the trajectory segment and the label length of the trajectory segment; determine the trajectory similarity representation data based on the length similarity representation data.

[0184] In a possible implementation manner, the image sequence includes a first image and a second image, a time sequence corresponding to the first image in the image sequence is earlier than a time sequence corresponding to the second image in the image sequence, and a time sequence corresponding to the first image in the image sequence is adjacent to a time sequence corresponding to the second image in the image sequence;

[0185] The predicted trajectory of the key point includes the predicted position corresponding to the key point in the first image and the predicted position corresponding to the key point in the second image;

[0186] The label track of the key point includes the label position corresponding to the key point in the first image and the label position corresponding to the key point in the second image;

[0187] The first determination unit 302 is specifically used to: determine the trajectory segment prediction direction representation data based on the direction representation data between the predicted position corresponding to the key point in the second image and the predicted position corresponding to the key point in the first image; determine the trajectory segment label direction representation data based on the direction representation data between the label position corresponding to the key point in the second image and the label position corresponding to the key point in the first image; determine the direction similarity representation data based on the trajectory segment prediction direction representation data and the trajectory segment label direction representation data; determine the trajectory similarity representation data based on the direction similarity representation data.

[0188] In one possible implementation, the trajectory similarity representation data includes length similarity representation data and / or direction similarity representation data; the length similarity representation data is used to indicate the degree of similarity between the predicted trajectory of the key point and the label trajectory of the key point in trajectory length; the direction similarity representation data is used to indicate the degree of similarity between the predicted trajectory of the key point and the label trajectory of the key point in trajectory direction.

[0189] In a possible implementation manner, the second determining unit 303 includes:

[0190] A first determination subunit is used to determine the prediction stability characterization data corresponding to the key point prediction sequence according to the trajectory similarity characterization data;

[0191] The second determining subunit is used to determine the key point prediction performance corresponding to the key point prediction sequence according to the prediction stability characterization data corresponding to the key point prediction sequence.

[0192] In a possible implementation manner, the data processing device 300 further includes:

[0193] A third determining unit, configured to determine prediction accuracy characterization data corresponding to the key point prediction sequence according to the key point prediction sequence and the key point label sequence;

[0194] The second determining subunit is specifically used to determine the key point prediction performance corresponding to the key point prediction sequence according to the prediction stability characterization data corresponding to the key point prediction sequence and the prediction accuracy characterization data corresponding to the key point prediction sequence.

[0195] In a possible implementation manner, the data acquisition unit 301 is specifically used to: perform key point prediction processing on the image sequence using a key point prediction model to obtain a key point prediction sequence of the image sequence;

[0196] The data processing device 300 further includes:

[0197] A model updating unit is used to update the key point prediction model according to the key point prediction performance corresponding to the key point prediction sequence.

[0198] Based on the relevant contents of the above-mentioned data processing device 300, it can be known that for the data processing device 300 provided in the embodiment of the present application, the key point prediction sequence of the image sequence and the key point label sequence corresponding to the image sequence are first obtained, so that the key point prediction sequence includes the predicted trajectory of at least one key point, and the key point label sequence includes the label trajectory of at least one key point; then, for any key point, the trajectory similarity characterization data between the predicted trajectory of the key point and the label trajectory of the key point is determined, so that the trajectory similarity characterization data can represent the degree of similarity between the predicted trajectory of the key point and the label trajectory of the key point in terms of trajectory length and / or trajectory direction, so that the trajectory The trajectory similarity characterization data can better express the degree of jitter of the predicted trajectory of the key point relative to the label trajectory of the key point; finally, the key point prediction performance corresponding to the key point prediction sequence is determined based on the trajectory similarity characterization data corresponding to all key points, so that the key point prediction performance can at least better express the degree of jitter of the predicted trajectory of each key point relative to its corresponding label trajectory, so that the key point prediction performance can at least better express the prediction stability presented when the key point prediction sequence is predicted for the image sequence, and then the key point prediction performance can better express the prediction performance presented when performing key point prediction processing on an image sequence.

[0199] In addition, an embodiment of the present application also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes any implementation of the data processing method provided in the embodiment of the present application.

[0200] See also Figure 4 , which shows a schematic diagram of the structure of an electronic device 400 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0201] like Figure 4As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0202] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0203] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0204] The electronic device provided by the embodiment of the present disclosure and the method provided by the above embodiment belong to the same inventive concept, and the technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0205] The embodiment of the present application also provides a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes any implementation of the data processing method provided in the embodiment of the present application.

[0206] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0207] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0208] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0209] The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device can execute the method.

[0210] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0211] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0212] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit / module does not, in some cases, constitute a limitation on the unit itself.

[0213] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0214] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0215] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0216] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0217] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0218] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0219] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire a key point prediction sequence of an image sequence and a key point label sequence corresponding to the image sequence; the key point prediction sequence includes a prediction trajectory of at least one key point; the key point label sequence includes a label trajectory of the at least one key point; For any of the key points, determining trajectory similarity characterization data between the predicted trajectory of the key point and the label trajectory of the key point; the trajectory similarity characterization data is used to indicate the degree of similarity between the predicted trajectory of the key point and the label trajectory of the key point in terms of trajectory length and / or trajectory direction; The key point prediction performance corresponding to the key point prediction sequence is determined according to the trajectory similarity characterization data.

2. The method according to claim 1, characterized in that: The image sequence includes a first image and a second image, the time sequence corresponding to the first image in the image sequence is earlier than the time sequence corresponding to the second image in the image sequence, and the time sequence corresponding to the first image in the image sequence is adjacent to the time sequence corresponding to the second image in the image sequence; The predicted trajectory of the key point includes the predicted position corresponding to the key point in the first image and the predicted position corresponding to the key point in the second image; The label track of the key point includes the label position corresponding to the key point in the first image and the label position corresponding to the key point in the second image; The process of determining the trajectory similarity characterization data includes: Calculating the distance between the predicted position corresponding to the key point in the second image and the predicted position corresponding to the key point in the first image to obtain a predicted length of the trajectory segment; Calculate the distance between the label position corresponding to the key point in the second image and the label position corresponding to the key point in the first image to obtain the trajectory segment label length; Determining length similarity representation data according to a difference between the predicted length of the trajectory segment and the label length of the trajectory segment; The trajectory similarity characterization data is determined according to the length similarity characterization data.

3. The method according to claim 1, characterized in that: The image sequence includes a first image and a second image, the time sequence corresponding to the first image in the image sequence is earlier than the time sequence corresponding to the second image in the image sequence, and the time sequence corresponding to the first image in the image sequence is adjacent to the time sequence corresponding to the second image in the image sequence; The predicted trajectory of the key point includes the predicted position corresponding to the key point in the first image and the predicted position corresponding to the key point in the second image; The label track of the key point includes the label position corresponding to the key point in the first image and the label position corresponding to the key point in the second image; The process of determining the trajectory similarity characterization data includes: Determining trajectory segment prediction direction representation data according to direction representation data between the predicted position corresponding to the key point in the second image and the predicted position corresponding to the key point in the first image; Determine the trajectory segment label direction representation data according to the direction representation data between the label position corresponding to the key point in the second image and the label position corresponding to the key point in the first image; Determining the direction similarity representation data according to the trajectory segment prediction direction representation data and the trajectory segment label direction representation data; The trajectory similarity characterization data is determined according to the direction similarity characterization data.

4. The method according to claim 1, characterized in that The trajectory similarity characterization data includes length similarity characterization data and / or direction similarity characterization data; the length similarity characterization data is used to indicate the degree of similarity between the predicted trajectory of the key point and the label trajectory of the key point in terms of trajectory length; the direction similarity characterization data is used to indicate the degree of similarity between the predicted trajectory of the key point and the label trajectory of the key point in terms of trajectory direction.

5. The method according to claim 1, characterized in that: Determining the key point prediction performance corresponding to the key point prediction sequence according to the trajectory similarity characterization data includes: Determining prediction stability characterization data corresponding to the key point prediction sequence according to the trajectory similarity characterization data; The key point prediction performance corresponding to the key point prediction sequence is determined according to the prediction stability characterization data corresponding to the key point prediction sequence.

6. The method according to claim 5, characterized in that Before determining the key point prediction performance corresponding to the key point prediction sequence according to the prediction stability characterization data corresponding to the key point prediction sequence, the method further includes: Determining prediction accuracy characterization data corresponding to the key point prediction sequence according to the key point prediction sequence and the key point label sequence; The determining, according to the prediction stability characterization data corresponding to the key point prediction sequence, the key point prediction performance corresponding to the key point prediction sequence comprises: The key point prediction performance corresponding to the key point prediction sequence is determined according to the prediction stability characterization data corresponding to the key point prediction sequence and the prediction accuracy characterization data corresponding to the key point prediction sequence.

7. The method according to any one of claims 1 to 6, characterized in that: The step of obtaining a key point prediction sequence of an image sequence comprises: Performing key point prediction processing on the image sequence using a key point prediction model to obtain a key point prediction sequence of the image sequence; After determining the key point prediction performance corresponding to the key point prediction sequence, the method further includes: The key point prediction model is updated according to the key point prediction performance corresponding to the key point prediction sequence.

8. A data processing device, characterized in that: include: A data acquisition unit, configured to acquire a key point prediction sequence of an image sequence and a key point label sequence corresponding to the image sequence; the key point prediction sequence includes a prediction trajectory of at least one key point; the key point label sequence includes a label trajectory of the at least one key point; A first determining unit is used to determine, for any of the key points, trajectory similarity characterization data between the predicted trajectory of the key point and the label trajectory of the key point; the trajectory similarity characterization data is used to indicate the degree of similarity between the predicted trajectory of the key point and the label trajectory of the key point in terms of trajectory length and / or trajectory direction; The second determining unit is used to determine the key point prediction performance corresponding to the key point prediction sequence according to the trajectory similarity characterization data.

9. An electronic device, characterized in that: The device comprises: a processor and a memory; The memory is used to store instructions or computer programs; The processor is used to execute the instructions or computer programs in the memory so that the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable medium, characterized in that The computer-readable medium stores instructions or computer programs, and when the instructions or computer programs are executed on a device, the device is enabled to execute the method according to any one of claims 1 to 7.

Images