Method and device for video annotation, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a method and device for video annotation, electronic equipment and a storage medium, and the method comprises the steps: extracting a video frame set from a target video; acquiring correction information of each key point in each video frame of the video frame set; based on the correction information and the annotation information of each key point in each video frame of the video frame set, determining whether the annotation information marked by the annotation model is credible; filtering the target video in response to the fact that the labeling information labeled by the labeling model is credible; determining whether the filtering effect meets the requirement or not based on the filtered labeling information of each key point in each video frame of the filtered target video and the labeling information of each key point in each video frame of the target video; and storing the filtered target video in response to the condition that the filtering effect meets the requirement. According to the embodiment of the invention, the video with the high-precision key point position of the target object can be obtained, and the recognition precision of object behavior recognition for the target video can be improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of video data processing, and in particular to a method, device, electronic device and storage medium for video annotation. Background Art

[0002] At present, the target objects in the video images are usually labeled by automatically labeling the manually labeled target objects through target tracking methods and correcting the labeling through manual inspection. Summary of the invention

[0003] According to a first aspect of an embodiment of the present disclosure, a method for video annotation is provided, including:

[0004] Extracting a video frame set from a target video; wherein each video frame of the target video carries annotation information of each key point of a target object in the video frame marked by an annotation model; the annotation information at least includes position coordinates; the video frame set includes a plurality of video frames sampled from the target video;

[0005] Obtaining correction information of each key point in each video frame of the video frame set; wherein the correction information at least includes position coordinates;

[0006] Determining whether the annotation information marked by the annotation model is credible based on the correction information and annotation information of each key point in each video frame of the video frame set;

[0007] In response to the labeling information marked by the labeling model being credible, filtering the target video to obtain a filtered target video; wherein each video frame of the filtered target video carries the filtered labeling information of each key point of the target object;

[0008] Determine whether the filtering effect meets the requirements based on the filtered annotation information of each key point in each video frame of the target video and the annotation information of each key point in each video frame of the target video;

[0009] In response to the filtering effect meeting the requirements, the filtered target video is stored.

[0010] According to a second aspect of an embodiment of the present disclosure, a device for video annotation is provided, including:

[0011] A video frame extraction module is used to extract a video frame set from a target video; wherein each video frame of the target video carries annotation information of each key point of the target object in the video frame marked by the annotation model; the annotation information at least includes position coordinates; the video frame set includes multiple video frames sampled from the target video;

[0012] A correction information acquisition module, used to obtain correction information of each key point in each video frame of the video frame set; wherein the correction information at least includes position coordinates;

[0013] A credible determination module, used to determine whether the annotation information marked by the annotation model is credible based on the correction information and annotation information of each key point in each video frame of the video frame set;

[0014] A video filtering module, configured to filter the target video to obtain a filtered target video in response to the fact that the annotation information marked by the annotation model is credible; wherein each video frame of the filtered target video carries the filtered annotation information of each key point of the target object;

[0015] A filtering effect determination module, used to determine whether the filtering effect meets the requirements based on the filtered annotation information of each key point in each video frame of the target video and the annotation information of each key point in each video frame of the target video;

[0016] The storage module is used for storing the filtered target video in response to the filtering effect meeting the requirements.

[0017] According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program for executing the method for video annotation according to the first aspect.

[0018] According to a fourth aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0019] processor;

[0020] a memory for storing processor-executable instructions;

[0021] The processor is used to read executable instructions from the memory and execute the instructions to implement the method for video annotation of the first aspect.

[0022] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0024] Figure 1is an exemplary system architecture diagram of an embodiment of the method or apparatus for video annotation that can be applied to the present disclosure;

[0025] Figure 2 is a flowchart of a method for video annotation in some embodiments of the present disclosure;

[0026] Figure 3 is a flowchart of step S10 in some embodiments of the present disclosure;

[0027] Figure 4 is a flowchart of step S10-2 in some embodiments of the present disclosure;

[0028] Figure 5 is a flowchart of step S10-2 in some other embodiments of the present disclosure;

[0029] Figure 6 is a flowchart of step S6 in some embodiments of the present disclosure;

[0030] Figure 7 is a flowchart of step S6 in some other embodiments of the present disclosure;

[0031] Figure 8 is a structural block diagram of an apparatus for video annotation in some embodiments of the present disclosure;

[0032] Fig. 9 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0034] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0035] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0036] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0037] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0038] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0039] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0040] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0041] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0042] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, and servers. Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules may include routines, programs, target programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. The computer system / server can be implemented in a distributed cloud computing environment, in which tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on a local or remote computing system storage medium including a storage device.

[0043] Figure 1 is an exemplary system architecture diagram that can be applied to an embodiment of the method or apparatus for video annotation of the present disclosure.

[0044] like Figure 1 As shown, the system architecture may include a terminal 1, a network 2, and a server 3. The network 2 may be a medium for providing a communication link between the terminal 1 and the server 3. The network 2 may include various connection types, such as a wired communication link (which may include optical fiber, cable, etc.), a wireless communication link (which may include Bluetooth, WIFI, etc.).

[0045] The terminal 1 may be a device with a video processing function, including but not limited to a desktop, a laptop, a mobile phone, etc. The terminal 1 may be an all-in-one device, that is, the terminal 1 may be installed with various client applications. The terminal 1 may also be a split device, the terminal 1 includes a display terminal for display and a computing unit with processing capabilities, and the computing unit uploads the picture to the display terminal for display. Here, for the convenience of representation, the all-in-one device and the split device may be collectively referred to as the terminal 1.

[0046] The terminal 1 can capture video. The terminal 1 can send the captured video to the server 3 through the network 2, and the server 3 calls the annotation model or sends the video to another server where the annotation model is located to mark the key points of the object in the video.

[0047] When the solution of the present disclosure is applied to server 3, server 3 extracts a video frame set from the target video, and obtains correction information of each key point in each video frame of the video frame set, and then determines whether the annotation information marked by the annotation model is credible based on the correction information and annotation information of each key point in each video frame of the video frame set, and stores the target video if the annotation information marked by the annotation model is credible.

[0048] It should be noted that the method for video annotation provided in the embodiment of the present disclosure can be executed by the server 3 , and accordingly, the device for video annotation can be set in the server 3 .

[0049] It should also be pointed out that although the solution of the present disclosure can be applied to the server 3 , it does not exclude that the solution can also be applied to the terminal 1 .

[0050] Exemplary Methods

[0051] Figure 2 FIG. 1 is a flow chart of a method for video annotation in some embodiments of the present disclosure. Figure 2 As shown, the following steps are included:

[0052] S2: Extract a set of video frames from the target video.

[0053] In some optional embodiments of the present disclosure, the target video may be a video including a target object captured by a video capture device.

[0054] In some optional embodiments of the present disclosure, the target object may include at least one of a human body, a hand, a face, an animal, etc.

[0055] In some optional embodiments of the present disclosure, the target video may be a video including a human body collected by a video acquisition device. In the first target time period, a human body appears in the collected video, and the target video including the human body may be collected by the video acquisition device. This embodiment may be applicable to the scene of human posture recognition.

[0056] In some optional embodiments of the present disclosure, the target video may be a video in the second target time period collected by a video collection device on the head mounted display device when the user wears the head mounted display device. In the second target time period, the user's hand appears in the collected video, and the target video including the user's hand may be collected by the video collection device on the head mounted display device. This embodiment may be applicable to the scene of gesture recognition.

[0057] In some optional embodiments of the present disclosure, the target video may also be a video including a face image collected by a video acquisition device. In the third target time period, a face appears in the collected video, and the target video including the face may be collected by the video acquisition device. This embodiment may be applicable to scenarios of expression recognition or emotion analysis.

[0058] In some optional embodiments of the present disclosure, the target video may also be a video including the target animal collected using a video acquisition device. In the fourth target time period, the target animal appears in the collected video, and the target video including the target animal may be collected by the video acquisition device. This embodiment may be applicable to the scene of animal behavior analysis.

[0059] In some optional embodiments of the present disclosure, after acquiring the target video, multiple video frames can be extracted from the target video according to a preset sampling frequency, and a video frame set is composed of multiple video frames. Wherein, assuming that the frame rate of the target video is M Hz and the preset sampling frequency is N Hz, then both M and N are greater than 0, and M>N. N can be set according to actual conditions. Assuming that the frame rate of the target video is 30 Hz, that is, 30 frames of images per second, and the preset sampling frequency can be 1 Hz, that is, 1 frame of image is extracted from the target video per second.

[0060] In some optional embodiments of the present disclosure, in the target video, each video frame carries the annotation information of each key point of the target object marked by the annotation model. Among them, the target object may include multiple key points. When the target object is a hand, the multiple key points may include points for characterizing each hand joint and fingertip. When the target object is a face, the multiple key points may include points for characterizing the eyebrows, the center of the eyebrows, the end of the eyebrows, the corners of the eyes, the pupils, the tip of the nose, the corners of the mouth, the center of the upper lip, the center of the lower lip, etc. When the target object is a human body, the multiple key points may include points for characterizing the head, torso, upper limbs, lower limbs, etc. When the target object is an animal, the multiple key points may include points for characterizing the eyes, nose, throat, ear bottom, tail, elbows, knees, claws, etc.

[0061] In some optional embodiments of the present disclosure, the annotation information at least includes the position coordinates for annotating the position of each key point. The annotation information may also include a key point serial number. Because each key point represents a preset part on the target object, the key point serial number can directly determine which part of the target object the marked position coordinates belong to. The key point serial number can be a number or a letter, etc., and the present disclosure does not limit this. In the case where the target object is a human body, if the key point serial number is 4, it can be determined that the position coordinates of the key point are the position coordinates of the waist key point. In the case where the target object is a hand, the annotation information may also include a left hand mark L and a right hand mark R.

[0062] In some optional embodiments of the present disclosure, the annotation information in the target video may be to call a pre-trained annotation model to mark the key points of the target object in the captured video. Alternatively, the video may be sent to another server where the annotation model is located, and the other server calls the annotation model to mark the key points of the target object in the video, and then returns the target video with the annotation information.

[0063] S4: Obtain correction information of each key point in each video frame of the video frame set.

[0064] In some optional embodiments of the present disclosure, after extracting a video frame set, each video frame of the video frame set also carries the annotation information of each key point of the target object in the video frame. If there are inaccurate key point markings in these annotation information, the annotation information of the inaccurate key points in each video frame of the video frame set can be manually adjusted, or each key point in each video frame of the video frame set can be manually re-annotated to obtain the correction information of each key point in each video frame of the video frame set. The adjustment can be to adjust the position coordinates, key point sequence number, etc. in the annotation information.

[0065] S6: Based on the correction information and annotation information of each key point in each video frame of the video frame set, determine whether the annotation information marked by the annotation model is credible.

[0066] In some optional embodiments of the present disclosure, according to a pre-set labeling information credibility condition, it is detected whether the correction information and the labeling information of each key point in each video frame of the video frame set meet the labeling information credibility condition. If the labeling information credibility condition is met, the labeling information marked by the labeling model is determined to be credible, otherwise the labeling information marked by the labeling model is determined to be unreliable. The labeling information credibility condition may include: for each key point of the target object, the position deviation percentile between the labeling information and the correction information of each key point meets the preset distance condition. The preset distance condition may be that the position deviation percentile between the labeling information and the correction information of each key point is not greater than the position deviation threshold. The position deviation threshold may be a pixel value, such as 1 pixel, 2 pixels, 3 pixels, 5 pixels, etc., which can be set according to the accuracy requirements of the labeling.

[0067] S8: In response to the fact that the annotation information marked by the annotation model is credible, the target video is filtered to obtain a filtered target video.

[0068] In some optional embodiments of the present disclosure, when the annotation information marked by the annotation model is credible, the target video can be filtered in various ways. For example, polynomial fitting filtering, Gaussian filtering, median filtering or mean filtering are performed on the data points in the window. After filtering, each video frame of the target video carries the filtered annotation information of each key point of the target object.

[0069] S10: Determine whether the filtering effect meets the requirements based on the filtered annotation information of each key point in each video frame of the filtered target video and the annotation information of each key point in each video frame of the target video.

[0070] In some optional embodiments of the present disclosure, it is possible to detect whether the filtered annotation information of each key point in each video frame of the target video after filtering and the annotation information of each key point in each video frame of the target video meet the preset filtering qualification conditions based on the preset filtering effect index. If the preset filtering qualification conditions are met, it is determined that the filtering effect meets the requirements; otherwise, it is determined that the filtering effect does not meet the requirements.

[0071] S12: In response to the filtering effect meeting the requirements, storing the filtered target video.

[0072] In some optional embodiments of the present disclosure, if the filtering effect does not meet the requirements, the filtering parameters can be adjusted, and the target video is filtered using the adjusted filtering parameters until the filtered target video meets the requirements, and the filtered target video is stored.

[0073] Compared with the time-consuming pure manual labeling or machine labeling followed by manual frame-by-frame correction in the related art, the method of the embodiment of the present invention first labels the target video with a labeling model, and then samples the target video to obtain a video frame set. The labeling information of the video frames in the video frame set can be manually corrected. Thereafter, the embodiment of the present invention determines whether the labeling information marked by the labeling model is credible and filtered and smoothed. When the filtering effect meets the requirements, the labeling of the target video is completed, which greatly improves the labeling efficiency and labeling accuracy, and significantly reduces the degree of manual participation while ensuring accuracy.

[0074] In this embodiment, after extracting a video frame set from the target video, determine whether the annotation information marked by the annotation model is credible based on the annotation information of each key point in each video frame in the video frame set and the correction information of each key point in each video frame in the video frame set. If the annotation information marked by the annotation model is credible, filter the target video and determine whether the filtering effect meets the requirements. If the filtering effect meets the requirements, store the filtered target video. On the one hand, the accuracy of key point annotation for the target video can be improved, and on the other hand, manual participation can be significantly reduced.

[0075] Figure 3 FIG. 1 is a flow chart of step S10 in some embodiments of the present disclosure. Figure 3 As shown, step S10 may include:

[0076] S10-2: Calculate the position error percentile and acceleration jitter percentile of the filtered target video based on the filtered annotation information of each key point in each video frame of the filtered target video and the annotation information of each key point in each video frame of the target video.

[0077] In some optional embodiments of the present disclosure, the position coordinates of each key point of the target object are extracted from the filtered annotation information of the filtered target video, and the position coordinates of each key point of the target object are extracted from the annotation information of the target video, and then the position error percentile and acceleration jitter percentile of each key point of the target object are calculated.

[0078] S10-4: In response to the position error percentile being less than or equal to the position error threshold, and the acceleration jitter percentile being less than or equal to the acceleration jitter threshold, determining that the filtering effect meets the requirements.

[0079] If the position error percentile value is less than the position error threshold value, and the acceleration jitter percentile value is less than the acceleration jitter threshold value, it is determined that the filtering effect meets the requirements.

[0080] If the position error percentile value is less than the position error threshold value, and the acceleration jitter percentile value is equal to the acceleration jitter threshold value, it is determined that the filtering effect meets the requirements.

[0081] If the position error percentile value is equal to the position error threshold value, and the acceleration jitter percentile value is less than the acceleration jitter threshold value, it is determined that the filtering effect meets the requirements.

[0082] If the position error percentile value is equal to the position error threshold value, and the acceleration jitter percentile value is equal to the acceleration jitter threshold value, it is determined that the filtering effect meets the requirements.

[0083] If the position error percentile value is greater than the position error threshold, or the acceleration jitter percentile value is greater than the acceleration jitter threshold, it is determined that the filtering effect does not meet the requirements.

[0084] In this embodiment, the position coordinates of each key point of the target object are extracted from the filtered annotation information, and the position coordinates of each key point of the target object are extracted from the annotation information, and then the error percentile and acceleration jitter percentile of each key point of the target object are calculated. It is possible to measure whether the filtering effect meets the requirements from the position error dimension and the acceleration jitter dimension. If it meets the requirements, the filtered target video is stored, which helps to improve the accuracy of key point annotation for the target video.

[0085] Figure 4 FIG. 1 is a flow chart of calculating the position error percentile value of the filtered target video in step S10-2 in some embodiments of the present disclosure. Figure 4 As shown, step S10-2 may include:

[0086] S10-2-2-A: For any key point in any video frame of the filtered target video, calculate the position error of the key point based on the filtered annotation information of the key point in the video frame and the annotation information of the key point corresponding to the video frame in the target video.

[0087] In some optional embodiments of the present disclosure, the position coordinates of the key point in the filtered target video are extracted from the filtered annotation information, and the position coordinates of the key point are extracted from the annotation information, and then the position error of the key point is calculated.

[0088] In some optional embodiments of the present disclosure, the position error of any key point of the video frame at time t may be calculated by the following formula:

[0089] PError t =abs(P t -P′ t )

[0090] Among them, PError t is the position error of the key point in the video frame at time t; P t is the filtered position coordinate of the key point in the video frame at time t; P′ tis the position coordinate of the key point before filtering in the video frame at time t; P t and P′ t They are all two-dimensional coordinates, such as pixel coordinate system coordinates or image coordinates, etc. abs represents the absolute value of the difference between the position coordinates of the key point after filtering and the position coordinates before filtering.

[0091] S10-2-4-A: Based on the position errors of each key point in the video frame, obtain the position error corresponding to the video frame.

[0092] The position error of each key point in the video frame can be calculated in the same manner as the position error of any key point in S10-2-2-A. The position error of each key point in the video frame is counted to obtain the position error corresponding to the video frame. For example, the average position error of each key point in the video frame can be determined as the position error corresponding to the video frame. The median position error of each key point in the video frame can also be determined as the position error corresponding to the video frame. The position error percentile of each key point in the video frame can also be determined as the position error corresponding to the video frame.

[0093] S10-2-6-A: Based on the position errors corresponding to each video frame in the filtered target video, obtain the position error percentile value of the filtered target video.

[0094] The position error corresponding to each video frame in the target video can be calculated in the same manner as the position error corresponding to a video frame of the target video in S10-2-4-A. The position error corresponding to each video frame in the filtered target video is counted to obtain the position error percentile value of the filtered target video.

[0095] In this embodiment, for any key point in any video frame of the target video after filtering, the position error of the key point can be calculated based on the position coordinates of the key point in the annotation information after filtering and the position coordinates in the annotation information, and the position errors of all key points in any video frame are counted to obtain the position error of any video frame, and then the position errors corresponding to each video frame in the target video after filtering are counted, so that the position error percentile value of the target video after filtering can be quickly and accurately determined.

[0096] Figure 5 FIG. 1 is a flow chart of calculating the acceleration jitter percentile value of the filtered target video in step S10-2 in some embodiments of the present disclosure. Figure 5 As shown, step S10-2 may include:

[0097] S10-2-2-B: For any key point in any video frame of the filtered target video, calculate the acceleration corresponding to the key point and the video frame based on the filtered annotation information of the key point in the video frame, the filtered annotation information corresponding to the key point in the previous video frame and the next video frame adjacent to the video frame, and the time interval between any two adjacent video frames in the filtered target video.

[0098] In some optional embodiments of the present disclosure, the acceleration corresponding to any key point and the video frame at time t may be calculated by the following formula:

[0099] Acc t =(P t+1 -2P t +P t-1 ) / 2(δt) 2

[0100] Among them, Acc t is the acceleration corresponding to the key point and the video frame at time t; P t+1 is the filtered position coordinate of the key point in the video frame at time t+1; t is the filtered position coordinate of the key point in the video frame at time t; P t-1 is the filtered position coordinate of the key point in the video frame at time t-1; δt is the time difference between time t+1 and time t, or it can be the time difference between time t and time t-1.

[0101] S10-2-4-B: Calculate the acceleration jitter value corresponding to the key point and the video frame through the acceleration corresponding to the key point and the video frame and the acceleration corresponding to the next video frame adjacent to the key point and the video frame.

[0102] The acceleration jitter value can be calculated by the following formula:

[0103] Jitter t =(Acc t+1 -Acc t ) 2

[0104] Among them, Jitter t is the acceleration jitter value corresponding to the key point and the video frame at time t, Acc t+1 is the acceleration corresponding to the key point and the video frame at time t+1.

[0105] S10-2-6-B: Based on the acceleration jitter values ​​corresponding to each key point in the video frame and the video frame, obtain the acceleration jitter value corresponding to the video frame.

[0106] The acceleration jitter value corresponding to each key point in the video frame and the video frame can be calculated by the formulas of S10-2-2-B and S10-2-4-B. The acceleration jitter value of each key point in the video frame is counted to obtain the acceleration jitter value corresponding to the video frame. For example, the average acceleration jitter value of each key point in the video frame can be determined as the acceleration jitter value corresponding to the video frame. The median acceleration jitter value of each key point in the video frame can also be determined as the acceleration jitter value corresponding to the video frame. The acceleration jitter percentile value of each key point in the video frame can also be determined as the acceleration jitter value corresponding to the video frame.

[0107] S10-2-8-B: Based on the acceleration jitter values ​​corresponding to each video frame in the filtered target video, the acceleration jitter percentile value of the filtered target video is obtained.

[0108] The acceleration jitter values ​​corresponding to each video frame in the target video can be calculated in the same manner as the acceleration jitter value corresponding to a video frame of the target video in S10-2-6-B. The acceleration jitter values ​​corresponding to each video frame in the filtered target video are statistically calculated to obtain the acceleration jitter percentile value of the filtered target video.

[0109] In some optional embodiments of the present disclosure, the acceleration jitter percentile value of the filtered target video may be the 95% percentile value Jitter of the acceleration jitter values ​​corresponding to each video frame in the filtered target video. .95 Of course, you can also use the 90% quantile Jitter .90 , 97% quantile Jitter .97 , 98% quantile Jitter .98 Etc., which can be set according to actual needs.

[0110] In this embodiment, for any key point in any video frame of the filtered target video, the acceleration corresponding to the key point and the video frame can be determined according to the position coordinates of the key point in the previous video frame adjacent to the video frame, the position coordinates of the video frame and the position coordinates of the next video frame adjacent to the video frame, and the time interval between adjacent video frames. The acceleration jitter value corresponding to the key point and the video frame is calculated according to the acceleration corresponding to the key point and the video frame and the acceleration corresponding to the next video frame adjacent to the video frame. Then, the acceleration jitter values ​​corresponding to each video frame in the filtered target video are counted, and the acceleration jitter percentile value of the filtered target video can be quickly and accurately determined.

[0111] Figure 6 FIG. 5 is a flow chart of step S6 in some embodiments of the present disclosure. Figure 6 As shown, step S6 may include:

[0112] S6-2-A: For any key point in any video frame of the video set, the position deviation of the key point is obtained through the correction information and annotation information of the key point.

[0113] In some optional embodiments of the present disclosure, for any key point in any video frame of a video set, the position coordinates of the key point are extracted from the correction information, and the position coordinates of the key point are extracted from the annotation information, thereby obtaining the position deviation of the key point.

[0114] S6-4-A: Obtain the position deviation corresponding to the video frame through the position deviation of each key point in the video frame.

[0115] In some optional embodiments of the present disclosure, the position deviations of each key point in the video frame are counted, and any one of a preset percentile value of the position deviation of each key point, an average value of the position deviation of each key point, and a median value of the position deviation of each key point can be determined as the position deviation corresponding to the video frame.

[0116] In some optional embodiments of the present disclosure, taking the average value of the position deviations of each key point as an example:

[0117] PE k =AVG(abs(P k manul -P k machine ))

[0118] Among them, PE k is the position deviation corresponding to the video frame at time k; P k manul is the position coordinate of the key point in the video frame at time k in the correction information, which can be considered as the true value; k machine is the position coordinate of the key point in the video frame at time k, which is the position coordinate of the key point predicted by the annotation model; abs(P k manul -P k machine ) is the position deviation of the key point, where abs represents the absolute value of the difference between the position coordinates in the correction information and the position coordinates in the annotation information of the key point; AVG represents the average of the position deviations of each key point in the video frame at time k.

[0119] S6-6-A: Obtain a position deviation percentile value corresponding to the video frame set through the position deviations corresponding to each video frame in the video frame set.

[0120] The position deviation corresponding to each video frame in the video frame set can be calculated in the manner of S6-4-A, and the position deviation corresponding to each video frame in the video frame set can be counted to obtain the position deviation percentile value corresponding to the video frame set. The position deviation percentile value corresponding to the video frame set can be the 95% percentile value PError of the position deviation corresponding to each video frame in the video frame set. .95 Of course, you can also use the 90% quantile value PError .90 、97% quantile PError .97 、98% quantile PError .98 Etc., which can be set according to actual needs.

[0121] S6-8-A: In response to the position deviation percentile value corresponding to the video frame set being less than or equal to the position deviation threshold, determining that the annotation information marked by the annotation model is credible.

[0122] In some optional embodiments of the present disclosure, it is determined whether the position deviation percentile value corresponding to the video set is less than or equal to the position deviation threshold E threshold If yes, it is determined that the annotation information marked by the annotation model is credible, otherwise it is determined that the annotation information marked by the annotation model is not credible. The position deviation threshold may be a pixel value, such as 1 pixel, 2 pixels, 3 pixels, 5 pixels, etc., which may be set according to the accuracy requirements of the annotation.

[0123] In this embodiment, the position deviations corresponding to each video frame in the video frame set are counted to obtain the position deviation percentile value corresponding to the video frame set. When the position deviation percentile value corresponding to the video frame set is less than or equal to the position deviation threshold value, the position annotation accuracy of the annotation model meets the requirements, so that it can be reasonably determined that the annotation information marked by the annotation model is credible.

[0124] Figure 7 FIG. 5 is a flow chart of step S6 in some other embodiments of the present disclosure. Figure 6 As shown, step S6 may also include:

[0125] S6-2-B: For any key point in any video frame of the video set, the position deviation of the key point is obtained through the correction information and annotation information of the key point.

[0126] In some optional embodiments of the present disclosure, for any key point in any video frame of a video set, the position coordinates of the key point are extracted from the correction information, and the position coordinates of the key point are extracted from the annotation information, thereby obtaining the position deviation of the key point.

[0127] S6-4-B: Obtain the position deviation percentile value corresponding to the video frame through the position deviation of each key point in the video frame.

[0128] In some optional embodiments of the present disclosure, the position deviations of each key point in the video frame are counted, and the percentile value of the position deviation of each key point can be determined as the position deviation percentile value corresponding to the video frame. For example, the 95% percentile value of the position deviation of each key point can be determined as the position deviation percentile value corresponding to the video frame.

[0129] S6-6-B: Obtain the position deviation percentile value corresponding to the video frame set through the position deviation percentile value corresponding to each video frame in the video frame set.

[0130] In some optional embodiments of the present disclosure, the average value, median value or percentile value of the position deviation percentile values ​​corresponding to each video frame in the video frame set may be determined as the position deviation percentile value corresponding to the video frame set.

[0131] In some optional embodiments of the present disclosure, the position deviation percentile value corresponding to the video frame at time k can be obtained by the following formula:

[0132] PE k .95 =E .95 (abs(P k manul -P k machine ))

[0133] Among them, PE k .95 is the 95% percentile value of the position deviation corresponding to the video frame at time k; k manul is the position coordinate of the key point in the video frame at time k in the correction information, which can be considered as the true value; k machine is the position coordinate of the key point in the video frame at time k, which is the position coordinate of the key point predicted by the annotation model; abs(P k manul -P k machine ) is the position deviation of the key point, where abs represents the absolute value of the difference between the position coordinates in the correction information and the position coordinates in the annotation information of the key point; E .95 It means taking the 95% percentile value of the position deviation of each key point in the video frame at time k.

[0134] Afterwards, the position deviation percentile value corresponding to each video frame in the video set may be counted to obtain the position deviation percentile value corresponding to the video set.

[0135] S6-8-B: In response to the position deviation percentile corresponding to the video frame set being less than or equal to the position deviation threshold, determining that the annotation information marked by the annotation model is credible.

[0136] In some optional embodiments of the present disclosure, it is determined whether the position deviation percentile value corresponding to the video set is less than or equal to the position deviation threshold E threshold If so, it is determined that the annotation information marked by the annotation model is credible, otherwise it is determined that the annotation information marked by the annotation model is not credible.

[0137] In this embodiment, the position deviation percentile corresponding to each video frame in the video frame set is counted to obtain the position deviation percentile corresponding to the video frame set. When the position deviation percentile corresponding to the video frame set is less than or equal to the position deviation threshold, the position annotation accuracy of the annotation model meets the requirements, so that it can be reasonably determined that the annotation information marked by the annotation model is credible.

[0138] In some embodiments of the present disclosure, after step S10, the following steps may also be included:

[0139] S14: In response to the filtering effect not meeting the requirements, adjusting the filtering parameters.

[0140] S16: Based on the adjusted filtering parameters, the steps of filtering the target video again to obtain the filtered target video and determining whether the filtering effect meets the requirements are performed until the end condition is met. The end condition includes any one of the following: reaching the set filtering times and the filtering effect meets the requirements.

[0141] In some optional embodiments of the present disclosure, at least one filtering parameter may be selected and the selected filtering parameter may be adjusted in a positive or negative direction with a fixed step size, wherein the positive adjustment is to adjust in a manner of increasing the value, and the negative adjustment is to adjust in a manner of decreasing the value.

[0142] If the selected filtering parameters can be adjusted in a positive direction or a negative direction so that the adjusted filtering effect meets the requirements, the adjustment of the filtering parameters is terminated.

[0143] If the selected filter parameters cannot be adjusted forward or reversely to meet the requirements, other filter parameters can be selected and adjusted forward or reversely with a fixed step size to determine whether the adjusted filter effect meets the requirements. When the adjusted filter effect meets the requirements, the adjustment of the filter parameters can be terminated.

[0144] If the filtering effect after adjusting any filtering parameter or any combination of filtering parameters does not meet the requirements, the adjustment of the filtering parameters will be terminated after the set filtering times are reached, and the user may be prompted that the filtering effect does not meet the requirements.

[0145] In this embodiment, when the filtering effect of the target video after filtering once does not meet the requirements, the target video can be filtered again after adjusting the filtering parameters until the filtering effect meets the requirements or the set number of filtering times is reached, so as to obtain the key point marking information of time series smoothness and prevent the filtering behavior from falling into an infinite loop.

[0146] In some optional embodiments of the present disclosure, the filtering parameters include at least one of a sliding window size and a fitting order, that is, by adjusting at least one of the sliding window size and the fitting order, the filtering effect can meet the requirements as soon as possible.

[0147] In some optional embodiments of the present disclosure, after step S12, the following steps may also be included:

[0148] S18: Use the stored video set that at least includes the filtered target video to update the annotation model, so that the updated annotation model annotates the next video to be annotated.

[0149] In some optional embodiments of the present disclosure, the video set includes at least the filtered target video. The video set contains videos with time-series smoothed key point annotation information after filtering. Updating the annotation model using the video set can improve the annotation precision and accuracy of the annotation model, thereby improving the annotation precision and accuracy of each key point of the target object in the next video to be annotated using the updated annotation model.

[0150] In some optional embodiments of the present disclosure, in response to the annotation information marked by the annotation model being credible, it may also include: using the correction information of each key point in each video frame of the video frame set to replace the annotation information of each key point in the corresponding video frame of the target video to obtain a replaced video.

[0151] Then, filtering the target video to obtain the filtered target video in step S8 includes: filtering the replacement video to obtain the filtered replacement video.

[0152] In some optional embodiments of the present disclosure, each key point in each video frame of the replacement video carries initial annotation information, and the initial annotation information is one of the following: annotation information and correction information. That is, in each video frame of the replacement video, if the video frame belongs to the video frame set, the initial annotation information is the correction information; if the video frame does not belong to the video frame set, the initial annotation information is the annotation information.

[0153] Accordingly, step S10 may include: determining whether the filtering effect meets the requirements based on the filtered annotation information of each key point in each video frame of the replaced video and the initial annotation information of each key point in each video frame of the replaced video.

[0154] Accordingly, step S12 may include: in response to the filtering effect meeting the requirements, storing the filtered replacement video.

[0155] In this embodiment, after replacing the annotation information of each key point in the corresponding video frame of the target video with the correction information of each key point in each video frame of the video frame set to obtain a replacement video, the replacement video is filtered to obtain a filtered replacement video, and it is determined whether the filtering effect meets the requirements. If the filtering effect meets the requirements, the filtered replacement video is stored, which can further improve the annotation accuracy of each key point of the target object in the stored video.

[0156] The present disclosure provides another method embodiment for video annotation, which focuses on the differences from the previous embodiment. The method may include the following steps:

[0157] S2': Extract a set of video frames from the target video.

[0158] S4': Obtain correction information of each key point in each video frame of the video frame set.

[0159] S6': Based on the correction information and annotation information of each key point in each video frame of the video frame set, determine whether the annotation information marked by the annotation model is credible.

[0160] Steps S2', S4', and S6' are completely consistent with S2, S4, and S6 in the aforementioned embodiment, and reference may be made to the aforementioned embodiment, and no further details will be given here.

[0161] S8': In response to the fact that the annotation information marked by the annotation model is credible, the annotation information of each key point in each video frame of the video frame set is replaced with the correction information of each key point in the corresponding video frame of the target video to obtain a replacement video.

[0162] In some optional embodiments of the present disclosure, each key point in each video frame of the replacement video carries initial annotation information. The initial annotation information may be one of the following: annotation information and correction information. If the video frame belongs to a video frame set, the initial annotation information is correction information. If the video frame does not belong to a video frame set, the initial annotation information is annotation information.

[0163] Since the correction information is more accurate than the annotation information, using the correction information to replace the annotation information of the corresponding video frame in the target video will improve the annotation accuracy of each key point in the target video.

[0164] S10': Filter the replacement video to obtain a filtered replacement video.

[0165] In some optional embodiments of the present disclosure, various filtering methods can be used to filter the replacement video, such as polynomial fitting filtering, Gaussian filtering, median filtering or mean filtering, etc. Each video frame of the filtered replacement video carries the filtered annotation information of each key point of the target object.

[0166] S12′: Determine whether the filtering effect meets the requirements based on the filtered annotation information of each key point in each video frame of the replaced video and the initial annotation information of each key point in each video frame of the replaced video.

[0167] In some optional embodiments of the present disclosure, it is possible to detect whether the filtered annotation information of each key point in each video frame of the replaced video and the initial annotation information of each key point in each video frame of the replaced video meet the preset filtering qualification conditions according to the preset filtering effect index. If the preset filtering qualification conditions are met, it is determined that the filtering effect meets the requirements, otherwise it is determined that the filtering effect does not meet the requirements.

[0168] S14': In response to the filtering effect meeting the requirement, storing the filtered replacement video.

[0169] In some optional embodiments of the present disclosure, if the filtering effect does not meet the requirements, the filtering parameters can be adjusted, and the replacement video can be filtered using the adjusted filtering parameters until the replaced video meets the requirements after filtering, and the replaced video after filtering is stored.

[0170] In some optional embodiments of the present disclosure, step S12' may include:

[0171] S12'-2: Calculate the position error percentile and acceleration jitter percentile of the filtered replacement video based on the filtered annotation information of each key point in each video frame of the replacement video and the initial annotation information of each key point in each video frame of the replacement video.

[0172] In some optional embodiments of the present disclosure, the position coordinates of each key point of the target object are extracted from the filtered annotation information of the filtered replacement video, and the position coordinates of each key point of the target object are extracted from the initial annotation information of the replacement video, and then the position error percentile and acceleration jitter percentile of each key point of the target object are calculated.

[0173] S12'-4: In response to the position error percentile being less than or equal to the position error threshold, and the acceleration jitter percentile being less than or equal to the acceleration jitter threshold, determining that the filtering effect meets the requirements.

[0174] In some optional embodiments of the present disclosure, calculating the position error percentile value of the filtered replacement video in step S12'-2 may include:

[0175] S12'-2-2-A: For any key point in any video frame of the filtered replacement video, calculate the position error of the key point based on the filtered annotation information of the key point in the video frame and the initial annotation information of the key point corresponding to the video frame in the replacement video.

[0176] In some optional embodiments of the present disclosure, the position coordinates of the key point are extracted from the filtered annotation information of the filtered replacement video, and the position coordinates of the key point are extracted from the initial annotation information of the replacement video, and then the position error of the key point is calculated.

[0177] In some optional embodiments of the present disclosure, the position error of any key point of the video frame at time t may be calculated by the following formula:

[0178] PError t =abs(P t -P′ t )

[0179] Among them, PError t is the position error of the key point in the video frame at time t; P t is the filtered position coordinate of the key point in the video frame at time t; P′ t is the position coordinate of the key point before filtering in the video frame at time t; abs represents the absolute value of the difference between the position coordinate of the key point after filtering and the position coordinate of the key point before filtering.

[0180] S12'-2-4-A: Based on the position errors of each key point in the video frame, obtain the position error corresponding to the video frame.

[0181] The position error of each key point in the video frame can be calculated in the same manner as the position error of any key point in S12'-2-2-A. The position error of each key point in the video frame is counted to obtain the position error corresponding to the video frame. For example, the average position error of each key point in the video frame can be determined as the position error corresponding to the video frame. The median position error of each key point in the video frame can also be determined as the position error corresponding to the video frame. The position error percentile of each key point in the video frame can also be determined as the position error corresponding to the video frame.

[0182] S12'-2-6-A: Based on the position errors corresponding to the respective video frames in the filtered replacement video, obtain the position error percentile value of the filtered replacement video.

[0183] The position error corresponding to each video frame in the replacement video can be calculated in the same manner as the position error corresponding to a video frame of the replacement video in S12'-2-4-A. The position error corresponding to each video frame in the replacement video after filtering is counted to obtain the position error percentile value of the replacement video after filtering.

[0184] In some optional embodiments of the present disclosure, step S12'-2 of calculating the acceleration jitter percentile value of the filtered replacement video may include:

[0185] S12'-2-2-B: For any key point in any video frame of the filtered replacement video, calculate the acceleration corresponding to the key point and the video frame based on the filtered annotation information of the key point in the video frame, the filtered annotation information corresponding to the key point in the previous video frame and the next video frame adjacent to the video frame, and the time interval between any two adjacent video frames in the filtered replacement video.

[0186] In some optional embodiments of the present disclosure, the acceleration corresponding to any key point and the video frame at time t may be calculated by the following formula:

[0187] Acc t =(P t+1 -2P t +P t-1 ) / 2(δt) 2

[0188] Among them, Acc t is the acceleration corresponding to the key point and the video frame at time t; P t+1 is the filtered position coordinate of the key point in the video frame at time t+1; t is the filtered position coordinate of the key point in the video frame at time t; P t-1 is the filtered position coordinate of the key point in the video frame at time t-1; δt is the time difference between time t+1 and time t, or it can be the time difference between time t and time t-1.

[0189] S12'-2-4-B: Calculate the acceleration jitter value corresponding to the key point and the video frame through the acceleration corresponding to the key point and the video frame and the acceleration corresponding to the next video frame adjacent to the key point and the video frame.

[0190] In some optional embodiments of the present disclosure, the acceleration jitter value may be calculated by the following formula:

[0191] Jittert t =(Acc t+1 -Acc t ) 2

[0192] Among them, Jitter t is the acceleration jitter value corresponding to the key point and the video frame at time t, Acc t+1 is the acceleration corresponding to the key point and the video frame at time t+1.

[0193] S12'-2-6-B: Based on the acceleration jitter values ​​corresponding to the video frame at each key point in the video frame, obtain the acceleration jitter value corresponding to the video frame.

[0194] The acceleration jitter value corresponding to each key point in the video frame and the video frame can be calculated by the formulas of S12'-2-2-B and S12'-2-4-B. The acceleration jitter value of each key point in the video frame is counted to obtain the acceleration jitter value corresponding to the video frame. For example, the average acceleration jitter value of each key point in the video frame can be determined as the acceleration jitter value corresponding to the video frame. The median acceleration jitter value of each key point in the video frame can also be determined as the acceleration jitter value corresponding to the video frame. The acceleration jitter value percentile of each key point in the video frame can also be determined as the acceleration jitter value corresponding to the video frame.

[0195] S12'-2-8-B: Based on the acceleration jitter value corresponding to each video frame in the filtered replacement video, obtain the acceleration jitter percentile value of the filtered replacement video.

[0196] The acceleration jitter values ​​corresponding to each video frame in the replacement video can be calculated in the same manner as the acceleration jitter value corresponding to a video frame of the replacement video in S12'-2-6-B. The acceleration jitter values ​​corresponding to each video frame in the replacement video after filtering are statistically calculated to obtain the acceleration jitter percentile value of the replacement video after filtering.

[0197] In some optional embodiments of the present disclosure, the acceleration jitter percentile value of the filtered replacement video may be the 95% percentile value Jitter of the acceleration jitter values ​​corresponding to each video frame in the filtered replacement video. .+5 Of course, you can also use the 90% quantile Jitter .90 , 97% quantile Jitter .97 , 98% quantile Jitter .98 Etc., which can be set according to actual needs.

[0198] In some optional embodiments of the present disclosure, after step S12', the following steps may also be included:

[0199] S16': In response to the filtering effect not meeting the requirements, adjusting the filtering parameters.

[0200] In some optional embodiments of the present disclosure, the filtering parameters include at least one of a sliding window size and a fitting order. At least one filtering parameter may be selected, and the selected filtering parameter may be adjusted in a positive or negative direction with a fixed step size. The positive adjustment is to adjust in a way of increasing the value, and the negative adjustment is to adjust in a way of decreasing the value.

[0201] S18': Based on the adjusted filtering parameters, the steps of filtering the replacement video again to obtain the filtered target video and determining whether the filtering effect meets the requirements are performed until the end condition is met. The end condition includes any one of the following: reaching the set filtering times and the filtering effect meets the requirements.

[0202] In some optional embodiments of the present disclosure, after step S14', the following steps may also be included:

[0203] S20': Use the stored video set that at least includes the filtered replacement video to update the annotation model, so that the updated annotation model annotates the next video to be annotated.

[0204] In some optional embodiments of the present disclosure, the video set includes at least a filtered replacement video. The video set contains videos with time-series smoothed key point annotation information after filtering. Updating the annotation model using the video set can improve the annotation precision and accuracy of the annotation model, thereby improving the annotation precision and accuracy of each key point of the target object in the next video to be annotated using the updated annotation model.

[0205] Since this embodiment has many similarities with the previous embodiment, the introduction is relatively brief. For related contents and effects, please refer to the previous embodiment.

[0206] The various optional embodiments and optional examples disclosed above can be flexibly selected and combined as needed to achieve corresponding functions and effects, and are not listed one by one in this disclosure.

[0207] Any method for video annotation provided in the embodiments of the present disclosure may be executed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server, etc. Alternatively, any method for video annotation provided in the embodiments of the present disclosure may be executed by a processor, such as the processor executing any method for video annotation mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.

[0208] Exemplary Devices

[0209] Figure 8 FIG. 1 is a structural block diagram of a device for video annotation in some embodiments of the present disclosure. Figure 8 As shown, the apparatus for video annotation may include:

[0210] The video frame extraction module 100 is used to extract a video frame set from a target video. Each video frame of the target video carries annotation information of each key point of the target object in the video frame marked by the annotation model. The annotation information at least includes position coordinates. The video frame set includes multiple video frames sampled from the target video.

[0211] The correction information acquisition module 200 is used to acquire the correction information of each key point in each video frame of the video frame set, wherein the correction information at least includes the position coordinates.

[0212] The credible determination module 300 is used to determine whether the annotation information marked by the annotation model is credible based on the correction information and annotation information of each key point in each video frame of the video frame set.

[0213] The video filtering module 400 is used to filter the target video to obtain a filtered target video in response to the labeling information marked by the labeling model being credible, wherein each video frame of the filtered target video carries the filtered labeling information of each key point of the target object.

[0214] The filtering effect determination module 500 is used to determine whether the filtering effect meets the requirements based on the filtered annotation information of each key point in each video frame of the target video and the annotation information of each key point in each video frame of the target video.

[0215] The storage module 600 is used to store the filtered target video in response to the filtering effect meeting the requirements.

[0216] In some optional embodiments of the present disclosure, the filtering effect determination module 500 is specifically used to: calculate the position error percentile and acceleration jitter percentile of the filtered target video based on the filtered annotation information of each key point in each video frame of the filtered target video and the annotation information of each key point in each video frame of the target video. In response to the position error percentile being less than or equal to the position error threshold, and the acceleration jitter percentile being less than or equal to the acceleration jitter threshold, it is determined that the filtering effect meets the requirements.

[0217] In some optional embodiments of the present disclosure, the filtering effect determination module 500 is specifically used to: for any key point in any video frame of the filtered target video, based on the filtered annotation information of the key point in the video frame and the annotation information of the key point corresponding to the video frame in the target video, calculate the position error of the key point. Based on the position errors of each key point in the video frame, obtain the position error corresponding to the video frame. Based on the position errors corresponding to each video frame in the filtered target video, obtain the position error percentile value of the filtered target video.

[0218] In some optional embodiments of the present disclosure, the filtering effect determination module 500 is specifically used to: for any key point in any video frame of the filtered target video, based on the filtered annotation information of the key point in the video frame, the filtered annotation information corresponding to the key point in the previous video frame and the next video frame adjacent to the video frame, and the time interval between any two adjacent video frames in the filtered target video, calculate the acceleration jitter value corresponding to the key point and the video frame through the acceleration corresponding to the key point and the video frame and the acceleration corresponding to the next video frame adjacent to the video frame. Based on the acceleration jitter values ​​corresponding to each key point in the video frame and the video frame, the acceleration jitter value corresponding to the video frame is obtained. Based on the acceleration jitter values ​​corresponding to each video frame in the filtered target video, the acceleration jitter percentile value of the filtered target video is obtained.

[0219] In some optional embodiments of the present disclosure, the credible determination module 300 is specifically used to: for any key point in any video frame of a video set, obtain the position deviation of the key point through the correction information and annotation information of the key point. Obtain the position deviation corresponding to the video frame through the position deviation of each key point in the video frame. Obtain the position deviation percentile corresponding to the video frame set through the position deviation corresponding to each video frame in the video frame set. In response to the position deviation percentile corresponding to the video frame set being less than or equal to the position deviation threshold, determine that the annotation information marked by the annotation model is credible.

[0220] In some optional embodiments of the present disclosure, the credible determination module 300 is specifically used to: for any key point in any video frame of a video set, obtain the position deviation of the key point through the correction information and annotation information of the key point. Obtain the position deviation percentile corresponding to the video frame through the position deviation of each key point in the video frame. Obtain the position deviation percentile corresponding to the video frame set through the position deviation percentile corresponding to each video frame in the video frame set. In response to the position deviation percentile corresponding to the video frame set being less than or equal to the position deviation threshold, determine that the annotation information marked by the annotation model is credible.

[0221] In some optional embodiments of the present disclosure, the filtering effect determination module 500 is further used to: in response to the filtering effect not meeting the requirements, adjust the filtering parameters. Based on the adjusted filtering parameters, the video filtering module 400 is called again to perform the steps of filtering the target video to obtain the filtered target video, and the filtering effect determination module 500 determines whether the filtering effect meets the requirements, until the end condition is met. The end condition includes any one of the following: reaching the set filtering times, and the filtering effect meets the requirements.

[0222] In some optional embodiments of the present disclosure, the filtering parameters include at least one of the following: a sliding window size and a fitting order.

[0223] In some optional embodiments of the present disclosure, the apparatus for video annotation may further include:

[0224] The updating module is used to update the annotation model using the stored video set that at least includes the filtered target video, so that the updated annotation model annotates the next video to be annotated.

[0225] In some optional embodiments of the present disclosure, the video filtering module 400 is further used to: in response to the labeling information marked by the labeling model being credible, replace the labeling information of each key point in the corresponding video frame of the target video with the correction information of each key point in each video frame of the video frame set to obtain a replacement video. Each key point in each video frame of the replacement video carries initial labeling information, and the initial labeling information is one of the following: labeling information and correction information.

[0226] In some optional embodiments of the present disclosure, the video filtering module 400 is further used to filter the replacement video to obtain a filtered replacement video.

[0227] In some optional embodiments of the present disclosure, the filtering effect determination module 500 is also used to determine whether the filtering effect meets the requirements based on the post-filtering annotation information of each key point in each video frame of the replaced video and the initial annotation information of each key point in each video frame of the replaced video.

[0228] In some optional embodiments of the present disclosure, the storage module 600 is further used to: in response to the filtering effect meeting the requirements, store the filtered replacement video.

[0229] It should be noted that the specific implementation of the device for video annotation in the embodiment of the present disclosure is similar to the specific implementation of the method for video annotation in the embodiment of the present disclosure. Please refer to the method for video annotation part for details. In order to reduce redundancy, it is not described again.

[0230] Exemplary Electronic Devices

[0231] Below, reference Fig. 9 To describe the electronic device according to the embodiment of the present disclosure. Fig. 9 As shown, the electronic device includes one or more processors 10 and a memory 20 .

[0232] The memory 20 is used to store executable instructions of the processor 10. The processor 10 is used to read the executable instructions from the memory 20 and execute the instructions to implement the method for video annotation described in the above embodiment.

[0233] The processor 10 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0234] The memory 20 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 10 may run the program instructions to implement the methods for video annotation of the various embodiments of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0235] In some optional embodiments of the present disclosure, the electronic device may further include: an input device 30 and an output device 40, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown). The input device 30 may be, for example, a keyboard, a mouse, etc. The output device 40 may include, for example, a display, a speaker, a printer, and a communication network and a remote output device connected thereto, etc.

[0236] Of course, to simplify, Fig. 9 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.

[0237] Exemplary computer-readable storage media

[0238] The computer-readable storage medium stores a computer program, and the computer program is used to execute the method for video annotation described in the above embodiment.

[0239] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0240] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0241] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0242] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0243] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0244] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0245] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0246] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for video annotation, comprising: Extracting a video frame set from a target video; wherein each video frame of the target video carries annotation information of each key point of the target object in the video frame marked by an annotation model; the annotation information at least includes position coordinates; the video frame set includes a plurality of video frames sampled from the target video; Obtaining correction information of each key point in each video frame of the video frame set; wherein the correction information at least includes position coordinates; Determining whether the annotation information marked by the annotation model is credible based on the correction information and annotation information of each key point in each video frame of the video frame set; In response to the labeling information marked by the labeling model being credible, filtering the target video to obtain a filtered target video; wherein each video frame of the filtered target video carries the filtered labeling information of each key point of the target object; Determining whether the filtering effect meets the requirements based on the filtered annotation information of each key point in each video frame of the filtered target video and the annotation information of each key point in each video frame of the target video; In response to the filtering effect meeting the requirement, the filtered target video is stored.

2. The method according to claim 1, wherein: Determining whether the filtering effect meets the requirements based on the filtered annotation information of each key point in each video frame of the filtered target video and the annotation information of each key point in each video frame of the target video includes: Calculate the position error percentile and acceleration jitter percentile of the filtered target video based on the filtered annotation information of each key point in each video frame of the filtered target video and the annotation information of each key point in each video frame of the target video; In response to the position error percentile being less than or equal to a position error threshold, and the acceleration jitter percentile being less than or equal to an acceleration jitter threshold, it is determined that the filtering effect meets the requirements.

3. The method according to claim 2, wherein: The calculating the position error percentile value of the filtered target video based on the filtered annotation information of each key point in each video frame of the filtered target video and the annotation information of each key point in each video frame of the target video comprises: For any key point in any video frame of the filtered target video: Calculating a position error of the key point based on the filtered annotation information of the key point in the video frame and the annotation information of the key point corresponding to the video frame in the target video; Based on the position errors of each key point in the video frame, obtaining the position error corresponding to the video frame; Based on the position errors corresponding to the respective video frames in the filtered target video, the position error percentile values ​​of the filtered target video are obtained.

4. The method according to claim 2, wherein: The calculating the acceleration jitter percentile value of the filtered target video based on the filtered annotation information of each key point in each video frame of the filtered target video and the annotation information of each key point in each video frame of the target video includes: For any key point in any video frame of the filtered target video: Calculate the acceleration corresponding to the key point and the video frame based on the filtered annotation information of the key point in the video frame, the filtered annotation information corresponding to the key point in the previous video frame and the next video frame adjacent to the video frame, and the time interval between any two adjacent video frames in the filtered target video; Calculate the acceleration jitter value corresponding to the key point and the video frame through the acceleration corresponding to the key point and the video frame and the acceleration corresponding to the next video frame adjacent to the key point and the video frame; Based on the acceleration jitter values ​​corresponding to the video frame at each key point in the video frame, the acceleration jitter value corresponding to the video frame is obtained; Based on the acceleration jitter values ​​corresponding to the respective video frames in the filtered target video, the acceleration jitter percentile values ​​of the filtered target video are obtained.

5. The method according to claim 1, wherein: The determining whether the annotation information marked by the annotation model is credible based on the correction information and annotation information of each key point in each video frame of the video frame set includes: For any key point in any video frame of the video set: The position deviation of the key point is obtained through the correction information and the annotation information of the key point; Obtaining the position deviation corresponding to the video frame through the position deviation of each key point in the video frame; Obtaining a position deviation percentile value corresponding to the video frame set through position deviations corresponding to each video frame in the video frame set; In response to the position deviation percentile corresponding to the video frame set being less than or equal to a position deviation threshold, it is determined that the annotation information marked by the annotation model is credible.

6. The method according to claim 1, wherein: The determining whether the annotation information marked by the annotation model is credible based on the correction information and annotation information of each key point in each video frame of the video frame set includes: For any key point in any video frame of the video set: The position deviation of the key point is obtained through the correction information and the annotation information of the key point; Obtaining a position deviation percentile value corresponding to the video frame through the position deviation of each key point in the video frame; Obtaining a position deviation percentile value corresponding to the video frame set through the position deviation percentile values ​​corresponding to each video frame in the video frame set; In response to the position deviation percentile corresponding to the video frame set being less than or equal to a position deviation threshold, it is determined that the annotation information marked by the annotation model is credible.

7. The method according to claim 1, wherein: Also includes: In response to the filtering effect not meeting the requirements, adjusting the filtering parameters; Based on the adjusted filtering parameters, the steps of filtering the target video to obtain the filtered target video and determining whether the filtering effect meets the requirements are performed again until the end condition is met; wherein the end condition includes any one of the following: reaching the set number of filtering times and the filtering effect meets the requirements.

8. The method according to any one of claims 7, wherein: The filtering parameters include at least one of the following: sliding window size and fitting order.

9. The method according to claim 1, wherein: Also includes: The labeling model is updated using a stored video set that at least includes the filtered target video, so that the updated labeling model labels the next video to be labeled.

10. The method according to any one of claims 1 to 9, wherein: In response to the annotation information marked by the annotation model being credible, the method further includes: Using the correction information of each key point in each video frame of the video frame set to replace the annotation information of each key point in the corresponding video frame of the target video, to obtain a replacement video; wherein each key point in each video frame of the replacement video carries initial annotation information, and the initial annotation information is one of the following: the annotation information and the correction information; The filtering the target video to obtain a filtered target video comprises: filtering the replacement video to obtain a filtered replacement video; The determining whether the filtering effect meets the requirements based on the filtered annotation information of each key point in each video frame of the filtered target video and the annotation information of each key point in each video frame of the target video includes: determining whether the filtering effect meets the requirements based on the filtered annotation information of each key point in each video frame of the filtered replacement video and the initial annotation information of each key point in each video frame of the replacement video; In response to the filtering effect meeting the requirement, storing the filtered target video includes: in response to the filtering effect meeting the requirement, storing the filtered replacement video.

11. A device for video annotation, comprising: A video frame extraction module, used to extract a video frame set from a target video; wherein each video frame of the target video carries annotation information of each key point of the target object in the video frame marked by the annotation model; the annotation information at least includes position coordinates; the video frame set includes multiple video frames sampled from the target video; A correction information acquisition module, used to acquire correction information of each key point in each video frame of the video frame set; wherein the correction information at least includes position coordinates; A credible determination module, used to determine whether the annotation information marked by the annotation model is credible based on the correction information and annotation information of each key point in each video frame of the video frame set; A video filtering module, configured to filter the target video to obtain a filtered target video in response to the labeling information marked by the labeling model being credible; wherein each video frame of the filtered target video carries the filtered labeling information of each key point of the target object; A filtering effect determination module, used to determine whether the filtering effect meets the requirements based on the filtered annotation information of each key point in each video frame of the filtered target video and the annotation information of each key point in each video frame of the target video; The storage module is used for storing the filtered target video in response to the filtering effect meeting the requirements.

12. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1-10.

13. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the method according to any one of claims 1 to 10.

Images