Method, electronic device, and storage medium for video action timestamp refinement
Patent Information
- Application Number
- CN202611040302.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-14
- Publication Date
- 2026-10-13
AI Technical Summary
[0007]本发明旨在解决上述技术问题，即，解决现有基于VLM的视频动作分段方法中推理输出的动作时间边界通常精度不足的问题
[0039]语义感知的精确定位：基于动作语义类型动态生成差异化起止规则，使VLM能够理解不同类型动作的时间边界判定标准（如拿起以"手握住物体"为准，放下以"物体下降"为准），显著提升时间边界定位的准确性。
Smart Images
Figure CN122909445A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and image recognition technology, and specifically provides a method, electronic device, and storage medium for refining video action timestamps. Background Technology
[0002] The Vision Language Model (VLM) is a multimodal generative artificial intelligence model that combines a language model with a visual encoder to achieve integrated processing of images, text, and video.
[0003] In VLM-based video action segmentation systems, the timing boundaries of the initial inference output are often insufficiently accurate. Existing timing boundary optimization methods mainly include:
[0004] The boundary post-processing method based on Non-Maximum Suppression (NMS) merges highly overlapping annotations in the action annotations output by VLM using NMS, and adjusts the boundary position by weighted averaging based on confidence. This method processes based on temporal overlap and confidence, without understanding the semantic content of the action, and cannot distinguish the differentiated temporal boundary requirements of different semantic types of actions (e.g., the start of "pick up" should be based on "hand holding the object", while the start of "put down" should be based on "object falling").
[0005] The boundary detection method based on optical flow / feature abrupt changes sets the action boundary at the frame with the greatest change by calculating the optical flow amplitude or visual feature cosine distance between adjacent frames. However, in scenarios with subtle operations where visual changes are not obvious (such as tightening, pressing, fine-tuning, etc.), the optical flow amplitude change is insufficient to accurately locate the boundary, resulting in a large error.
[0006] The unified re-inference method based on VLM uses the initial output annotations of VLM as a reference and re-inputs them into VLM along with the complete video frame sequence, requiring the model to adjust the temporal boundaries based on the reference annotations. This method inputs the entire video and all annotations into VLM simultaneously, requiring the model to handle boundary adjustments for multiple actions in a long context. This attentional distraction leads to limited accuracy improvement, and the inference cost is proportional to the length of the entire video, resulting in low efficiency. Summary of the Invention
[0007] The present invention aims to solve the above-mentioned technical problems, namely, to solve the problem that the motion time boundary of the inference output in existing VLM-based video motion segmentation methods is usually not accurate enough.
[0008] In a first aspect, the present invention provides a method for refining video action timestamps, comprising:
[0009] Based on the action annotation information of the target video, a refined search range for each action in the target video is determined, wherein the action annotation information includes the first start time, the first end time, the action name, and the description information of each action;
[0010] Based on the action name or description information of each action, determine the start and end rules corresponding to each action;
[0011] Using a visual language model, based on the start and end rules corresponding to each action, the refined search range of each action, and refined prompt word information, the start and end times of each action are refined and located to obtain the second start time and the second end time of each action.
[0012] In some embodiments of the present invention, determining the refined search range of each action in the target video based on the action annotation information of the target video includes:
[0013] Based on the description information of each action, determine whether each action is a repetitive action;
[0014] For each non-repetitive action, the search starting point of each non-repetitive action is determined based on the first start time of the previous action and the refined start time, and the search ending point of each non-repetitive action is determined based on the end time of the next action and the total duration of the target video.
[0015] In some embodiments of the present invention, determining the start and end rules corresponding to each action based on the action name or description information of each action includes:
[0016] Based on the keywords in the action name or description of each action, determine the start and end rules corresponding to each action.
[0017] In some embodiments of the present invention, the step of using a visual language model to refine and locate the start and end times of each action based on the start and end rules corresponding to each action, the refined search range of each action, and refined prompt word information, to obtain the second start time and the second end time of each action, includes:
[0018] The refined prompt word information is constructed based on the refined prompt word template, the description information of each action, and the start and end rules corresponding to each action;
[0019] Based on the search range of each action, extract the target video frame sequence from the target video;
[0020] Using the visual language model, based on the refined prompt information and the target video frame, the start and end times of each action are refined and located to obtain the third start time and the third end time of each action.
[0021] The third start time and third end time of each action are checked for minimum duration, and the second start time and second end time of each action are determined based on the minimum duration check result.
[0022] In some embodiments of the present invention, after refining the start and end times of each action by utilizing a visual language model, based on the start and end rules corresponding to each action, the refined search range of each action, and refined prompt word information, to obtain the second start time and the second end time of each action, the method further includes:
[0023] The second start time and the second end time of each action are subjected to time overlap resolution processing.
[0024] In some embodiments of the present invention, the time overlap resolution process for the second start time and the second end time of each action includes at least one of the following:
[0025] In response to a time overlap between the second start time of a later action and the second end time of a earlier action between adjacent action pairs, if the duration of the later action is greater than the duration of the earlier action, the second start time of the later action is adjusted; if the duration of the later action is less than the duration of the earlier action, the second end time of the earlier action is adjusted.
[0026] For each action with a preset semantic type, the second start time is adjusted to the second end time of the previous action;
[0027] In response to the fact that the later action and the earlier action are consecutive actions between adjacent action pairs, the second end time of the earlier action is adjusted to the second start time of the later action.
[0028] In some embodiments of the present invention, after performing time overlap resolution processing on the second start time and second end time of each action, the method further includes:
[0029] Based on the second start time and second end time of each action after time overlap resolution processing, the video frame sequence corresponding to each action is extracted from the target video.
[0030] Based on the video frame sequence corresponding to each action, the description information of each action, and the description information of adjacent actions of each action, the consistency of visual content and description information is checked to obtain the consistency check result of each action.
[0031] In some embodiments of the present invention, after performing a consistency check between visual content and descriptive information based on the video frame sequence corresponding to each action, the descriptive information of each action, and the descriptive information of adjacent actions of each action, and obtaining the consistency check result of each action, the method further includes:
[0032] Provide prompts for actions whose visual content and descriptive information are inconsistent in the consistency verification results.
[0033] A second aspect of the present invention provides an electronic device, comprising:
[0034] Memory, used to store computer program products;
[0035] A processor is configured to execute a computer program product stored in the memory, and when the computer program product is executed, to implement the method described in the first aspect above.
[0036] A third aspect of this disclosure provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the method described in the first aspect.
[0037] A fifth aspect of this disclosure provides a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform the method described in the first aspect.
[0038] When the above technical solution is adopted, the present invention has the following technical effects:
[0039] Precise positioning based on semantic perception: Based on the dynamic generation of differentiated start and end rules according to the semantic type of the action, VLM can understand the time boundary judgment criteria of different types of actions (such as picking up is based on "the hand grasps the object", and putting down is based on "the object falls"), which significantly improves the accuracy of time boundary positioning.
[0040] Sub-second time precision: By limiting the search range for each action and using VLM fine-grained inference, the time boundary precision is improved from the initial second level to the sub-second level, meeting the requirements of fine-grained operation review.
[0041] Physically sound timeline: The three-stage overlap resolution algorithm ensures that the output timeline is non-overlapping, navigation actions are seamless, and pick-up and put-down actions are uninterrupted, fully conforming to the physical logic of actual operation.
[0042] Semantic constraints of preceding and following context: The semantic descriptions of adjacent actions are injected into the start and end rules as time anchors, which effectively prevents VLM from positioning the boundary of the current action within the time range of adjacent actions.
[0043] Robust handling of repetitive actions: By detecting adjacent actions with the same description and skipping VLM refinement, the boundaries of repetitive actions are prevented from being incorrectly merged, thus maintaining the rationality of the original segmentation.
[0044] The refining process does not produce cascading offsets: the design of calculating the search range through deep copy + original timestamp ensures that the new timestamp of the refined action will not affect the calculation of the search range of subsequent actions.
[0045] Double reliability assurance: minimum duration verification + VLM retry mechanism ensures the validity of the refined results, segment visual verification provides independent verification of the refined results, and suspicious labels are marked for manual review. Attached Figure Description
[0046] The preferred embodiments of the present invention are described below with reference to the accompanying drawings, in which:
[0047] Figure 1 This is a flowchart illustrating the method for refining video action timestamps in some embodiments of the present invention;
[0048] Figure 2 These are structural block diagrams of electronic devices in some embodiments of the present invention. Detailed Implementation
[0049] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.
[0050] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.
[0051] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.
[0052] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.
[0053] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.
[0054] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.
[0055] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.
[0056] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.
[0057] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.
[0058] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.
[0059] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.
[0060] Here, we will introduce the relevant terminology involved in this invention.
[0061] Timestamp Refinement: Fine-tunes the start and end time boundaries of the initial action annotation.
[0062] Dynamic Start / End Rules: Automatically generate differentiated time boundary determination criteria based on the semantic type of the action.
[0063] Search Range: The time range of the video frame subsequence determined for each action to be refined, which limits the localization search space of the VLM.
[0064] Overlap Resolution: When refined adjacent actions overlap in time, the time boundaries are adjusted according to a preset strategy to eliminate the overlap.
[0065] Navigation Action: Operator's own displacement actions (such as moving forward, walking, turning around, standing up, etc.), which have different time boundary processing logic from object manipulation actions.
[0066] Pick-Place Bridging: When a pick-up action is immediately followed by a put-down action, the end time of the pick-up is aligned with the start time of the put-down action to ensure the continuity of the operation.
[0067] Clip Visual Verification: For each refined action annotation, the consistency between the video content and the action description within its time frame is verified using VLM.
[0068] Figure 1 This is a flowchart illustrating the method for refining video action timestamps in some embodiments of the present invention. For example... Figure 1 As shown, the method for refining video action timestamps includes the following steps:
[0069] S1: Based on the action annotation information of the target video, determine the refined search range of each action in the target video.
[0070] Among them, the action annotation information is obtained by processing the target video using image recognition technology and automatic annotation technology through a pre-trained multimodal model.
[0071] The action annotation information includes the first start time (i.e., the start time of the multimodal model output without timestamp refinement), the first end time (i.e., the end time of the multimodal model output without timestamp refinement), the action name, and the description information for each action. The action annotation information for each action can form a first action annotation sequence, which is stored separately from the video frame sequence corresponding to the target video (e.g., all video frames provided in the form of an OSS URL list).
[0072] Using the first start time and first end time of each action as a benchmark, and combining the action name (e.g., picking up / putting down an item) and descriptive information (further description of the action name, such as someone picking up an item with a certain hand), image recognition technology can be used to obtain a refined search range for each action in the target video.
[0073] In some embodiments of the present invention, S1 may include the following steps:
[0074] S1-1: Based on the description information of each action, determine whether each action is a repetitive action.
[0075] For each action, check if it is a duplicate action: compare the description_cn of the current action with the description_cn of the previous and next actions. If it is exactly the same as the description of any adjacent action, it is determined to be a duplicate action, and its initial timestamp is directly retained, skipping VLM refinement. This strategy avoids confusion in VLM when processing duplicate actions and prevents incorrect merging of time boundaries.
[0076] S1-2: For each non-repetitive action, determine the search starting point of each non-repetitive action based on the first start time of the previous action and the refined start time, and determine the search ending point of each non-repetitive action based on the end time of the next action and the total duration of the target video.
[0077] For non-repetitive actions, the search scope is determined in the following way.
[0078] Search starting point: Take the larger value between the start time of the previous action and the refined start time of the previous action (if the previous action has been refined); if it is the first action, the search starting point is 0.
[0079] Search endpoint: Take the end time of the next action; if it is the last action, the search endpoint is the total video duration.
[0080] Based on the time interval of the search range, the corresponding frame subsequence is extracted from the video frame sequence and used as the visual input for VLM refined inference.
[0081] The present invention uses the search range of each action to make the VLM search space cover the current action and the transition area before and after it, which avoids the inaccuracy of positioning due to the search range being too large, and avoids the omission of the real boundary due to the search range being too small.
[0082] S2: Based on the action name or description information of each action, determine the start and end rules corresponding to each action.
[0083] In some embodiments of the present invention, S2 may include the following steps: determining the start and end rules corresponding to each action based on the action name or keywords in the description information of each action.
[0084] In some examples of this invention, the start and end rules corresponding to each action are determined in the following manner.
[0085] For actions involving picking up objects (matching keywords: pick up, take, grab, hold), the starting rule is: the hand firmly grips the object; the ending rule is: the object is noticeably lifted.
[0086] For actions involving putting down (matching keywords: put down, place, put in, put in), the starting rule is: the object clearly descends; the ending rule is: the hand clearly releases the object.
[0087] For general actions (not matching the keywords above), the starting rule is: the action is explicitly started; the ending rule is: the entire process of the action is completed.
[0088] Add semantic descriptions of adjacent actions as temporal constraint anchors after the rule text. Add the following to the start rule: (after the previous action has been completely completed); add the following to the end rule: (before the next action has started).
[0089] S3: Using a visual language model, based on the start and end rules corresponding to each action, the refined search range of each action, and the refined prompt word information, the start and end times of each action are refined and located to obtain the second start time and the second end time of each action.
[0090] Among them, the refined prompt information is used to refine and limit the start and end times of each action in the visual language model, which can improve the efficiency and accuracy of refinement.
[0091] By leveraging the visual processing capabilities of the visual language model, within the refined search range of each action and the limited range of refined prompt information, and combined with the start and end rules for determining the start and end times of each task, timestamps can be refined for each action to obtain the refined start and end times of each action, which are recorded as the second start time and the second end time of each action, respectively.
[0092] In some embodiments of the present invention, S3 may include the following steps:
[0093] S3-1: Construct refined prompt information based on the refined prompt template, the description information of each action, and the start and end rules corresponding to each action.
[0094] The refined prompt template is a text template file containing multiple placeholders. Replacing the placeholders in the refined prompt template with specific values yields the refined prompt information.
[0095] In some examples of this invention, the placeholders in the refined prompt template are replaced in the following way:
[0096] [ACTION_DESC]: Description of the current action (description_cn);
[0097] [PREV_DESC]: Description of the previous action; if none, it indicates preparation to begin.
[0098] [NEXT_DESC]: Description of the next action; if none, the task is complete.
[0099] [CUR_VISUAL]: Visual content description of the current action;
[0100] [PREV_VISUAL]: Visual description of the previous action;
[0101] [NEXT_VISUAL]: Visual description of the next action;
[0102] [START_RULE]: The starting rule for each action;
[0103] [END_RULE]: The termination rule for each action;
[0104] [CUR_TIMESTAMP]: The time offset of the current action's name relative to the start of the search range;
[0105] [MIN_DURATION]: Minimum duration constraint (e.g., 0.4 seconds).
[0106] The frame subsequence (e.g., less than 4 frames as individual images, 4 frames or more as video) is assembled with refined cue words into a VLM input message.
[0107] S3-2: Extract the target video frame sequence from the target video based on the search range of each action.
[0108] S3-3: Using a visual language model, based on refined cue word information and target video frames, the start and end times of each action are refined and located to obtain the third start time and the third end time of each action.
[0109] The VLM is invoked to infer the frame subsequence and output the time offset relative to the start of the frame subsequence. The relative time is added to the absolute start time of the search range to obtain the refined start and end times in the global time coordinate system, which are the third start time and the third end time of each action.
[0110] S3-4: Perform minimum duration verification on the third start time and third end time of each action, and determine the second start time and second end time of each action based on the minimum duration verification result.
[0111] Minimum duration check is performed on each action: If the time difference between the third start time and the third end time of each action is less than the minimum duration (e.g., 0.4 seconds), the refinement is deemed to have failed, and a retry is triggered. A maximum of 5 retries are allowed, with a 1-second interval between each. If the maximum number of retries is exceeded, the first start time and the second end time of each action are used as a fallback.
[0112] In some embodiments of the present invention, the following steps may be included after S3:
[0113] S4: Perform time overlap resolution on the second start time and the second end time of each action.
[0114] Since there may be time overlap between adjacent actions, for example, there is a time overlap between the second end time of the first action and the second start time of the second action after the first action. Therefore, time overlap resolution is performed according to the preset time overlap resolution rules based on the second start time and the second end time.
[0115] In some embodiments of the present invention, S4 may include the following steps:
[0116] S4-1: In response to a time overlap between the second start time of the later action and the second end time of the earlier action between adjacent action pairs, if the duration of the later action is greater than the duration of the earlier action, the second start time of the later action is adjusted; if the duration of the later action is less than the duration of the earlier action, the second end time of the earlier action is adjusted.
[0117] Iterate through all adjacent action pairs. When the second start time of the later action in an adjacent action pair is earlier than the second end time of the earlier action (there is a time overlap), compare the duration of the later action with that of the earlier action. Actions with shorter durations retain their time boundaries unchanged, while actions with longer durations yield to each other at the overlap point (i.e., the second end time of the earlier action is truncated to the second start time of the later action, or the second start time of the later action is postponed to the second end time of the earlier action).
[0118] S4-2: Adjust the second start time of each action with a semantic type of the preset type to the second end time of the previous action.
[0119] Iterate through all action annotations. For annotations with the semantic type of navigation action (action name matching: forward, walk, stand up, operator self-movement, squat, squat down, sit down, turn around, and stand up, etc.), set its second start time to the second end time of the previous action (if it exists), and set its second end time to the second start time of the next action (if it exists). This strategy enables navigation actions to automatically fill the gaps between adjacent operation actions, achieving seamless timeline transitions.
[0120] S4-3: In response to the fact that the later action and the earlier action are consecutive actions between adjacent action pairs, the second end time of the earlier action is adjusted to the second start time of the later action.
[0121] Iterate through all adjacent action pairs. When the preceding action in an adjacent action pair is a pick-up type (action name matches pick-up / take / grab / hold) and the following action is a put-down type (action_name matches put-down / place / put in / place), align the second end time of the preceding action with the second start time of the following action to ensure the time continuity of the pick-up and put-down operation process.
[0122] In some embodiments of the present invention, the following steps may be included after S4:
[0123] S5: Based on the second start time and second end time of each action after time overlap resolution, extract the video frame sequence corresponding to each action from the target video.
[0124] S6: Based on the video frame sequence corresponding to each action, the description information of each action, and the description information of adjacent actions, perform consistency verification between visual content and description information to obtain the consistency verification result of each action.
[0125] Based on the annotation information of each action, extract the video frame subsequence corresponding to its refined time range (second start time and second end time), construct a verification prompt word containing the current action description and the descriptions of the actions before and after, and call VLM to verify whether the visual content in the frame subsequence is consistent with the action description.
[0126] S7: Provide a prompt for actions that are inconsistent between the visual content and the descriptive information in the consistency verification results, for reference by manual reviewers.
[0127] In addition, embodiments of this disclosure also provide an electronic device, including:
[0128] Memory, used to store computer programs;
[0129] A processor is configured to execute a computer program stored in the memory, wherein, when the computer program is executed, it implements the video action timestamp refinement method described in any of the above embodiments of the present disclosure.
[0130] Below, for reference Figure 2 To describe an electronic device according to embodiments of this disclosure. For example... Figure 2 As shown, the electronic device includes one or more processors and memory.
[0131] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.
[0132] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the video action timestamp refinement methods of the various embodiments of this disclosure described above, and / or other desired functions.
[0133] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).
[0134] In addition, the input device may also include, for example, a keyboard, a mouse, etc.
[0135] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.
[0136] Of course, for the sake of simplicity, Figure 2 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.
[0137] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods for refining video motion timestamps according to various embodiments of this disclosure as described in the foregoing portion of this specification.
[0138] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of this disclosure. The programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0139] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps of the video motion timestamp refinement method according to various embodiments of this disclosure as described in the foregoing portion of this specification.
[0140] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.
[0141] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.
[0142] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
[0143] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.
[0144] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.
[0145] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.
[0146] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.
[0147] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.
[0148] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will all fall within the scope of protection of the present invention.
Claims
1. A method for refining video action timestamps, characterized in that, include: Based on the action annotation information of the target video, a refined search range for each action in the target video is determined, wherein the action annotation information includes the first start time, the first end time, the action name, and the description information of each action; Based on the action name or description information of each action, determine the start and end rules corresponding to each action; Using a visual language model, based on the start and end rules corresponding to each action, the refined search range of each action, and refined prompt word information, the start and end times of each action are refined and located to obtain the second start time and the second end time of each action.
2. The method according to claim 1, characterized in that, The determination of the refined search range for each action in the target video based on the action annotation information of the target video includes: Based on the description information of each action, determine whether each action is a repetitive action; For each non-repetitive action, the search starting point of each non-repetitive action is determined based on the first start time of the previous action and the refined start time, and the search ending point of each non-repetitive action is determined based on the end time of the next action and the total duration of the target video.
3. The method according to claim 1, characterized in that, The step of determining the start and end rules corresponding to each action based on the action name or description information of each action includes: Based on the keywords in the action name or description of each action, determine the start and end rules corresponding to each action.
4. The method according to claim 1, characterized in that, The method utilizes a visual language model to refine and locate the start and end times of each action based on the start and end rules corresponding to each action, the refined search range of each action, and refined prompt word information, thereby obtaining the second start time and the second end time of each action, including: The refined prompt word information is constructed based on the refined prompt word template, the description information of each action, and the start and end rules corresponding to each action; Based on the search range of each action, extract the target video frame sequence from the target video; Using the visual language model, based on the refined prompt information and the target video frame, the start and end times of each action are refined and located to obtain the third start time and the third end time of each action. The third start time and third end time of each action are checked for minimum duration, and the second start time and second end time of each action are determined based on the minimum duration check result.
5. The method according to claim 1 or 4, characterized in that, After refining the start and end times of each action by using a visual language model, based on the start and end rules corresponding to each action, the refined search range of each action, and refined prompt word information, to obtain the second start time and the second end time of each action, the method further includes: The second start time and the second end time of each action are subjected to time overlap resolution processing.
6. The method according to claim 5, characterized in that, The time overlap resolution process for the second start time and second end time of each action includes at least one of the following: In response to a time overlap between the second start time of a later action and the second end time of a earlier action between adjacent action pairs, if the duration of the later action is greater than the duration of the earlier action, the second start time of the later action is adjusted; if the duration of the later action is less than the duration of the earlier action, the second end time of the earlier action is adjusted. For each action with a preset semantic type, the second start time is adjusted to the second end time of the previous action; In response to the fact that the later action and the earlier action are consecutive actions between adjacent action pairs, the second end time of the earlier action is adjusted to the second start time of the later action.
7. The method according to claim 5, characterized in that, After performing time overlap resolution processing on the second start time and second end time of each action, the method further includes: Based on the second start time and second end time of each action after time overlap resolution processing, the video frame sequence corresponding to each action is extracted from the target video. Based on the video frame sequence corresponding to each action, the description information of each action, and the description information of adjacent actions of each action, the consistency of visual content and description information is checked to obtain the consistency check result of each action.
8. The method according to claim 7, characterized in that, After performing a consistency check on the visual content and description information based on the video frame sequence corresponding to each action, the description information of each action, and the description information of adjacent actions of each action, and obtaining the consistency check result of each action, the method further includes: Provide prompts for actions whose visual content and descriptive information are inconsistent in the consistency verification results.
9. An electronic device, characterized in that, include: Memory, used to store computer program products; A processor for executing a computer program product stored in the memory, wherein when the computer program product is executed, it implements the method described in any one of claims 1-8.
10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-8.