Key event extraction and positioning processing feedback system and method

Abstract

The invention belongs to the technical field of video intelligent analysis, and more specifically relates to a key event extraction and positioning processing feedback system and method, which can match a proper video analysis algorithm model and a software tool according to the application scene type and the target key event type of each video file to be processed in a task data packet. And then, in combination with real-time computing power information, configuring a software tool and an operation platform with minimum computing power waste for each to-be-processed video file matched with the video analysis algorithm model and the software tool. Therefore, for diversified scenes, a task execution strategy is automatically generated through the task planning model, autonomous intelligent scheduling of an algorithm model, a software tool and an operation platform is realized, and a structured task processing result and a target key video segment of each to-be-processed video file are accurately obtained from each to-be-processed video file. The accuracy of key event extraction is improved, and meanwhile the overall operation efficiency of the system is improved.

Description

Technical Field

[0001] This application belongs to the field of video intelligent analysis technology, and more specifically, relates to a key event extraction, location processing and feedback system and method. Background Technology

[0002] The rapid development of sensor technology and the decline in hardware costs have led to an explosive growth in the amount of surveillance video data. Video surveillance systems for large-scale facilities such as smart agriculture, smart factories, and emergency command centers have shifted from simple monitoring to the storage, management, and analysis of video data. Therefore, video surveillance systems are required to provide information extraction, alarm, and traceability support for key event-related data from massive amounts of video data.

[0003] However, most existing methods for key event extraction and localization rely on feature matching algorithms to match preset key event reference videos with the video to be processed, thereby locating the video data corresponding to the key event within the video to be processed. These methods largely focus on calling video processing and analysis algorithms in a single domain, lacking adaptability and scalability to different application scenarios. They also lack intelligent autonomy in algorithm scheduling for diverse scenarios, requiring significant human intervention.

[0004] Furthermore, due to the massive amount of video data in each large-scale video surveillance system, multiple computing platforms are required to process the video data in a multi-threaded manner. However, different video processing and analysis algorithms have different computational power consumption. Existing key event extraction and location methods lack intelligent scheduling of task allocation for computing power platforms, resulting in problems such as wasted computing power or untimely user response. Summary of the Invention

[0005] To address the aforementioned deficiencies in existing technologies, this application provides a critical event extraction and localization processing feedback system and method, aiming to solve the problems of poor scenario adaptability, insufficient utilization of computing power, and low retrieval and localization efficiency in critical event extraction and localization in video surveillance systems.

[0006] Firstly, this application provides a method for extracting, locating, processing, and providing feedback on critical events, including: S1. Obtain user input commands and determine the type of input command. The types of input commands include key event extraction and key event retrieval. S2. When the type of input instruction is key event extraction, the user interaction information corresponding to the input instruction is encapsulated into a task data package. The task data package includes the target key event type, the address of at least one video file to be processed, and the application scenario type corresponding to the video file to be processed, and the available information set of the task is obtained. S3. Based on the task data package and the available task information set, perform key event extraction processing on each video file to be processed to obtain the target key video segments that meet the target key event type in each video file to be processed, as well as the structured task processing results corresponding to the target key video segments, and store each target key video segment to disk and each structured task processing result to the database. S4. When the type of input instruction is critical event retrieval, obtain the user interaction information corresponding to the input instruction, and perform critical event retrieval processing on the database based on the user interaction information.

[0007] Furthermore, based on the task data package and the available task information set, key event extraction processing is performed on each video file to be processed, including: S31. Extract the key events that match the target key event type from all the video files to be processed, and the target key video segments corresponding to the target key events as the target task. S32. Based on the task data package, the available task information set, and the preset prompt word template, the target task is planned through the task planning model to obtain the task execution strategy, and the target task is executed based on the task execution strategy.

[0008] Based on the task data package, the available task information set, and the preset prompt word template, the target task is planned using a task planning model to obtain the task execution strategy, including: Input the task data package and the available task information set into the task planning model, and constrain the output of the task planning model based on the preset prompt word template to obtain the task execution strategy corresponding to the target task.

[0009] The output of the task planning model is constrained based on a preset prompt word template, including: Constraints are set based on preset prompt word templates, and the task planning model is constrained based on these constraints. The constraints include the configuration requirements of each computing platform, software tool, and video analysis algorithm model.

[0010] The configuration requirements corresponding to the preset prompt word template can be used to constrain the task execution strategy generated by the task planning model, aiming to improve the accuracy of key event extraction and the overall system operating efficiency. This application generates task execution strategies corresponding to target tasks through a task planning model. It can match appropriate video analysis algorithm models based on the application scenario type of each video file to be processed within the task data package, and, combined with real-time computing power information, configure software tools and computing platforms with minimal computing power waste for each video file to be processed that has been matched with the video analysis algorithm model.

[0011] Furthermore, the task execution strategy is a sequence of sub-tasks consisting of multiple sub-steps. Each sub-step is a sub-task, and the attributes of the sub-task include the belonging task ID, the sub-task step number, the sub-task status, the type of software tool used, the type of video analysis algorithm model, and the number of the selected computing platform.

[0012] Furthermore, key event extraction processing is performed on each video file to be processed, resulting in target key video segments within each video file that conform to the target key event type, and the structured task processing results corresponding to each target key video segment, including: Obtain the subtask execution sequence of the task execution strategy, and package each subtask corresponding to the subtask execution sequence into a subtask package. The subtask package fields include the attributes of the subtasks. Based on the computing platform, software tools and video analysis algorithm model corresponding to the subtask package, key events are extracted from each video file to be processed to obtain the subtask processing results and key video segments of the subtask corresponding to the subtask package. The subtask processing results and key video segments corresponding to the subtask package are integrated to form the structured task processing results and target key video segments for each video file to be processed.

[0013] Furthermore, the structured task processing results include: the event type, event time, event subject, event ID, ID of the video to be processed corresponding to the target key event, video address, timing information of the target key event within the video to be processed, key frame address, and key frame sequence address. The timing information of the target key event includes the duration of the event within the video to be processed, the start time series timestamp, and the end time series timestamp.

[0014] Furthermore, based on user interaction information, key event retrieval processing is performed on the database, including: S41. Parse the user interaction information into key event retrieval instructions. The key event retrieval instructions include the event type, event subject, and event time range of the key event to be retrieved. S42. Based on the key event retrieval command, retrieve the results of each structured task in the database to obtain the structured task processing results corresponding to the key event retrieval command. S43. Obtain the target key video segment based on the event key frame address and the event key frame sequence address in the structured task processing results.

[0015] Secondly, this application also provides a critical event extraction, location processing, and feedback system for implementing any of the methods in the first aspect, including: The instruction acquisition module is used to acquire user input instructions; The task data acquisition module is used to encapsulate the user interaction information corresponding to the input command into a task data package when the type of input command is key event extraction; The execution strategy acquisition module is used to plan the target task based on the task data package, the acquired task availability information set and the preset prompt word template, and obtain the task execution strategy through the task planning model. The task scheduling module is used to obtain the subtask execution sequence of the task execution strategy and package each subtask corresponding to the subtask execution sequence into a subtask package; The key event extraction module is used to extract key events from each video file to be processed. On the computing platform corresponding to each subtask package, it calls the software tools and video analysis algorithm models corresponding to the subtask to extract key events from the video file to be processed, and obtains the subtask processing results and key video segments of each subtask package. The key event integration module integrates the subtask processing results and key video segments of the subtask corresponding to the subtask package to form the structured task processing results and target key video segments of each video file to be processed. The event retrieval module is used to obtain user interaction information corresponding to the input command when the input command type is key event retrieval, and to perform key event retrieval processing on the database based on the user interaction information. Overall, compared with the prior art, the above technical solution conceived by this invention has the following beneficial effects: 1. The key event extraction and location processing feedback system and method provided in this application, when obtaining the input command for key event extraction, generates a task execution strategy corresponding to the target task through a task planning model. It can match suitable video analysis algorithm models and software tools according to the application scenario type and target key event type of each video file to be processed in the task data package, and, combined with real-time computing power information, configure the software tools and computing platform with minimal computing power waste for each video file to be processed that has been matched with the video analysis algorithm model and software tools. Thus, it can automatically generate task execution strategies through the task planning model for diverse scenarios, realize the autonomous intelligent scheduling of algorithm models, software tools, and computing platforms, accurately obtain the structured task processing results and target key video segments of each video file to be processed, improve the accuracy of key event extraction, and enhance the overall operating efficiency of the system.

[0016] 2. By storing the pre-acquired structured task processing results and target key video segments corresponding to various types of key events in the database and on the disk respectively, and associating the key frame or key frame sequence address field in the structured task processing results with the target key video segments, the structured task processing results in the database can be retrieved by using the user interaction information corresponding to the input command when a key event retrieval input command is obtained. This allows for accurate and rapid location and retrieval of the corresponding structured task processing results and target key video segments, thereby improving the efficiency of key event-based retrieval and location. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the key event extraction, location processing, and feedback method provided in the embodiments of this application.

[0019] Figure 2 This is a flowchart illustrating the steps for retrieving and processing key events in a database, as provided in an embodiment of this application.

[0020] Figure 3 This is a schematic diagram of the key event extraction and location processing feedback system provided in the embodiments of this application. Detailed Implementation

[0021] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0022] In the following description, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The following description provides multiple embodiments of this application, which can be substituted or combined with each other. Therefore, this application can also be considered to include all possible combinations of the same and / or different embodiments described. Thus, if one embodiment includes features A, B, and C, and another embodiment includes features B and D, then this application should also be considered to include embodiments containing one or more other possible combinations of A, B, C, and D, even if such embodiments are not explicitly described in the following text.

[0023] The following description provides examples and does not limit the scope, applicability, or examples set forth in the claims. Changes may be made to the function and arrangement of the described elements without departing from the scope of this application. Various processes or components may be appropriately omitted, substituted, or added to the examples. Furthermore, features described with respect to some examples may be combined into other examples.

[0024] Figure 1 This is a flowchart illustrating the key event extraction and location processing feedback method provided in the embodiments of this application, such as... Figure 1 As shown, the method includes at least the following steps: S1. Obtain user input commands and determine the type of input command. The types of input commands include key event extraction and key event retrieval.

[0025] In this embodiment, the execution entity of the method can be the central processing unit of a video surveillance system for large facilities. Input commands can be entered via buttons, menus, etc., thus eliminating the need for users to input data according to fixed templates and text formats, thereby improving the convenience of the method in this embodiment.

[0026] S2. When the type of input instruction is key event extraction, the user interaction information corresponding to the input instruction is encapsulated into a task data package. The task data package includes the target key event type, the address of at least one video file to be processed, and the application scenario type corresponding to the video file to be processed, and the available information set of the task is obtained.

[0027] In this embodiment, the user determines the target key event type, the address of the video file to be processed, and the application scenario type corresponding to the video file to be processed by selecting through buttons, menus, dialog boxes, etc. When encapsulating the task data package, the target key event type and the application scenario type can be distinguished by setting simple identifiers. The task data package itself does not include the video file to be processed, and it needs to be retrieved according to the storage address information of the video file to be processed included in the task data package.

[0028] S3. Based on the task data package and the available task information set, perform key event extraction processing on each video file to be processed to obtain the target key video segments that meet the target key event type in each video file to be processed, as well as the structured task processing results corresponding to the target key video segments. Then, store each target key video segment to disk and each structured task processing result to the database.

[0029] In one possible implementation, the available task information set includes platform information to be configured and real-time computing power information. The platform information to be configured includes the number of each computing platform, computing power baseline information, available software tools and video analysis algorithm models, computing power requirements for running each video analysis algorithm model and each software tool, and real-time computing power information includes the real-time available computing power of each computing platform.

[0030] In this embodiment, the software tools are usable application tools or pre-packaged functions or software that can be directly executed. The computing power baseline information of each computing platform, the available software tools and video analysis algorithm models on each computing platform, and the computing power requirements of the software tools and algorithm models can be stored as configuration files at fixed addresses on each computing platform. Relevant information can be obtained by reading the configuration files of each computing platform. The platform's computing power baseline information may include, but is not limited to, platform computing power values ​​and total memory. The computing power requirements of the software tools and algorithm models may include, but are not limited to, a list of deployable platform types, GPU / NPU utilization, and memory usage. The current computing power usage of the computing platform can be continuously collected by enabling resource monitoring services, which may include, but is not limited to, GPU / NPU utilization, total CPU utilization, and available memory.

[0031] In one possible implementation, based on the task data packet and the task available information set, key event extraction processing is performed on each video file to be processed, including: S31. Extract the key events that match the target key event type from all the video files to be processed, and the target key video segments corresponding to the target key events as the target task. S32. Based on the task data package, the available task information set, and the preset prompt word template, the target task is planned through the task planning model to obtain the task execution strategy, and the target task is executed based on the task execution strategy.

[0032] In this embodiment of the application, step S32 includes: inputting the task data packet and the available task information set into the task planning model, and constraining the output of the task planning model based on the preset prompt word template to obtain the task execution strategy corresponding to the target task.

[0033] The task execution strategy is a sequence of subtasks consisting of multiple sub-steps. Each step is a subtask, and its attributes include the task ID, subtask step number, subtask status, type of tool or algorithm model invoked, and platform device number. Since the task execution strategy of this embodiment is implemented through multiple computing platforms and the software tools and video analysis algorithm models configured on them, and given the varying adaptability of the computing platforms' built-in systems to different software tools and video analysis algorithm models, and the difficulty in matching the real-time computing power of each platform with the real-time computing power consumption requirements of each video processing algorithm model, a preset prompt word template is needed to input these configuration requirements into the task planning model. This generates a stable task execution strategy under multiple scenario and multi-platform configurations. The preset prompt word template can also preset other constraints to adjust the generated results of the task planning model.

[0034] For example, the prompt template could be: "Please generate a task execution strategy to extract key events from the video file to be processed." When generating the task execution strategy, select the video analysis algorithm model type and tool type corresponding to the target key events for this type of scenario. When determining the specific algorithm model, prioritize the video analysis algorithm model with the highest accuracy in the application environment corresponding to the task data package. For each type of video analysis algorithm model, prioritize the computing platform that has already deployed that type of algorithm model, has the lowest real-time load, and has sufficient available computing power for execution.

[0035] In this embodiment, each computing platform can be distinguished by a different platform ID number, the video files to be processed can be distinguished by different video IDs, the algorithm models can be distinguished by model names, and the software tools can be distinguished by software names and version numbers.

[0036] For ease of understanding, the steps of the subtask execution sequence for extracting key events related to "regional intrusion" in a certain video, as described in the task execution strategy, can be as follows: ① The task of extracting the key events of "regional intrusion" from the video was assigned to the computing platform with ID=1; ② Run the "Get Video Frame Sequence" function to obtain the sequence of frames; ③ Run the YOLO v8 object detection model frame by frame to obtain object detection results; ④ Run the "Region Intersection Judgment" function to determine whether there is an intersection between the target area and the warning area; ⑤ Store the key frame times, corresponding target detection results, and key frames in the corresponding regions where there is overlap.

[0037] In addition, the task planning model is obtained by fine-tuning the existing large-scale model based on the constructed dataset. The dataset includes application scenario types, task instructions, real-time computing power information of the computing platform, information on available algorithm model sets, information on available tool sets, and execution strategies corresponding to the above data. It may also include synthetic data of the large model that has been manually verified.

[0038] In this embodiment of the application, the steps for data training and parameter fine-tuning of the task planning model can be as follows: Constructing a seed dataset: A small batch of high-quality data samples are manually compiled by domain experts (such as algorithm engineers and system architects). The samples cover various typical application scenarios, task instruction types, computing platform computing power, algorithm combinations and tool combinations to ensure diversity and accuracy, and serve as seed samples. Large model generates contextual information synthetic data: Select a base large model, take seed data as input, and use "generate similar data" as the prompt word to generate synthetic data containing different contextual information in batches; Contextual information dataset processing: The generated synthetic data undergoes manual review and correction. The criteria for manual review include, but are not limited to, whether the samples contain diversity and accuracy. A portion of the seed data and the verified and corrected synthetic data is selected as the test set. Several data samples are selected from the remaining data to design execution strategies as example samples, and the remaining data is used as the contextual dataset. Large model generates synthetic data for execution strategies: Select a base large model, take the context dataset as input, use the preset prompt word template in step S32 and the example samples as examples, and generate synthetic data for execution strategies in batches through context learning; Manual verification and correction: The synthetic data of the execution strategy is subject to strict manual review and correction. The criteria for manual review include, but are not limited to: whether the generated sub-task sequence is reasonable and feasible, whether it matches the resources described by the computing power of the computing platform, algorithm combination, and tool combination in the context dataset, and correcting any erroneous sub-task sequences. Training data preprocessing and formatting: Convert the manually verified and corrected data (contextual data + execution strategy data) into the format required for model training, concatenate all information except the subtask sequence into a text as input, and use the subtask sequence as the target output that the model needs to learn and generate; Select a base model for fine-tuning: Select a base model or a task planning model, choose a fine-tuning method, and use preprocessed training data to fine-tune the model to form a mapping from task objectives and resource status information to "sub-task sequences". Model Iterative Optimization: Set a set of key evaluation indicators for iterative optimization, evaluate the fine-tuned model on the test set, calculate the pass rate of the evaluation indicators, treat data that do not exceed the pass rate threshold as failure cases, and use the failure cases as new data samples. Repeat the fine-tuning steps and perform multiple rounds of iterative training until the model is stable and forms a task planning model that can be used for inference.

[0039] The key performance indicators for iterative optimization include, but are not limited to: the logical correctness of the subtask sequence, the rationality of the platform device number selected by the model for each subtask, and the total execution time of the subtask sequence.

[0040] The evaluation indicators for the rationality of the running platform device number include, but are not limited to: Platform compatibility: Whether the selected platform is in the list of supported models / tools.

[0041] Load awareness: When multiple supported platforms are available, is there a preference for choosing the platform with lower computational load?

[0042] Load avoidance: Whether tasks are avoided from being assigned to platforms with extremely high loads (unless there is no other choice).

[0043] Final efficiency: Whether the overall scheduling decision is reflected in the total execution time of the subtask sequence (which has advantages over random or fixed allocation).

[0044] In one possible implementation, key event extraction processing is performed on each video file to be processed to obtain target key video segments within each video file that conform to the target key event type, and the structured task processing results corresponding to the target key video segments, including: Obtain the subtask execution sequence of the task execution strategy, and package each subtask corresponding to the subtask execution sequence into a subtask package. The subtask package fields include the attributes of the subtasks. Based on the computing platform, software tools and video analysis algorithm model corresponding to the subtask package, key events are extracted from each video file to be processed to obtain the subtask processing results and key video segments of the subtask corresponding to the subtask package. The subtask processing results and key video segments corresponding to the subtask package are integrated to form the structured task processing results and target key video segments for each video file to be processed.

[0045] In this embodiment, the task execution strategy is a sequence of subtasks consisting of multiple sub-steps. Each step is a subtask, and the subtask attributes include the subordinate task ID, subtask step number, subtask status, type of tool or algorithm model called, and platform device number.

[0046] In this embodiment of the application, the structured task processing result includes: the event type, event time, event subject, event ID, ID of the video to be processed corresponding to the target key event, video address, timing information of the target key event in the video to be processed, event key frame address, and address of the event key frame sequence. The timing information of the target key event includes the duration of the event in the video to be processed, the start time series timestamp, and the end time series timestamp.

[0047] S4. When the type of input instruction is critical event retrieval, obtain the user interaction information corresponding to the input instruction, and perform critical event retrieval processing on the database based on the user interaction information.

[0048] In one possible implementation, key event retrieval processing is performed on the database based on user interaction information, including: S41. Parse the user interaction information into key event retrieval instructions. The key event retrieval instructions include the event type, event subject, and event time range of the key event to be retrieved. S42. Based on the key event retrieval command, retrieve the results of each structured task in the database to obtain the structured task processing results corresponding to the key event retrieval command. S43. Obtain the target key video segment based on the event key frame address and the event key frame sequence address in the structured task processing results.

[0049] In the embodiments of this application, such as Figure 2 As shown, user interaction information can be in the form of natural language. For example, if the user interaction information is "check the area intrusions in the last week", then the corresponding key event retrieval instruction can be "Event type: area intrusion; event time period: one week before the current time".

[0050] This application embodiment narrows the target retrieval scope to key fields retrieved from the video structured task processing results in the disk database. It can quickly locate the target key video segment and original video that meet the retrieval requirements from fields such as "event key frame sequence address" and "original video segment address" in the video structured task processing results that match the key event retrieval fields. This satisfies the user's need to quickly locate and play back key video segments or the original video where key events are located, and effectively improves the speed of retrieval and location of key events.

[0051] Figure 3 This is a schematic diagram of the structure of the key event extraction and location processing feedback system provided in the embodiments of this application, as shown below. Figure 3 As shown, the system includes at least: The instruction acquisition module is used to acquire user input instructions; The task data acquisition module is used to encapsulate the user interaction information corresponding to the input command into a task data package when the type of input command is key event extraction; The execution strategy acquisition module is used to plan the target task based on the task data package, the acquired task availability information set and the preset prompt word template, and obtain the task execution strategy through the task planning model. The task scheduling module is used to obtain the subtask execution sequence of the task execution strategy and package each subtask corresponding to the subtask execution sequence into a subtask package; The key event extraction module is used to extract key events from each video file to be processed. On the computing platform corresponding to each subtask package, it calls the software tools and video analysis algorithm models corresponding to the subtask to extract key events from the video file to be processed, and obtains the subtask processing results and key video segments of each subtask package. The key event integration module integrates the subtask processing results and key video segments of the subtask corresponding to the subtask package to form the structured task processing results and target key video segments of each video file to be processed. The event retrieval module is used to obtain the user interaction information corresponding to the input command when the type of input command is critical event retrieval, and to perform critical event retrieval processing on the database based on the user interaction information.

[0052] In one possible implementation, the critical event extraction and location processing feedback system operates as follows when a user performs critical event extraction: The instruction acquisition module acquires user input instructions and determines the type of the input instruction as a key event extraction. The task data acquisition module encapsulates the user interaction information (including but not limited to information interacted through buttons, menus, dialog boxes, etc.) corresponding to the input instructions into a task data package; The execution strategy acquisition module, based on the task data package, the acquired task availability information set, and the preset prompt word template, plans the target task through a fine-tuned task planning model to obtain the task execution strategy; The task scheduling module generates subtask packages based on the task execution strategy and publishes the subtasks. The "subtask status" field of the subtask package is initialized to "pending processing".

[0053] When the key event extraction module deployed on the computing platform with platform ID=k reads that the "Running Platform ID" field of the subtask package is also ID=k, it further reads other field information of the subtask package. For subtask packages with the same subordinate task ID, it processes them according to the order of the subtask step number. Based on the type of tool or algorithm model read, it calls the matching tool or video analysis algorithm model to perform video processing and analysis, forming the subtask processing result and the subtask key video segment, and updates the status field information of the subtask package to "processing completed". The critical event integration module processes the results of subtasks with the same task ID. Based on the order of the subtask step numbers, the processing results of each subtask are integrated to form the structured task processing results and target key video segments corresponding to the critical events. The structured task processing results are stored in a disk database, and keyframes or key video segments are stored on disk. The event keyframe address and event keyframe sequence address fields in the structured task processing results are updated according to the storage addresses of the keyframes or key video segments.

[0054] The key event extraction module runs on various computing platforms. Due to differences in computing resources, the video analysis algorithm model sets deployed on different computing platforms vary. The remaining modules run on the main control server. In the above embodiments, it can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line DSL) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., SSDs (SolID State Disks)).

[0055] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0056] Those skilled in the art will readily understand that the above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for extracting, locating, and processing feedback of key events, characterized in that, include: S1. Obtain user input instructions and determine the type of the input instructions, wherein the type of the input instructions includes key event extraction and key event retrieval; S2. When the type of the input instruction is key event extraction, the user interaction information corresponding to the input instruction is encapsulated into a task data package. The task data package includes the target key event type, the address of at least one video file to be processed, and the application scenario type corresponding to the video file to be processed, and the task available information set is obtained. S3. Based on the task data package and the task available information set, perform key event extraction processing on each of the video files to be processed to obtain target key video segments that conform to the target key event type in each of the video files to be processed, as well as the structured task processing results corresponding to the target key video segments, and store each of the target key video segments to the disk and each of the structured task processing results to the database. S4. When the type of the input instruction is key event retrieval, obtain the user interaction information corresponding to the input instruction, and perform key event retrieval processing on the database based on the user interaction information.

2. The key event extraction, location processing, and feedback method according to claim 1, characterized in that, The step of extracting key events from each of the video files to be processed based on the task data packet and the available task information set includes: S31. Extract all key events that conform to the target key event type from all the video files to be processed, and the target key video segments corresponding to the target key events as the target task; S32. Based on the task data package, the available task information set, and the preset prompt word template, the target task is planned using a task planning model to obtain a task execution strategy, and the target task is executed based on the task execution strategy.

3. The key event extraction, location processing, and feedback method according to claim 2, characterized in that, The available information set for the task includes information on the platform to be configured and real-time computing power information. The information on the platform to be configured includes the number of each computing platform, computing power baseline information, available software tools and video analysis algorithm models, the computing power requirements for running each video analysis algorithm model and each software tool, and the real-time computing power information includes the real-time available computing power of each computing platform.

4. The key event extraction, location processing, and feedback method according to claim 3, characterized in that, The step of planning the target task based on the task data package, the available task information set, and the preset prompt word template, through a task planning model, to obtain a task execution strategy includes: The task data package and the available task information set are input into the task planning model, and the output of the task planning model is constrained based on the preset prompt word template to obtain the task execution strategy corresponding to the target task.

5. The key event extraction, location processing, and feedback method according to claim 4, characterized in that, The constraint on the output of the task planning model based on the preset prompt word template includes: Constraints are set based on the preset prompt word template, and the task planning model is constrained based on the constraints. The constraints include the configuration requirements of each computing platform, each software tool, and each video analysis algorithm model.

6. The key event extraction, location processing, and feedback method according to claim 5, characterized in that, The task execution strategy is a sequence of sub-tasks consisting of multiple sub-steps, each sub-step being a sub-task. The attributes of the sub-task include the belonging task ID, the sub-task step number, the sub-task status, the type of the selected software tool, the type of the video analysis algorithm model, and the number of the selected computing platform.

7. The critical event extraction, location processing, and feedback method according to claim 6, characterized in that, The step of extracting key events from each of the video files to be processed, to obtain target key video segments within each video file that conform to the target key event type, and the structured task processing results corresponding to the target key video segments, includes: Obtain the subtask execution sequence of the task execution strategy, and package each subtask corresponding to the subtask execution sequence into a subtask package, wherein the subtask package fields include the attributes of the subtask; Based on the computing platform, software tools and video analysis algorithm model corresponding to the subtask package, key event extraction processing is performed on each of the video files to be processed to obtain the subtask processing results and key video segments of the subtask corresponding to the subtask package. The processing results of the subtasks corresponding to the subtask packages and the key video segments of the subtasks are integrated to form structured task processing results and target key video segments for each video file to be processed.

8. The key event extraction, location processing, and feedback method according to claim 7, characterized in that, The structured task processing result includes: the event type, event time, event subject, event ID, and the ID of the video to be processed corresponding to the target key event in the task data packet; the video address; the timing information of the target key event in the video to be processed; the address of the event key frame; and the address of the event key frame sequence. The timing information of the target key event includes the duration of the event in the video to be processed, the start time series timestamp, and the end time series timestamp.

9. The key event extraction, location processing, and feedback method according to claim 8, characterized in that, The key event retrieval processing based on the user interaction information in the database includes: S41. Parse the user interaction information into a key event retrieval instruction, wherein the key event retrieval instruction includes the event type, event subject, and event time segment of the key event to be retrieved; S42. Based on the key event retrieval instruction, retrieve the structured task processing results in the database to obtain the structured task processing results corresponding to the key event retrieval instruction; S43. Based on the event keyframe address and the event keyframe sequence address in the structured task processing result, the target key video segment is obtained.

10. A critical event extraction, location processing, and feedback system, used to implement the method described in any one of claims 1-9, characterized in that, include: The instruction acquisition module is used to acquire user input instructions; The task data acquisition module is used to encapsulate the user interaction information corresponding to the input instruction into a task data package when the type of the input instruction is key event extraction; The execution strategy acquisition module is used to plan the target task based on the task data package, the acquired task availability information set and the preset prompt word template, and to obtain the task execution strategy by using a task planning model. The task scheduling module is used to obtain the subtask execution sequence of the task execution strategy and package each subtask corresponding to the subtask execution sequence into a subtask package; The key event extraction module is used to extract key events from each of the video files to be processed. On the computing platform corresponding to each subtask package, the software tools and video analysis algorithm model corresponding to the subtask are called to extract key events from the video files to be processed, and the subtask processing results and key video segments of each subtask package are obtained. The key event integration module integrates the subtask processing results and key video segments of the subtask corresponding to the subtask package to form structured task processing results and target key video segments for each video file to be processed. The event retrieval module is used to obtain the user interaction information corresponding to the input instruction when the type of the input instruction is a key event retrieval, and to perform key event retrieval processing on the database based on the user interaction information.

Images