Method and system for positioning specific event in long video

Abstract

The invention is suitable for the technical field of video processing, and provides a method and a system for positioning a specific event in a long video. According to the method, the lightweight model is constructed, and local deployment is carried out; when the lightweight model can be directly used, a local positioning result is directly obtained; and when the lightweight model cannot be directly used, performing auxiliary reasoning by using the multi-modal large model to obtain an auxiliary positioning result, and performing feedback upgrading on the lightweight model. A local positioning result can be directly obtained when a lightweight model can be directly used; when a lightweight model cannot be directly used, a far-end multi-mode large model is used for auxiliary reasoning, and the lightweight model is subjected to feedback upgrading, so that the lightweight model and the multi-mode large model are effectively and cooperatively fused, and the positioning capability of a specific event in a long video is improved while the positioning efficiency of the specific event in the long video is improved. And the self-upgrading of the locally deployed lightweight model can be realized by utilizing the reasoning result of the multi-modal large model, so that the effect that the lightweight model is better and better to use is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of video processing technology, and in particular relates to a method and system for locating specific events in long videos. Background Art

[0002] To locate key events in long videos or streaming media, one or more lightweight models deployed on local terminal devices are usually required to process the input long videos. Training lightweight models deployed on local terminal devices usually requires a large amount of high-quality supervised data, which involves a lot of manual calibration costs. When key events are low-frequency events, it is even more difficult to build a high-quality, widely distributed training set.

[0003] Compared with lightweight models, large multimodal models have powerful scene understanding, reasoning, and contextual learning capabilities, and have better generalization performance. However, their high deployment and fine-tuning costs affect their large-scale deployment on local terminal devices. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a method and system for locating specific events in a long video, aiming to solve the technical problems existing in the existing technology mentioned in the background technology.

[0005] The embodiment of the present invention is implemented as follows:

[0006] A method for locating a specific event in a long video, the method specifically comprising the following steps:

[0007] Obtain a basic training set, perform model training, build a lightweight model, and deploy it locally on the terminal device, and build a memory pool based on the basic training set;

[0008] Receive input video data, segment the video segments, compare the similarities of the video segments through the memory pool, and determine whether the lightweight model can be used directly;

[0009] When a lightweight model can be used directly, the local positioning result is obtained directly;

[0010] When the lightweight model cannot be used directly, a remote multimodal large model is used for auxiliary reasoning to obtain auxiliary positioning results and to feed back and upgrade the lightweight model;

[0011] Visually display local positioning results or auxiliary positioning results.

[0012] As a further limitation of the technical solution of the embodiment of the present invention, obtaining a basic training set, performing model training, building a lightweight model, and locally deploying it on a terminal device, and building a memory pool based on the basic training set specifically include the following steps:

[0013] Obtain the basic training set, perform model training, and build a lightweight model;

[0014] Locally deploying the lightweight model on a terminal device;

[0015] Performing clustering processing on the basic training set to obtain multiple cluster information;

[0016] A memory pool is constructed based on the plurality of cluster information.

[0017] As a further limitation of the technical solution of the embodiment of the present invention, the receiving of input video data, segmenting the video segments, performing similarity comparison on the video segments through the memory pool, and determining whether the lightweight model can be directly used specifically include the following steps:

[0018] receiving input video data;

[0019] Segmenting the video data to obtain video segments;

[0020] Performing similarity comparison analysis on the video clip and the cluster centers in the memory pool to obtain similarity accuracy;

[0021] Based on a preset threshold, the result is compared with the similarity accuracy to determine whether the lightweight model can be used directly.

[0022] As a further limitation of the technical solution of the embodiment of the present invention, when the lightweight model cannot be used directly, the remote multimodal large model is used for auxiliary reasoning to obtain auxiliary positioning results, and the lightweight model is fed back and upgraded, which specifically includes the following steps:

[0023] When the lightweight model cannot be used directly, a prompt is dynamically constructed according to the video clip;

[0024] Inputting the prompt into a remote multimodal large model;

[0025] Use a remote multimodal large model for auxiliary reasoning to obtain auxiliary positioning results;

[0026] Get manual positioning results;

[0027] Feedback and upgrade the lightweight model.

[0028] As a further limitation of the technical solution of the embodiment of the present invention, the feedback upgrade of the lightweight model specifically includes the following steps:

[0029] Record multiple auxiliary positioning results and / or multiple manual positioning results;

[0030] Matching multiple target segments corresponding to multiple auxiliary positioning results and / or multiple manual positioning results;

[0031] Determining whether the plurality of auxiliary positioning results and / or the plurality of manual positioning results meet a preset update condition;

[0032] When the plurality of auxiliary positioning results and / or the plurality of manual positioning results meet the update condition, the plurality of target segments, the plurality of auxiliary positioning results and / or the plurality of manual positioning results are integrated to construct an optimized training set;

[0033] Starting the autonomous learning algorithm of the lightweight model, and fine-tuning and upgrading the lightweight model through the optimized training set;

[0034] Reconstruct the memory pool based on the optimized training set.

[0035] As a further limitation of the technical solution of the embodiment of the present invention, the visual display of the local positioning result or the auxiliary positioning result specifically includes the following steps:

[0036] Receive user's display request;

[0037] Creating a display interface according to the display request;

[0038] In the display interface, the local positioning result or the auxiliary positioning result is displayed.

[0039] A system for locating specific events in long videos, comprising a lightweight model construction and deployment module, a model selection and judgment module, a lightweight model use module, a multimodal large model use module, and a positioning result display module, wherein:

[0040] A lightweight model construction and deployment module is used to obtain a basic training set, perform model training, build a lightweight model, and deploy it locally on the terminal device, and build a memory pool based on the basic training set;

[0041] A model selection and judgment module is used to receive input video data, segment the video segments, compare the similarities of the video segments through the memory pool, and determine whether the lightweight model can be directly used;

[0042] The lightweight model usage module is used to directly obtain local positioning results when the lightweight model can be used directly;

[0043] The multimodal large model usage module is used to use the remote multimodal large model for auxiliary reasoning when the lightweight model cannot be used directly, obtain auxiliary positioning results, and feed back and upgrade the lightweight model;

[0044] The positioning result display module is used to visually display local positioning results or auxiliary positioning results.

[0045] As a further limitation of the technical solution of the embodiment of the present invention, the model selection judgment module specifically includes:

[0046] An input video receiving unit, configured to receive input video data;

[0047] A video segmentation unit, configured to segment the video data to obtain video segments;

[0048] A similarity comparison unit, configured to perform similarity comparison analysis on the video clip and the cluster centers in the memory pool to obtain similarity accuracy;

[0049] The model use judgment unit is used to compare the similarity accuracy with a preset threshold to determine whether the lightweight model can be used directly.

[0050] As a further limitation of the technical solution of the embodiment of the present invention, the multimodal large model utilization module specifically includes:

[0051] A prompt construction unit, used to dynamically construct a prompt according to the video clip when the lightweight model cannot be used directly;

[0052] A prompt input unit, used to input the prompt into a remote multimodal large model;

[0053] The auxiliary reasoning unit is used to perform auxiliary reasoning using a remote multimodal large model to obtain auxiliary positioning results;

[0054] A manual result acquisition unit, used for acquiring manual positioning results;

[0055] The feedback upgrade unit is used to perform feedback upgrade on the lightweight model.

[0056] As a further limitation of the technical solution of the embodiment of the present invention, the feedback upgrade unit specifically includes:

[0057] A result recording unit, used to record multiple auxiliary positioning results and / or multiple manual positioning results;

[0058] a segment matching unit, configured to match multiple target segments corresponding to multiple auxiliary positioning results and / or multiple manual positioning results;

[0059] An update condition judgment unit, configured to judge whether a plurality of auxiliary positioning results and / or a plurality of manual positioning results meet a preset update condition;

[0060] an optimized training set construction unit, configured to construct an optimized training set by integrating the plurality of target segments, the plurality of auxiliary positioning results and / or the plurality of manual positioning results when the plurality of auxiliary positioning results and / or the plurality of manual positioning results meet an update condition;

[0061] A model fine-tuning and upgrading unit, configured to start an autonomous learning algorithm of the lightweight model and fine-tune and upgrade the lightweight model using the optimized training set;

[0062] The memory pool reconstruction unit is used to reconstruct the memory pool according to the optimized training set.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] The embodiment of the present invention constructs a lightweight model and deploys it locally; when the lightweight model can be used directly, the local positioning result is directly obtained; when the lightweight model cannot be used directly, a multimodal large model is used for auxiliary reasoning to obtain the auxiliary positioning result, and the lightweight model is fed back and upgraded. When the lightweight model can be used directly, the local positioning result can be directly obtained; when the lightweight model cannot be used directly, a remote multimodal large model is used for auxiliary reasoning, and the lightweight model is fed back and upgraded, thereby effectively synergizing the lightweight model with the multimodal large model. While improving the positioning capability of specific events in long videos, the reasoning results of the multimodal large model can also be used to achieve self-upgrade of the locally deployed lightweight model, achieving an effect of getting better with use. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A flowchart of a method for locating a specific event in a long video provided by an embodiment of the present invention is shown;

[0066] Figure 2 A flowchart of constructing a lightweight model and a memory pool in a method provided by an embodiment of the present invention is shown;

[0067] Figure 3 A flowchart of determining whether a lightweight model can be used directly in a method provided by an embodiment of the present invention is shown;

[0068] Figure 4 A flowchart of the multimodal large model performing assisted reasoning in the method provided by an embodiment of the present invention is shown;

[0069] Figure 5 A flowchart of feedback upgrade of a lightweight model in the method provided by an embodiment of the present invention is shown;

[0070] Figure 6 A flowchart showing the visual display of positioning results in the method provided by an embodiment of the present invention is shown;

[0071] Figure 7 The following is an application architecture diagram of a system for locating specific events in long videos provided by an embodiment of the present invention;

[0072] Figure 8 It shows a structural block diagram of a model selection and judgment module in a system provided by an embodiment of the present invention;

[0073] Figure 9 The following is a structural block diagram of a multimodal large model using module in a system provided by an embodiment of the present invention;

[0074] Figure 10 The structure block diagram of the feedback upgrade unit in the system provided by the embodiment of the present invention is shown. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0076] Understandably, locating key events in long videos or streaming media typically requires the use of one or more lightweight models deployed locally on terminal devices to process the input long video. Training lightweight models deployed locally on terminal devices typically requires a large amount of high-quality supervised data, which involves significant manual calibration costs. When key events are low-frequency, building high-quality, widely distributed training sets is even more difficult. Compared to lightweight models, large multimodal models possess powerful scene understanding, reasoning, and contextual learning capabilities, and exhibit good generalization performance. However, their high deployment and fine-tuning costs hinder their large-scale deployment on local terminal devices.

[0077] To solve the above problems, an embodiment of the present invention discloses a method and system for locating specific events in a long video. The method and system obtain a basic training set, perform model training, build a lightweight model, and deploy it locally on a terminal device. A memory pool is constructed based on the basic training set. The input video data is received, video segments are segmented, and the video segments are compared for similarity through the memory pool to determine whether the lightweight model can be used directly. When the lightweight model can be used directly, the local positioning result is directly obtained. When the lightweight model cannot be used directly, a remote multimodal large model is used for auxiliary reasoning to obtain auxiliary positioning results, and the lightweight model is fed back and upgraded. The local positioning result or the auxiliary positioning result is visually displayed. When the lightweight model can be used directly, the local positioning results can be directly obtained; when the lightweight model cannot be used directly, the remote multimodal large model is used for auxiliary reasoning, and the lightweight model is fed back and upgraded, thereby effectively synergizing the lightweight model with the multimodal large model. While improving the positioning capability of specific events in long videos, the reasoning results of the multimodal large model can also be used to realize the self-upgrade of the locally deployed lightweight model, achieving the effect of getting better and better with use.

[0078] Specifically, Figure 1 A flowchart of a method for locating a specific event in a long video provided by an embodiment of the present invention is shown.

[0079] In a preferred embodiment of the present invention, a method for locating a specific event in a long video includes the following steps:

[0080] Step S101: obtain a basic training set, perform model training, build a lightweight model, and deploy it locally on the terminal device, and build a memory pool based on the basic training set.

[0081] In an embodiment of the present invention, a basic training set is obtained to perform model training, a lightweight model is constructed, and the lightweight model is locally deployed on a local terminal device. The basic training set is clustered to obtain multiple cluster information including cluster number, cluster center, accuracy, cluster type, etc., and then a memory pool is constructed based on the multiple cluster information.

[0082] It can be understood that a lightweight model is a model that can run on a local terminal, has small parameters, and runs fast.

[0083] It can be understood that the clusters of the memory pool are obtained by clustering the training set, and the cluster center is the cluster center in the clustering operation.

[0084] Specifically, Figure 2 A flowchart of constructing a lightweight model and a memory pool in the method provided by an embodiment of the present invention is shown.

[0085] In another preferred embodiment of the present invention, obtaining a basic training set, performing model training, building a lightweight model, and locally deploying it on a terminal device, and building a memory pool based on the basic training set specifically include the following steps:

[0086] Step S1011: Obtain a basic training set, perform model training, and build a lightweight model;

[0087] Step S1012: Locally deploy the lightweight model on the terminal device;

[0088] Step S1013: performing clustering processing on the basic training set to obtain multiple cluster information;

[0089] Step S1014: construct a memory pool based on the plurality of cluster information.

[0090] Furthermore, the method for locating a specific event in a long video further includes the following steps:

[0091] Step S102: receiving input video data, dividing the video segments, and performing similarity comparison on the video segments through the memory pool to determine whether the lightweight model can be used directly.

[0092] In an embodiment of the present invention, video data input by a user is received and segmented to divide a long video into multiple video segments. The video segments are then compared and analyzed with the cluster centers in the memory pool for similarity to obtain similarity accuracy. The similarity accuracy is compared with a preset threshold to determine whether a lightweight model can be used directly. Specifically, when the similarity accuracy is greater than the preset threshold, it is determined that the lightweight model can be used directly; when the similarity accuracy is not greater than the preset threshold, it is determined that the lightweight model cannot be used directly.

[0093] Specifically, Figure 3 A flowchart for determining whether a lightweight model can be used directly in the method provided by an embodiment of the present invention is shown.

[0094] In another preferred embodiment of the present invention, the steps of receiving input video data, dividing the video segments, comparing the video segments for similarity using a memory pool, and determining whether the lightweight model can be used directly include the following steps:

[0095] Step S1021: receiving input video data;

[0096] Step S1022: Segment the video data to obtain video segments;

[0097] Step S1023: performing a similarity comparison analysis between the video clip and the cluster centers in the memory pool to obtain a similarity accuracy;

[0098] Step S1024: Based on a preset threshold, the result is compared with the similarity accuracy to determine whether the lightweight model can be used directly.

[0099] Furthermore, the method for locating a specific event in a long video further includes the following steps:

[0100] Step S103: When the lightweight model can be used directly, the local positioning result is directly obtained.

[0101] In an embodiment of the present invention, when the lightweight model can be used directly, the video clip is directly imported into the lightweight model, and then the local positioning result derived from the lightweight model is obtained.

[0102] Step S104: When the lightweight model cannot be used directly, the remote multimodal large model is used for auxiliary reasoning to obtain auxiliary positioning results, and the lightweight model is fed back and upgraded.

[0103] In an embodiment of the present invention, when a lightweight model cannot be used directly, a prompt containing high-quality instance samples is dynamically constructed based on a video clip, and the prompt is then input into a remote multimodal large model. Auxiliary reasoning is performed using the remote multimodal large model to obtain auxiliary positioning results and manual positioning results. Multiple auxiliary positioning results and / or multiple manual positioning results are recorded, and multiple target clips corresponding to the multiple auxiliary positioning results and / or multiple manual positioning results are matched, and it is determined whether the multiple auxiliary positioning results and / or multiple manual positioning results meet the preset update conditions. If the multiple auxiliary positioning results and / or multiple manual positioning results meet the preset update conditions, the multiple target clips and / or multiple manual positioning results are integrated to construct an optimized training set, and then the autonomous learning algorithm of the lightweight model is started. The lightweight model is fine-tuned and upgraded through the optimized training set, and the memory pool is reconstructed based on the optimized training set.

[0104] It is understandable that high-quality instance samples include positive and negative examples, which are selected from the example samples within the cluster; multimodal large models are usually general models, which may lead to poor performance on specific events. By dynamically constructing prompts, it is helpful to improve the accuracy of multimodal large models.

[0105] It is understandable that reconstructing the memory pool mainly involves re-clustering the samples in the memory pool and recalculating the accuracy of each cluster.

[0106] It is understandable that when the lightweight model is fine-tuned and upgraded, samples within each cluster will be evenly selected, so that the lightweight model after fine-tuning and upgrading will not only perform well in a few clusters, but will show a balanced state.

[0107] It can be understood that the construction of the prompt is a dynamic construction process. If the video clip has not appeared before (that is, the threshold of a cluster in the memory pool is lower than the preset value), it will be directly sent to the multimodal large model; if the video clip has appeared before, an accuracy analysis will be performed. If the accuracy is low, several example samples will be selected in the cluster and sent to the multimodal large model together.

[0108] It is understandable that in the case of errors in the judgment of the multimodal large model, manual calibration can also be performed to obtain manual positioning results; manual positioning results can also be used in the fine-tuning and upgrading process of the lightweight model.

[0109] Specifically, Figure 4 A flowchart of auxiliary reasoning using a multimodal large model in the method provided by an embodiment of the present invention is shown.

[0110] Among them, in another preferred embodiment provided by the present invention, when the lightweight model cannot be used directly, the remote multimodal large model is used for auxiliary reasoning to obtain auxiliary positioning results, and the lightweight model is fed back and upgraded, which specifically includes the following steps:

[0111] Step S1041: When the lightweight model cannot be used directly, dynamically construct a prompt based on the video clip;

[0112] Step S1042: input the prompt to the remote multimodal large model;

[0113] Step S1043: Use the remote multimodal large model to perform auxiliary reasoning to obtain auxiliary positioning results;

[0114] Step S1044: Obtain manual positioning results;

[0115] Step S1045: Feedback and upgrade the lightweight model.

[0116] Specifically, Figure 5 A flow chart of feedback upgrade of a lightweight model in the method provided by an embodiment of the present invention is shown.

[0117] Among them, in another preferred embodiment provided by the present invention, the feedback upgrade of the lightweight model specifically includes the following steps:

[0118] Step S10451: Record multiple auxiliary positioning results and / or multiple manual positioning results;

[0119] Step S10452: Match multiple target segments corresponding to multiple auxiliary positioning results and / or multiple manual positioning results;

[0120] Step S10453: determining whether the plurality of auxiliary positioning results and / or the plurality of manual positioning results meet a preset update condition;

[0121] Step S10454: when the plurality of auxiliary positioning results and / or the plurality of manual positioning results meet the update condition, the plurality of target segments, the plurality of auxiliary positioning results and / or the plurality of manual positioning results are integrated to construct an optimized training set;

[0122] Step S10455: starting the autonomous learning algorithm of the lightweight model, and fine-tuning and upgrading the lightweight model through the optimized training set;

[0123] Step S10456: reconstruct the memory pool according to the optimized training set.

[0124] Furthermore, the method for locating a specific event in a long video further includes the following steps:

[0125] Step S105: Visually display the local positioning result or the auxiliary positioning result.

[0126] In the embodiment of the present invention, a display request from a user is received, and a display interface is created according to the display request, and then the local positioning result or the auxiliary positioning result is displayed in the display interface.

[0127] Specifically, Figure 6 A flowchart of visual display of positioning results in the method provided by an embodiment of the present invention is shown.

[0128] In another preferred embodiment of the present invention, the visual display of the local positioning result or the auxiliary positioning result specifically includes the following steps:

[0129] Step S1051: Receive a user's display request;

[0130] Step S1052: Create a display interface according to the display request;

[0131] Step S1053: Display the local positioning result or the auxiliary positioning result in the display interface.

[0132] Further, Figure 7 The following is a diagram showing the application architecture of a system for locating specific events in long videos provided by an embodiment of the present invention.

[0133] Specifically, in another preferred embodiment of the present invention, a system for locating a specific event in a long video includes:

[0134] The lightweight model construction and deployment module 101 is used to obtain a basic training set, perform model training, build a lightweight model, and deploy it locally on the terminal device, and build a memory pool based on the basic training set.

[0135] In an embodiment of the present invention, the lightweight model construction and deployment module 101 obtains a basic training set, performs model training, constructs a lightweight model, and locally deploys the lightweight model on a local terminal device. It also performs clustering processing on the basic training set to obtain multiple cluster information including cluster number, cluster center, accuracy, cluster type, etc., and then constructs a memory pool based on the multiple cluster information.

[0136] The model selection and judgment module 102 is used to receive input video data, segment the video segments, compare the similarities of the video segments through the memory pool, and determine whether the lightweight model can be directly used.

[0137] In an embodiment of the present invention, the model selection judgment module 102 receives video data input by the user, segments the video data, divides the long video into multiple video segments, and then performs similarity comparison analysis on the video segments and the cluster centers in the memory pool to obtain similarity accuracy, compares the similarity accuracy with a preset threshold, and determines whether the lightweight model can be used directly. Specifically, when the similarity accuracy is greater than the preset threshold, it is determined that the lightweight model can be used directly; when the similarity accuracy is not greater than the preset threshold, it is determined that the lightweight model cannot be used directly.

[0138] Specifically, Figure 8 FIG. 1 shows a structural block diagram of the model selection and judgment module 102 in the system provided by an embodiment of the present invention.

[0139] In another preferred embodiment of the present invention, the model selection and judgment module 102 specifically includes:

[0140] An input video receiving unit 1021 is configured to receive input video data;

[0141] The video segmentation unit 1022 is configured to segment the video data to obtain video segments;

[0142] A similarity comparison unit 1023 is configured to perform similarity comparison analysis on the video clip and the cluster centers in the memory pool to obtain similarity accuracy;

[0143] The model use judgment unit 1024 is used to compare the similarity accuracy with a preset threshold to judge whether the lightweight model can be used directly.

[0144] Furthermore, the system for locating specific events in the long video also includes:

[0145] The lightweight model using module 103 is used to directly obtain the local positioning result when the lightweight model can be directly used.

[0146] In the embodiment of the present invention, when the lightweight model can be used directly, the lightweight model using module 103 directly imports the video clip into the lightweight model, and then obtains the local positioning result derived from the lightweight model.

[0147] The multimodal large model using module 104 is used to use the remote multimodal large model for auxiliary reasoning when the lightweight model cannot be used directly, obtain auxiliary positioning results, and feed back and upgrade the lightweight model.

[0148] In an embodiment of the present invention, when the lightweight model cannot be used directly, the multimodal large model using module 104 dynamically constructs a prompt containing high-quality instance samples based on the video clip, and then inputs the prompt into the remote multimodal large model. By using the remote multimodal large model for auxiliary reasoning, auxiliary positioning results and manual positioning results are obtained. By recording multiple auxiliary positioning results and / or multiple manual positioning results, and matching multiple target segments corresponding to the multiple auxiliary positioning results and / or multiple manual positioning results, and judging whether the multiple auxiliary positioning results and / or multiple manual positioning results meet the preset update conditions, and then, when the multiple auxiliary positioning results and / or multiple manual positioning results meet the preset update conditions, the multiple target segments and / or multiple manual positioning results are integrated to construct an optimized training set, and then the autonomous learning algorithm of the lightweight model is started. The lightweight model is fine-tuned and upgraded through the optimized training set, and the memory pool is reconstructed based on the optimized training set.

[0149] Specifically, Figure 9 The structure block diagram of the multimodal large model using module 104 in the system provided by the embodiment of the present invention is shown.

[0150] In another preferred embodiment of the present invention, the multimodal large model using module 104 specifically includes:

[0151] A prompt construction unit 1041 is used to dynamically construct a prompt according to the video clip when the lightweight model cannot be used directly;

[0152] A prompt input unit 1042 is used to input the prompt to the remote multimodal large model;

[0153] The auxiliary reasoning unit 1043 is used to perform auxiliary reasoning using the remote multimodal large model to obtain auxiliary positioning results;

[0154] The manual result obtaining unit 1044 is used to obtain the manual positioning result;

[0155] The feedback upgrade unit 1045 is used to perform feedback upgrade on the lightweight model.

[0156] Specifically, Figure 10 FIG. 1 shows a structural block diagram of the feedback upgrade unit 1045 in the system provided by an embodiment of the present invention.

[0157] In another preferred embodiment of the present invention, the feedback upgrade unit 1045 specifically includes:

[0158] A result recording unit 10451 is used to record multiple auxiliary positioning results and / or multiple manual positioning results;

[0159] a segment matching unit 10452, configured to match multiple target segments corresponding to multiple auxiliary positioning results and / or multiple manual positioning results;

[0160] An update condition determination unit 10453 is configured to determine whether the plurality of auxiliary positioning results and / or the plurality of manual positioning results meet a preset update condition;

[0161] An optimized training set construction unit 10454 is configured to construct an optimized training set by integrating the target segments, the auxiliary positioning results, and / or the manual positioning results when the plurality of auxiliary positioning results and / or the manual positioning results meet an update condition;

[0162] The model fine-tuning and upgrading unit 10455 is used to start the autonomous learning algorithm of the lightweight model and fine-tune and upgrade the lightweight model through the optimized training set;

[0163] The memory pool reconstruction unit 10456 is used to reconstruct the memory pool according to the optimized training set.

[0164] Furthermore, the system for locating specific events in the long video also includes:

[0165] The positioning result display module 105 is used to visually display the local positioning result or the auxiliary positioning result.

[0166] In the embodiment of the present invention, the positioning result display module 105 receives a display request from the user, creates a display interface according to the display request, and then displays the local positioning result or the auxiliary positioning result in the display interface.

[0167] It should be understood that, although the various steps in the flow chart of each embodiment of the present invention are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0168] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0169] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A method for locating a specific event in a long video, characterized in that: The method specifically comprises the following steps: Obtain a basic training set, perform model training, build a lightweight model, and deploy it locally on the terminal device, and build a memory pool based on the basic training set; Receive input video data, segment the video segments, compare the similarities of the video segments through the memory pool, and determine whether the lightweight model can be used directly; When a lightweight model can be used directly, the local positioning result is obtained directly; When the lightweight model cannot be used directly, a remote multimodal large model is used for auxiliary reasoning to obtain auxiliary positioning results and to feed back and upgrade the lightweight model; Visually display local positioning results or auxiliary positioning results.

2. The method for locating a specific event in a long video according to claim 1, wherein: The steps of obtaining a basic training set, performing model training, building a lightweight model, and locally deploying the model on a terminal device, and building a memory pool based on the basic training set specifically include the following steps: Obtain the basic training set, perform model training, and build a lightweight model; Locally deploying the lightweight model on a terminal device; Performing clustering processing on the basic training set to obtain multiple cluster information; A memory pool is constructed based on the plurality of cluster information.

3. The method for locating a specific event in a long video according to claim 1, wherein: The receiving of input video data, segmenting the video segments, and performing similarity comparison on the video segments through the memory pool to determine whether the lightweight model can be directly used specifically include the following steps: receiving input video data; Segmenting the video data to obtain video segments; Performing similarity comparison analysis on the video clip and the cluster centers in the memory pool to obtain similarity accuracy; Based on a preset threshold, the result is compared with the similarity accuracy to determine whether the lightweight model can be used directly.

4. The method for locating a specific event in a long video according to claim 1, wherein: When the lightweight model cannot be used directly, the remote multimodal large model is used for auxiliary reasoning to obtain auxiliary positioning results, and the lightweight model is fed back and upgraded. Specifically, the following steps are included: When the lightweight model cannot be used directly, a prompt is dynamically constructed according to the video clip; Inputting the prompt into a remote multimodal large model; Use a remote multimodal large model for auxiliary reasoning to obtain auxiliary positioning results; Get manual positioning results; Feedback and upgrade the lightweight model.

5. The method for locating a specific event in a long video according to claim 4, characterized in that: The feedback upgrade of the lightweight model specifically includes the following steps: Record multiple auxiliary positioning results and / or multiple manual positioning results; Matching multiple target segments corresponding to multiple auxiliary positioning results and / or multiple manual positioning results; Determining whether the plurality of auxiliary positioning results and / or the plurality of manual positioning results meet a preset update condition; When the plurality of auxiliary positioning results and / or the plurality of manual positioning results meet the update condition, the plurality of target segments, the plurality of auxiliary positioning results and / or the plurality of manual positioning results are integrated to construct an optimized training set; Starting the autonomous learning algorithm of the lightweight model, and fine-tuning and upgrading the lightweight model through the optimized training set; Reconstruct the memory pool based on the optimized training set.

6. The method for locating a specific event in a long video according to claim 1, wherein: The visual display of the local positioning result or the auxiliary positioning result specifically includes the following steps: Receive user's display request; Creating a display interface according to the display request; In the display interface, the local positioning result or the auxiliary positioning result is displayed.

7. A system for locating specific events in long videos, characterized in that: The system includes a lightweight model construction and deployment module, a model selection and judgment module, a lightweight model use module, a multimodal large model use module, and a positioning result display module, wherein: A lightweight model construction and deployment module is used to obtain a basic training set, perform model training, build a lightweight model, and deploy it locally on the terminal device, and build a memory pool based on the basic training set; A model selection and judgment module is used to receive input video data, segment the video segments, compare the similarities of the video segments through the memory pool, and determine whether the lightweight model can be directly used; The lightweight model usage module is used to directly obtain local positioning results when the lightweight model can be used directly; The multimodal large model usage module is used to use the remote multimodal large model for auxiliary reasoning when the lightweight model cannot be used directly, obtain auxiliary positioning results, and feed back and upgrade the lightweight model; The positioning result display module is used to visually display local positioning results or auxiliary positioning results.

8. The system for locating specific events in long videos according to claim 7, characterized in that: The model selection judgment module specifically includes: An input video receiving unit, configured to receive input video data; A video segmentation unit, configured to segment the video data to obtain video segments; A similarity comparison unit, configured to perform similarity comparison analysis on the video clip and the cluster centers in the memory pool to obtain similarity accuracy; The model use judgment unit is used to compare the similarity accuracy with a preset threshold to determine whether the lightweight model can be used directly.

9. The system for locating specific events in long videos according to claim 7, characterized in that: The multimodal large model usage module specifically includes: A prompt construction unit, used to dynamically construct a prompt according to the video clip when the lightweight model cannot be used directly; A prompt input unit, used to input the prompt into a remote multimodal large model; The auxiliary reasoning unit is used to perform auxiliary reasoning using a remote multimodal large model to obtain auxiliary positioning results; A manual result acquisition unit, used for acquiring manual positioning results; The feedback upgrade unit is used to perform feedback upgrade on the lightweight model.

10. The system for locating specific events in long videos according to claim 9, characterized in that: The feedback upgrade unit specifically includes: A result recording unit, used to record multiple auxiliary positioning results and / or multiple manual positioning results; a segment matching unit, configured to match multiple target segments corresponding to multiple auxiliary positioning results and / or multiple manual positioning results; An update condition judgment unit, configured to judge whether a plurality of auxiliary positioning results and / or a plurality of manual positioning results meet a preset update condition; an optimized training set construction unit, configured to construct an optimized training set by integrating the plurality of target segments, the plurality of auxiliary positioning results and / or the plurality of manual positioning results when the plurality of auxiliary positioning results and / or the plurality of manual positioning results meet an update condition; A model fine-tuning and upgrading unit, configured to start an autonomous learning algorithm of the lightweight model and fine-tune and upgrade the lightweight model using the optimized training set; The memory pool reconstruction unit is used to reconstruct the memory pool according to the optimized training set.

Images