A video recognition method, system, device and medium based on clue positioning

Abstract

The application discloses a video recognition method, system, device and medium based on clue positioning, and belongs to the technical field of multi-modal large models and long video understanding. The method comprises the following steps: performing time sequence sampling on audio and video data to obtain a video frame sequence and audio data; inputting query information, the video frame sequence and the audio data into a preset video recognition model to extract text features, visual features and audio features and perform fusion; obtaining event time sequence positioning results and object space positioning results according to the fused features to serve as multiple candidate recognition clues of the audio and video data; obtaining the type of a recognition target according to the query information, and performing recognition instance classification on the multiple candidate recognition clues to obtain a recognition instance set; and outputting a video recognition result of the audio and video data according to the recognition instance set. Therefore, through implementation of the application, high-precision and interpretable recognition of multi-modal content of a long video can be realized.

Description

Technical Field

[0001] This invention relates to multimodal large model and long video understanding technology, and in particular to a video recognition method, system, device and medium based on cue localization. Background Technology

[0002] Existing long video recognition and counting schemes are mainly designed for images or short video clips. For long videos exceeding one minute, they often use sliding windows or segment sampling to process each local segment independently, and then accumulate the recognition results of each segment to obtain the final output.

[0003] However, existing technologies still suffer from the following core technical shortcomings in real-world long-video applications: 1. Severe cross-frame repetition recognition issues: In long videos, the same object or event often appears repeatedly in multiple consecutive segments. Directly accumulating the recognition results of each segment easily leads to a large number of duplicate recognitions. The lack of an effective instance-level differentiation mechanism results in a significant decrease in recognition accuracy. 2. Limited modal utilization: Real-world video scenarios are rich in audio information related to the target being identified, but most existing solutions only support visual single-modal input, ignoring the auxiliary role of audio signals in target recognition. They cannot support joint audio and visual cross-modal recognition and are ill-suited to the actual recognition needs in complex video scenarios. 3. Lack of spatiotemporal positioning basis for recognition results: Most existing solutions only output the final numerical result and rarely provide explicit spatiotemporal information such as the corresponding time interval and spatial location. This makes it difficult to trace the source of recognition errors and determine whether the error is due to temporal boundary perception error or spatial positioning deviation. Summary of the Invention

[0004] This invention provides a video recognition method, system, device, and medium based on cue localization. By implementing this invention, high-precision and interpretable recognition of multimodal content in long videos can be achieved.

[0005] This invention provides a video recognition method based on cue localization, comprising: The audio and video data to be processed and the query information of the target to be identified in the audio and video data are obtained, and the audio and video data are sampled in time sequence to obtain video frame sequence and audio data; The query information, the video frame sequence, and the audio data are input into a preset video recognition model to extract text features, visual features, and audio features and fuse them to obtain fused features; Based on the fusion features, the event temporal localization result and the object spatial localization result are obtained as multiple candidate recognition clues for the audio and video data; The type of the target to be identified is obtained based on the query information, and multiple candidate identification clues are classified into identification instances according to the type of the target to obtain a set of identification instances; The video recognition result of the audio and video data is output based on the set of recognition instances.

[0006] This invention achieves multimodal feature extraction and fusion by temporally sampling audio and video data to obtain video frame sequences and synchronized audio data. The query information, video frame sequences, and audio data are then input into a video recognition model for multimodal feature extraction and fusion. This approach comprehensively utilizes visual content, audio signals, and semantic queries to complete recognition, effectively adapting to the multimodal recognition needs of real-world complex video scenarios. By using event temporal localization results and object spatial localization results obtained based on fused features as candidate recognition clues, explicit localization of target spatiotemporal information in video frames is achieved, providing traceable temporal and spatial evidence for the recognition results, facilitating subsequent verification and error localization. Furthermore, by instantiating and classifying candidate recognition clues according to the type of the target being identified, a structured set of recognition instances is formed. This approach avoids the problem of repeated recognition caused by simply accumulating the scattered recognition results of each frame, thus improving the accuracy of the recognition results. Finally, it outputs structured video recognition results, making the recognition results both quantitative and statistically verifiable, which is convenient for subsequent business system calls. This achieves high-precision and interpretable recognition of multimodal content in long videos. Compared with the problems of existing technologies, such as the lack of traceable spatiotemporal evidence for recognition results, the decrease in accuracy due to repeated recognition across frames, and the single modality utilization, this application obtains candidate recognition clues through multimodal feature fusion and performs instantiation and classification according to the type of recognition target, outputting structured video recognition results. This achieves traceability and high precision of recognition results, effectively improving the reliability of target recognition in long video multimodal scenarios.

[0007] Further, the audio and video data are subjected to time-series sampling to obtain a video frame sequence and audio data, specifically as follows: The video segments of the audio and video data are segmented to obtain an initial video frame sequence; Multiple target video frames are extracted from the initial video frame sequence according to a preset sampling interval, and the timestamp of each target video frame is recorded to obtain a video frame sequence; Audio data is obtained by extracting audio segments synchronized with each of the target video frames from the audio track of the audio and video data based on the timestamp.

[0008] By performing time-series sampling of audio and video data, and through segmentation and target video frame extraction, the amount of redundant information in the video data can be effectively reduced. While retaining key visual content, the computational burden of subsequent processing is significantly reduced, thereby improving the overall recognition efficiency. By recording the precise timestamps of each target video frame and extracting the corresponding audio segments, strict alignment between the video frame sequence and the audio data in the time dimension is ensured, providing a time-consistent data foundation for the joint extraction and fusion of subsequent multimodal features.

[0009] Furthermore, the training process of the video recognition model includes: Obtain a preset recognition training dataset, and perform supervised fine-tuning of the initial model based on the recognition training dataset to obtain the first training model; The recognition training dataset is grouped according to the difficulty of the training task, and the first training model is subjected to group reinforcement learning by combining the grouping relative policy optimization algorithm to obtain the second training model. The recognition training dataset is sampled evenly according to the training task type to obtain a sampled training set. The second training model is then subjected to full-task reinforcement learning based on the sampled training set to obtain a trained video recognition model.

[0010] By acquiring a pre-set recognition training dataset and supervising the fine-tuning of the initial model, the model can quickly learn basic multimodal feature representations and recognition capabilities, providing stable initialization parameters for subsequent refined training. By grouping the recognition training dataset according to the difficulty of the training task and combining the grouping relative policy optimization algorithm to perform grouped reinforcement learning on the first training model, the model can progressively adapt to the recognition patterns of samples of different difficulty levels from easy to difficult. The grouping relative policy optimization effectively alleviates the policy bias problem caused by uneven distribution of sample difficulty in traditional reinforcement learning, enhances the model's discrimination ability on difficult samples, and improves the stability and convergence speed of the training process. By performing balanced sampling of the recognition training dataset according to the training task type to obtain a sampled training set, and performing full-task reinforcement learning on the second training model, it is possible to ensure that the model obtains sufficient and balanced optimization on various types of recognition tasks, comprehensively improving the model's generalization ability to diverse video scenes.

[0011] Furthermore, after obtaining the trained video recognition model, the process also includes: Obtain a recognition test dataset containing multiple audio and video test samples and manually labeled data; The recognition test dataset is input into the trained video recognition model to obtain the predicted recognition results of multiple audio and video test samples; wherein, the predicted recognition results include the predicted occurrence frequency and predicted location clues of multiple recognition targets in multiple audio and video test samples; Based on the predicted frequency of occurrence and the actual frequency of occurrence of the manually labeled data, the evaluation index of the recognition result of the trained video recognition model is calculated. Based on the predicted location clues and the manually labeled actual location clues, calculate the recognition process evaluation index of the trained video recognition model; The trained video recognition model is updated based on the recognition result evaluation index and the recognition process evaluation index to obtain an updated video recognition model, which is then used to perform video recognition on the audio and video data to be processed.

[0012] By acquiring a recognition test dataset containing multiple audio and video test samples and manually annotated data, and inputting this dataset into a trained video recognition model to obtain the predicted recognition results for each sample, a standardized test benchmark covering diverse real-world scenarios can be constructed, providing a reliable data foundation for the quantitative evaluation of model performance. By calculating recognition result evaluation metrics based on predicted occurrence frequency and actual occurrence frequency in manually annotated data, the accuracy of the model in target counting and category discrimination can be objectively quantified, effectively reflecting the reliability of the model's final output. Furthermore, by calculating recognition process evaluation metrics based on predicted location cues and actual location cues in manually annotated data, the spatiotemporal positioning accuracy dimension can be independently evaluated. The evaluation model's performance in the intermediate inference stage reveals specific deviations in temporal and spatial localization, providing fine-grained analytical basis for diagnosing model weaknesses. Furthermore, the trained video recognition model is updated based on recognition result evaluation metrics and recognition process evaluation metrics, achieving dual-driven optimization. This allows the model to adjust not only based on the final recognition accuracy but also to make targeted corrections based on spatiotemporal localization accuracy errors, significantly improving the efficiency and directional accuracy of model iterative optimization. The resulting updated video recognition model can perform video recognition on processed audio and video data with higher accuracy and localization precision, effectively ensuring recognition quality in practical deployment applications.

[0013] Furthermore, event temporal localization results and object spatial localization results are obtained based on the fusion features, and used as multiple candidate recognition clues for the audio and video data, specifically: Temporal boundary detection is performed on the fusion features to obtain each recognition event of the audio and video data and the start timestamp and end timestamp corresponding to each recognition event, so as to use the start timestamp and end timestamp as the temporal localization results of multiple events of the audio and video data; The fused features are used to detect objects and obtain the bounding box coordinates of each identified object in the audio and video data, so as to use the bounding box coordinates as the spatial localization results of multiple objects in the audio and video data. The event time-series localization results and object spatial localization results are verified, and the verified event time-series localization results and object spatial localization results are used as multiple candidate recognition clues for the audio and video data.

[0014] By performing temporal boundary detection on the fused features to obtain event temporal localization results, the temporal boundaries of each identified event can be accurately segmented from a continuous video stream, achieving explicit localization of the occurrence interval of dynamic events and providing accurate time anchors for subsequent temporal tracing of identification results. By performing object detection on the fused features to obtain object spatial localization results, the spatial position of the target object in each relevant frame can be accurately locked, achieving fine-grained spatial localization of visual targets in the image plane, providing clear positional evidence for spatial tracing and visual verification of identification results. Furthermore, by performing fact verification on multiple event temporal localization results and multiple object spatial localization results, and using the verified localization results as candidate identification clues, false localizations caused by model misdetection or feature noise can be effectively eliminated, ensuring the inherent consistency of candidate identification clues in the temporal and spatial dimensions, significantly reducing the false detection rate and false alarm rate, while also reducing computational redundancy and interference in the subsequent instantiation and classification process.

[0015] Furthermore, based on the type of the target being identified, multiple candidate identification clues are classified into identification instances to obtain a set of identification instances, specifically as follows: When the target to be identified is an event type, the time intervals of the same identification event among multiple candidate identification clues are aggregated into the same event instance to obtain several event instances; When the target is identified as an object, cross-frame association matching is performed on the identified objects in each of the candidate identification clues, and the detection results of the same identified object instance are merged into the same object instance to obtain several object instances. When the target to be identified is an attribute type, multiple objects in each of the candidate identification clues are spatially located, and the detection results of multiple objects are clustered and grouped according to the attribute category of the query information to obtain several attribute instances. A set of identified instances is obtained based on several event instances, several object instances, and several attribute instances.

[0016] By aggregating time intervals belonging to the same recognition event from multiple candidate recognition cues into a single event instance, the model can effectively merge multiple time-series segments detected by the model according to event affiliation. This ensures that the final event instance is complete, continuous, and non-overlapping in time intervals, significantly improving the accuracy and completeness of event-level recognition results. Furthermore, by performing cross-frame association matching on the objects in each candidate recognition cues and merging the detection results of the same object instance into a single object instance, the model can track the same target entity in the temporal dimension based on object appearance features, motion trajectories, or spatial location consistency. This ensures that each object instance is recorded only once within a time span, while preserving its complete spatiotemporal trajectory information, which is crucial for subsequent quantization. Statistical and behavioral analysis provides reliable entity-level data. By spatially locating multiple objects in each candidate recognition clue and clustering the detection results of multiple objects according to the attribute categories of the query information, several attribute instances are obtained. This allows for the aggregation of object recognition results with the same attribute category according to semantic category, accurately distinguishing the attribute categories to which different objects belong, and making the statistical results of the attribute dimension clear and explicit. Furthermore, based on several event instances, object instances, and attribute instances, a structured set of recognition instances is obtained, realizing the categorized and hierarchical instantiation organization of various recognition targets in video content, so that each instance has a clear type label, time interval or spatial range, and associated attribute information.

[0017] Furthermore, based on the set of recognition instances, the video recognition result of the audio and video data is output, specifically as follows: The recognition results of each recognition target in the audio and video data are output based on multiple recognition instances in the recognition instance set, and are used as video recognition results. Specifically, when the recognition target is an event, the occurrence frequency and time information of the corresponding event instance are output; when the recognition target is an object, the occurrence frequency of the corresponding object instance and the time and spatial information of its first appearance are output; when the recognition target is an attribute category, the occurrence frequency of the corresponding attribute instance and the spatial information and attribute category identifier of each object instance under each attribute instance are output.

[0018] By outputting the recognition results of each recognition target in the audio and video data based on multiple recognition instances in the recognition instance set, and adopting differentiated output strategies for different recognition target types, the event-level recognition results can be presented in the form of quantified frequency and clear time intervals, accurately restore the initial appearance time and spatial area of ​​the object in the video, and display the distribution of objects with certain attribute characteristics. This makes the recognition results easy for business systems to directly call and easy to quickly locate specific target instances during review.

[0019] Another embodiment of the present invention provides a video recognition system based on clue localization, including: an initialization processing module, a fusion feature generation module, a candidate recognition clue acquisition module, a recognition instance classification module, and a video recognition result acquisition module; The initialization processing module is used to acquire the audio and video data to be processed and the query information of the target to be identified in the audio and video data, and to perform time-series sampling on the audio and video data to obtain video frame sequences and audio data; The fusion feature generation module is used to input the query information, the video frame sequence, and the audio data into a preset video recognition model to extract text features, visual features, and audio features and fuse them to obtain fusion features; The candidate identification clue acquisition module is used to acquire event temporal positioning results and object spatial positioning results based on the fusion features, so as to serve as multiple candidate identification clues for the audio and video data; The identification instance classification module is used to obtain the type of the identification target based on the query information, and classify multiple candidate identification clues into identification instances based on the type of the identification target to obtain a set of identification instances; The video recognition result acquisition module is used to output the video recognition result of the audio and video data based on the recognition instance set.

[0020] This invention achieves multimodal feature extraction and fusion by temporally sampling audio and video data to obtain video frame sequences and synchronized audio data. The query information, video frame sequences, and audio data are then input into a video recognition model for multimodal feature extraction and fusion. This approach comprehensively utilizes visual content, audio signals, and semantic queries to complete recognition, effectively adapting to the multimodal recognition needs of real-world complex video scenarios. By using event temporal localization results and object spatial localization results obtained based on fused features as candidate recognition clues, explicit localization of target spatiotemporal information in video frames is achieved, providing traceable temporal and spatial evidence for the recognition results, facilitating subsequent verification and error localization. Furthermore, by instantiating and classifying candidate recognition clues according to the type of the target being identified, a structured set of recognition instances is formed. This approach avoids the problem of repeated recognition caused by simply accumulating the scattered recognition results of each frame, thus improving the accuracy of the recognition results. Finally, it outputs structured video recognition results, making the recognition results both quantitative and statistically verifiable, which is convenient for subsequent business system calls. This achieves high-precision and interpretable recognition of multimodal content in long videos. Compared with the problems of existing technologies, such as the lack of traceable spatiotemporal evidence for recognition results, the decrease in accuracy due to repeated recognition across frames, and the single modality utilization, this application obtains candidate recognition clues through multimodal feature fusion and performs instantiation and classification according to the type of recognition target, outputting structured video recognition results. This achieves traceability and high precision of recognition results, effectively improving the reliability of target recognition in long video multimodal scenarios.

[0021] Another embodiment of the present invention provides a terminal device, including: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the steps of the video recognition method based on cue localization of the present invention.

[0022] Another embodiment of the present invention also provides a computer-readable storage medium item, including: a stored computer program, which, when the computer program is running, controls the device where the computer-readable storage medium is located to perform the steps of the video recognition method based on cue localization of the present invention. Attached Figure Description

[0023] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating an embodiment of the video recognition method based on cue localization provided by the present invention; Figure 2 This is an architecture diagram of a video recognition model for another embodiment of the video recognition method based on cue localization provided by the present invention; Figure 3 This is a flowchart of the training process of a video recognition model, representing another embodiment of the video recognition method based on cue localization provided by the present invention. Figure 4 This is a schematic diagram of a module of another embodiment of the video recognition system based on clue localization provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0027] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0028] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0029] In the description of this invention, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed herein.

[0030] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0031] See Figure 1 To address the video recognition problem in existing technologies, an embodiment of the present invention provides a video recognition method based on cue localization, comprising steps S1 to S5, the specific steps of which are as follows: S1. Obtain the audio and video data to be processed and the query information of the target identified in the audio and video data, and perform time-series sampling on the audio and video data to obtain video frame sequences and audio data; Specifically, the video segments of the audio and video data are segmented to obtain an initial video frame sequence. Multiple target video frames are extracted from the initial video frame sequence according to a preset sampling interval, and the timestamps of each target video frame are recorded to obtain a video frame sequence. Audio segments synchronized with each target video frame are extracted from the audio track of the audio and video data according to the timestamps to obtain audio data.

[0032] S2. Input the query information, the video frame sequence, and the audio data into a preset video recognition model to extract text features, visual features, and audio features and fuse them to obtain fused features; Specifically, the video frame sequence is encoded by a visual encoder in the video recognition model to obtain visual features; the audio data is encoded by an audio encoder in the video recognition model to obtain audio features; the query information is text-encoded to obtain text features; the visual features and the audio features are mapped to a feature space adapted to the language decoding module by a feature projection module; the target modal features required for the current recognition task are determined from the mapped visual features and audio features according to the modal dependencies indicated by the query information; and the target modal features are aligned and fused with the text features to obtain fused features.

[0033] S3. Obtain event temporal localization results and object spatial localization results based on the fusion features, so as to serve as multiple candidate recognition clues for the audio and video data; Specifically, temporal boundary detection is performed on the fused features to obtain each recognition event in the audio and video data, along with its corresponding start and end timestamps, which are then used as multiple event temporal localization results for the audio and video data. Object detection is also performed on the fused features to obtain each recognition object in the audio and video data, along with its bounding box coordinates, which are then used as multiple object spatial localization results for the audio and video data. Finally, fact verification is performed on the multiple event temporal localization results and the multiple object spatial localization results, and the verified multiple event temporal localization results and the multiple object spatial localization results are used as multiple candidate recognition clues for the audio and video data.

[0034] S4. Obtain the type of the target to be identified based on the query information, and classify the multiple candidate identification clues into identification instances based on the type of the target to obtain a set of identification instances; Specifically, when the target is an event type, time intervals belonging to the same event among multiple candidate identification clues are aggregated into a single event instance, resulting in several event instances; when the target is an object type, cross-frame association matching is performed on the objects in each candidate identification clue, and the detection results of the same object instance are merged into a single object instance, resulting in several object instances; when the target is an attribute type, spatial localization is performed on multiple objects in each candidate identification clue, and the detection results of multiple objects are clustered and grouped according to the attribute category of the query information, resulting in several attribute instances; and a set of identification instances is obtained based on the several event instances, the several object instances, and the several attribute instances.

[0035] S5. Output the video recognition result of the audio and video data according to the set of recognition instances; Specifically, the recognition results of each recognition target in the audio and video data are output based on multiple recognition instances in the recognition instance set, serving as the video recognition results; when the recognition target is an event, the occurrence frequency and time information of the corresponding event instance are output; when the recognition target is an object, the occurrence frequency of the corresponding object instance and the time and spatial information of its first appearance are output; when the recognition target is an attribute category, the occurrence frequency of the corresponding attribute instance and the spatial information and attribute category identifier of each object instance under each attribute instance are output.

[0036] This invention achieves multimodal feature extraction and fusion by temporally sampling audio and video data to obtain video frame sequences and synchronized audio data. The query information, video frame sequences, and audio data are then input into a video recognition model for multimodal feature extraction and fusion. This approach comprehensively utilizes visual content, audio signals, and semantic queries to complete recognition, effectively adapting to the multimodal recognition needs of real-world complex video scenarios. By using event temporal localization results and object spatial localization results obtained based on fused features as candidate recognition clues, explicit localization of target spatiotemporal information in video frames is achieved, providing traceable temporal and spatial evidence for the recognition results, facilitating subsequent verification and error localization. Furthermore, by instantiating and classifying candidate recognition clues according to the type of the target being identified, a structured set of recognition instances is formed. This approach avoids the problem of repeated recognition caused by simply accumulating the scattered recognition results of each frame, thus improving the accuracy of the recognition results. Finally, it outputs structured video recognition results, making the recognition results both quantitative and statistically verifiable, which is convenient for subsequent business system calls. This achieves high-precision and interpretable recognition of multimodal content in long videos. Compared with the problems of existing technologies, such as the lack of traceable spatiotemporal evidence for recognition results, the decrease in accuracy due to repeated recognition across frames, and the single modality utilization, this application obtains candidate recognition clues through multimodal feature fusion and performs instantiation and classification according to the type of recognition target, outputting structured video recognition results. This achieves traceability and high precision of recognition results, effectively improving the reliability of target recognition in long video multimodal scenarios.

[0037] In one embodiment, the audio and video data are time-series sampled to obtain a video frame sequence and audio data, including steps S201 to S203, each step of which is as follows: S201. The video segments of the audio and video data are segmented to obtain an initial video frame sequence; S202. Extract multiple target video frames from the initial video frame sequence according to a preset sampling interval, and record the timestamp of each target video frame to obtain a video frame sequence; S203. Based on the timestamp, extract audio segments synchronized with each of the target video frames from the audio track of the audio and video data to obtain audio data; For data containing audio, audio data corresponding to the video frame or corresponding time interval is obtained along the same time axis; the sampling interval is determined based on the video duration, the maximum input length of the model, and deployment resources.

[0038] This invention, through temporal sampling of audio and video data and segmentation processing and target video frame extraction, effectively reduces redundant information in video data, significantly reduces the computational burden of subsequent processing while retaining key visual content, and improves overall recognition efficiency. By recording the precise timestamps of each target video frame and extracting synchronized audio segments accordingly, it ensures strict alignment between the video frame sequence and audio data in the temporal dimension, providing a temporally consistent data foundation for the joint extraction and fusion of subsequent multimodal features.

[0039] In one embodiment, the training process of the video recognition model includes steps S301 to S303, each step being as follows: S301. Obtain a preset recognition training dataset, and perform supervised fine-tuning of the initial model based on the recognition training dataset to obtain the first training model; S302. The recognition training dataset is grouped according to the difficulty of the training task, and the first training model is subjected to group reinforcement learning by combining the grouping relative policy optimization algorithm to obtain the second training model. S303. The recognition training dataset is sampled evenly according to the training task type to obtain a sampled training set, and the second training model is subjected to full-task reinforcement learning based on the sampled training set to obtain a trained video recognition model. The video recognition model uses Ola-Omni-7B as the base, SigLIP as the visual encoder, Whisper as the audio encoder, and the language decoding module uses the language model structure corresponding to the base model. The alignment of visual features, audio features and language model feature space is completed through the MLP projection layer. The localization results are in JSON format. See Figure 2 The video recognition model uses a visual encoder to encode features in the video frame sequence, and an audio encoder to encode features in the corresponding audio data when audio data is present. It also represents the counting query information as text features. The feature projection module maps the visual features and the audio features when they are present to a feature space that is compatible with the language decoding module. The language decoding module combines the counting query to perform cross-modal feature fusion, query-guided spatiotemporal clue localization, and instance-level counting inference, and outputs the temporal localization result, spatial localization result, and counting result. The training data covers audio / video question answering, temporal localization, spatial localization, and counting tasks. Audio / video question answering tasks can use AVQA and MUSIC-AVQA; spatiotemporal localization tasks can use AVE, UnAV, LLP, and AVSS; and counting tasks can use DVD-Counting and RepCount. For the ARIG task, the AVSS segmentation mask can be converted into bounding boxes while preserving the original image resolution to reduce localization errors caused by coordinate normalization or scaling. For training tasks of varying difficulty (audio / video question answering, spatiotemporal localization, and counting), format rewards and task-specific performance rewards are set. The format reward checks whether the question-answering or counting output contains preset inference and answer blocks, and whether the localization result can be parsed and whether the required fields are complete. The task-specific performance rewards use question-answering accuracy, localization intersection-union ratio, and counting relative mean absolute error to measure the output quality of different tasks. When the standard count value The formula for calculating the relative mean absolute error reward is as follows: when When the time is right, the calculation formula is as follows: in, The reward is the relative mean absolute error; Pred is the model prediction value. See Figure 3 The first stage of training is the cold start supervised fine-tuning stage, which uses temporal localization, spatial localization, and counting task data to supervise and fine-tune the base model, enabling the model to acquire basic spatiotemporal localization, counting inference, and structured result output capabilities. In practical applications, this application sets the batch size to 64 and the learning rate to [missing information]. The training iterations were 20,000 steps, using the AdamW optimizer with a weight decay of 0.01 and a maximum sequence length of 32,768. The second stage is a course-based reinforcement learning stage with a stage review mechanism. The training tasks are divided into three levels—question answering, spatiotemporal localization, and counting—according to increasing difficulty, and training is completed sequentially based on a grouping relative strategy optimization algorithm. Before each training stage, the same training sample is inferred multiple times using a reference model. For question answering and counting tasks, simple samples that are correct in multiple inferences are filtered out. For localization tasks, simple samples with an average intersection-union ratio (IU) higher than a preset threshold are filtered out. To reduce the decline in prior task capabilities caused by subsequent task learning, a preset proportion of completed training task samples are mixed into the current training stage. In practical applications, before each round of course-based reinforcement learning, each training sample is inferred 5 times using the reference model. For audio / video question answering and counting tasks, the corresponding training sample is filtered when all 5 inference results match the standard answer. For spatiotemporal localization tasks, the average IU between the 5 localization results and the labeled results is calculated; when the average IU is greater than 0.9, the corresponding training sample is filtered out. During training, a stage review mechanism is used, mixing 20% ​​of the trained task samples into subsequent training stages. The third stage is the full-task reinforcement learning stage, which involves balanced sampling from samples of different tasks and difficulties, and then performing full-task GRPO reinforcement learning to balance the acquisition of new capabilities and the retention of existing capabilities, thereby improving the overall generalization performance of the model.

[0040] This invention, through obtaining a pre-set recognition training dataset and supervising the fine-tuning of the initial model, enables the model to quickly learn basic multimodal feature representations and recognition capabilities, providing stable initialization parameters for subsequent refined training. By grouping the recognition training dataset according to the difficulty of the training task and combining it with a grouping relative policy optimization algorithm to perform grouped reinforcement learning on the first training model, the model can progressively adapt to the recognition patterns of samples of different difficulty levels from easy to difficult. The grouping relative policy optimization effectively alleviates the policy bias problem caused by uneven distribution of sample difficulty in traditional reinforcement learning, enhances the model's discrimination ability on difficult samples, and improves the stability and convergence speed of the training process. By performing balanced sampling of the recognition training dataset according to the training task type to obtain a sampled training set and performing full-task reinforcement learning on the second training model, it can ensure that the model obtains sufficient and balanced optimization on various types of recognition tasks, comprehensively improving the model's generalization ability to diverse video scenes.

[0041] In one embodiment, after obtaining the trained video recognition model, steps S401 to S405 are further included, and the specific steps are as follows: S401. Obtain a recognition test dataset containing multiple audio and video test samples and manually labeled data. The identification test dataset is constructed in three stages: question generation, answer annotation, and clue annotation. First, a multimodal model generates candidate counting questions and query intervals for long videos, which are then manually reviewed, corrected, or supplemented. Next, annotators watch the videos and determine the standard answers and reference time intervals. Finally, fine-grained clues are annotated according to the type of counting target. Event counting annotates the start and end timestamps of events, object counting annotates the bounding boxes when an object first appears or the independent bounding boxes of multiple objects in the same frame, and attribute counting first completes object-level annotation and then groups according to the query attributes. In practical applications, long videos covering multiple content categories such as sports, daily life, tutorials, and humor are selected from CCGBench. A multimodal model is used to generate candidate counting questions and query intervals for each video. Human annotators review the questions for their reasonableness, unambiguity, and answerability, making corrections, rejections, or additions. After review, annotators review the entire video content to determine the standard answer and reference time interval. For questions requiring a global understanding, the reference interval can cover the entire video; for local counting questions, the reference interval corresponds to relevant segments of the target. In the clue annotation stage, event counting annotates the start and end times of each independent event, object counting annotates the bounding box of each object instance when it first appears, and annotates multiple object instances in the same scene separately. Attribute counting first completes the object-level bounding box annotation, and then groups the object instances according to the query attributes. After annotation, the number of clues and the grouping results are checked for consistency with the standard answer, and manual arbitration is performed on inconsistent samples. In this application, the recognition test dataset is constructed based on 497 long videos, containing 1027 multimodal counting questions and 5845 manually annotated fine-grained counting clues.

[0042] S402. Input the recognition test dataset into the trained video recognition model to obtain the predicted recognition results of multiple audio and video test samples; wherein, the predicted recognition results include the predicted occurrence frequency and predicted location clues of multiple recognition targets in multiple audio and video test samples; S403. Calculate the recognition result evaluation index of the trained video recognition model based on the predicted frequency of occurrence and the actual frequency of occurrence of the manually labeled data. The evaluation metrics for the recognition results are obtained using black-box evaluation, including full-video Long Acc mode and reference interval RefAcc mode. The final counting results are measured using Acc, Ob-1 Accuracy, Mean Absolute Error (MAE), and Root Mean Square Error (RMSE). Let the number of test samples be N, and the predicted count value and standard count value of the i-th sample be respectively... and The specific formula is as follows: , , , in, The number of samples whose predicted count is exactly equal to the standard count; This is the number of samples for which the absolute value of the difference between the predicted count and the standard count is no greater than 1.

[0043] S404. Based on the predicted positioning clues and the manually labeled actual positioning clues, calculate the recognition process evaluation index of the trained video recognition model; The evaluation metrics for the recognition process are obtained using white-box evaluation, including event counting using temporal intersection-over-union ratio (IoU), object counting using bounding box IoU, and attribute counting using group matching. The white-box counting score (WCS) considers both localization accuracy and counting accuracy, and the specific calculation formula is as follows: , , Where K is the number of instance groups; The average localization accuracy of the kth group of instances after matching; This is a counting precision penalty term for the k-th group; and These are the set of real instances and the set of predicted instances for the k-th group, respectively. Indicates the intersection-union ratio (IU) of a time series interval or spatial bounding box; The instruction adherence accuracy (IFA) is used to measure the proportion of model outputs that meet preset format requirements. The calculation formula is as follows: in, The number of samples required to conform to the preset output format.

[0044] S405. Update the trained video recognition model according to the recognition result evaluation index and the recognition process evaluation index to obtain an updated video recognition model, so as to perform video recognition on the audio and video data to be processed according to the updated video recognition model. Before the update, some test results for different models are shown in Table 1: Table 1. Performance of various MLLMs in CG-AV-Counting Where V represents visual input, and A+V represents combined audio and video input; Based on the evaluation metrics for the recognition results and the evaluation metrics for the recognition process, the trained video recognition model is updated and trained. The updated and trained model is validated using CG-AV-Counting, DVD-Counting, and other audio and video understanding datasets. In the Long Acc mode of CG-AV-Counting, the model receives a complete long video and simultaneously completes temporal localization and counting. In the Ref Acc mode, only the reference interval corresponding to the counting problem is input to evaluate the complete processing capability and the pure counting capability, respectively. The comparison results of the base model and AV-Reasoner on the counting-specific benchmark after the update are shown in Table 2: Table 2. Performance comparison of each model on the counting benchmark. In this context, AV-Reasoner-Thinking indicates a setting that requires the model to explicitly output the inference process; AV-Reasoner indicates a setting that does not force the output of the explicit inference process. Table 3 shows the comparison results of AV-Reasoner and some closed-source models on the comprehensive audio and video benchmark: Table 3. Performance comparison of AV-Reasoner with closed-source multimodal large language models on benchmark sets. Table 4 shows the comparison results of AV-Reasoner with some open-source models in audio and video question answering, spatiotemporal localization, and counting tasks: Table 4. Performance comparison of AV-Reasoner with open-source multimodal large language models on benchmark sets. The above results show that, compared with the video recognition model before the update, the AV-Reasoner obtained through further updated training, supervised fine-tuning, curriculum-based reinforcement learning, stage review mechanism and full-task reinforcement learning, achieves significant improvements on multiple counting and audio-visual understanding tasks.

[0045] This invention, through obtaining a recognition test dataset containing multiple audio and video test samples and manually annotated data, inputs this dataset into a trained video recognition model to obtain the predicted recognition results for each sample. This enables the construction of a standardized test benchmark covering diverse real-world scenarios, providing a reliable data foundation for the quantitative evaluation of model performance. By calculating recognition result evaluation metrics based on predicted occurrence frequency and actual occurrence frequency in manually annotated data, the accuracy of the model in target counting and category discrimination is objectively quantified, effectively reflecting the reliability of the model's final output. Furthermore, by calculating recognition process evaluation metrics based on predicted positioning cues and actual positioning cues in manually annotated data, the spatiotemporal positioning accuracy dimension can be improved. Independent evaluation of the model's performance in the intermediate inference stage reveals specific deviations in temporal and spatial localization, providing fine-grained analytical basis for diagnosing model weaknesses. Furthermore, the trained video recognition model is updated based on both recognition result evaluation metrics and recognition process evaluation metrics, achieving dual-driven optimization. This allows the model to adjust not only based on the final recognition accuracy but also to make targeted corrections based on spatiotemporal localization accuracy errors, significantly improving the efficiency and directional accuracy of model iterative optimization. The resulting updated video recognition model can perform video recognition on processed audio and video data with higher accuracy and localization precision, effectively ensuring recognition quality in practical deployment applications.

[0046] In one embodiment, event temporal localization results and object spatial localization results are obtained based on the fusion features to serve as multiple candidate recognition cues for the audio and video data, including steps S501 to S503, each step of which is as follows: S501. Perform temporal boundary detection on the fusion features, obtain each recognition event of the audio and video data and the start timestamp and end timestamp corresponding to each recognition event, so as to use the start timestamp and end timestamp as the temporal positioning results of multiple events of the audio and video data; S502. Perform object detection on the fused features to obtain the bounding box coordinates of each identified object in the audio and video data, so as to use the bounding box coordinates as the spatial positioning result of multiple objects in the audio and video data. S503. Perform fact verification on the multiple event temporal positioning results and multiple object spatial positioning results, and use the multiple event temporal positioning results and multiple object spatial positioning results that pass the verification as multiple candidate recognition clues for the audio and video data. Specifically, the visual features and the audio features present when they are present are mapped to a feature space adapted to the language decoding module through the feature projection module; according to the modal dependency relationship indicated by the counting query information, the target modal features corresponding to the current counting task are determined from the mapped visual features and the audio features present when they are present, and the target modal features are aligned and fused with the text features to obtain fused features for spatiotemporal cue localization and counting reasoning; The event timing location results and object spatial location results are verified to check whether the timing location results are within the effective time range of the video and whether the object spatial location results are within the image range. The necessary fields corresponding to the target type are also checked to ensure they are complete. For results that do not meet the structured output format but can be recovered through preset rules, format recovery can be performed. Results that cannot be recovered are marked as invalid location results and removed.

[0047] This invention, through temporal boundary detection of fused features to obtain event temporal localization results, can accurately segment the temporal boundaries of each identified event from a continuous video stream, achieving explicit localization of the occurrence interval of dynamic events and providing accurate time anchors for subsequent temporal tracing of identification results. By performing object detection on fused features to obtain object spatial localization results, it can accurately pinpoint the spatial position of the target object in each relevant frame, achieving fine-grained spatial localization of visual targets within the image plane, providing clear positional evidence for spatial tracing and visual verification of identification results. Furthermore, by performing fact verification on multiple event temporal localization results and multiple object spatial localization results, and using the verified localization results as candidate identification clues, it can effectively eliminate false localizations caused by model misdetection or feature noise, ensuring the inherent consistency of candidate identification clues in temporal and spatial dimensions, significantly reducing false detection and false alarm rates, while also reducing computational redundancy and interference in subsequent instantiation and classification stages.

[0048] In one embodiment, multiple candidate identification clues are classified into identification instances according to the type of the identification target to obtain an identification instance set, including steps S601 to S604, each step of which is as follows: S601. When the target to be identified is an event type, the time intervals of the same identification event among the multiple candidate identification clues are aggregated into the same event instance to obtain several event instances; S602. When the target is an object, cross-frame association matching is performed on the objects in each of the candidate identification clues, and the detection results of the same object instance are merged into the same object instance to obtain several object instances. S603. When the target to be identified is an attribute type, spatial positioning is performed on multiple objects in each of the candidate identification clues, and the detection results of multiple objects to be identified are clustered and grouped according to the attribute category of the query information to obtain several attribute instances. The query information describes the recognition target and recognition events of the audio and video data, and indicates the modal dependencies corresponding to video recognition, including visual monomodality, audio monomodality, visual reference-audio recognition, audio reference-visual recognition, and audio-video joint recognition. Visual monomodality refers to completing target localization and recognition based on video visual information; audio monomodality refers to completing target localization and recognition based on audio information; visual reference-audio counting refers to using visual information to determine the time range corresponding to the target to be recognized, and completing recognition based on the audio information within that time range; audio reference-visual counting refers to using audio information to determine the time range corresponding to the target to be recognized, and completing recognition based on the visual information within that time range; and audio-video joint counting refers to combining visual and audio information to jointly determine recognition clues and complete recognition.

[0049] S604. Obtain a set of identified instances based on a plurality of event instances, a plurality of object instances, and a plurality of attribute instances; This invention, by aggregating time intervals belonging to the same identification event from multiple candidate identification clues into a single event instance, effectively merges multiple time-series segments detected by the model according to event affiliation. This ensures that the final event instance is complete, continuous, and non-overlapping in time intervals, significantly improving the accuracy and completeness of event-level identification results. Furthermore, by performing cross-frame association matching on the identified objects in each candidate identification clue and merging the detection results of the same identified object instance into a single object instance, it can track the same target entity in the time dimension based on object appearance features, motion trajectory, or spatial location consistency. This ensures that each object instance is recorded only once within a time span, while preserving its complete spatiotemporal trajectory information for subsequent... Quantitative statistics and behavioral analysis provide reliable entity-level data. By spatially locating multiple objects in each candidate recognition clue and clustering the detection results of multiple objects according to the attribute categories of the query information, several attribute instances are obtained. This allows for the aggregation of object recognition results with the same attribute category according to semantic category, accurately distinguishing the attribute categories to which different objects belong, and making the statistical results of the attribute dimension clear and explicit. Furthermore, based on several event instances, object instances, and attribute instances, a structured set of recognition instances is obtained, realizing the categorized and hierarchical instantiation organization of various recognition targets in video content, so that each instance has a clear type label, time interval or spatial range, and associated attribute information.

[0050] like Figure 4As shown in the video recognition based on clue localization, a corresponding system implementation is provided based on the above method implementation.

[0051] This invention provides a video recognition system based on clue localization, comprising: an initialization processing module 701, a fusion feature generation module 702, a candidate recognition clue acquisition module 703, a recognition instance classification module 704, and a video recognition result acquisition module 705; The initialization processing module 701 is used to acquire the audio and video data to be processed and the query information of the target to be identified in the audio and video data, and to perform time-series sampling on the audio and video data to obtain video frame sequences and audio data. The fusion feature generation module 702 is used to input the query information, the video frame sequence and the audio data into a preset video recognition model to extract text features, visual features and audio features and fuse them to obtain fusion features; The candidate identification clue acquisition module 703 is used to acquire event temporal positioning results and object spatial positioning results based on the fusion features, so as to serve as multiple candidate identification clues for the audio and video data. The identification instance classification module 704 is used to obtain the type of the identification target according to the query information, and classify multiple candidate identification clues into identification instances according to the type of the identification target to obtain an identification instance set; The video recognition result acquisition module 705 is used to output the video recognition result of the audio and video data according to the recognition instance set.

[0052] This invention achieves multimodal feature extraction and fusion by temporally sampling audio and video data to obtain video frame sequences and synchronized audio data. The query information, video frame sequences, and audio data are then input into a video recognition model for multimodal feature extraction and fusion. This approach comprehensively utilizes visual content, audio signals, and semantic queries to complete recognition, effectively adapting to the multimodal recognition needs of real-world complex video scenarios. By using event temporal localization results and object spatial localization results obtained based on fused features as candidate recognition clues, explicit localization of target spatiotemporal information in video frames is achieved, providing traceable temporal and spatial evidence for the recognition results, facilitating subsequent verification and error localization. Furthermore, by instantiating and classifying candidate recognition clues according to the type of the target being identified, a structured set of recognition instances is formed. This approach avoids the problem of repeated recognition caused by simply accumulating the scattered recognition results of each frame, thus improving the accuracy of the recognition results. Finally, it outputs structured video recognition results, making the recognition results both quantitative and statistically verifiable, which is convenient for subsequent business system calls. This achieves high-precision and interpretable recognition of multimodal content in long videos. Compared with the problems of existing technologies, such as the lack of traceable spatiotemporal evidence for recognition results, the decrease in accuracy due to repeated recognition across frames, and the single modality utilization, this application obtains candidate recognition clues through multimodal feature fusion and performs instantiation and classification according to the type of recognition target, outputting structured video recognition results. This achieves traceability and high precision of recognition results, effectively improving the reliability of target recognition in long video multimodal scenarios.

[0053] It is understood that the above system embodiments correspond to the method embodiments of the present invention, and can implement the video recognition method based on cue localization provided by any of the above method embodiments of the present invention.

[0054] It should be noted that the system embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the system embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0055] For ease of description and brevity, the system embodiments of the present invention include all the implementation methods described in the above-described cue-based video recognition method embodiments, and will not be repeated here.

[0056] Based on the above embodiments of the video recognition method based on cue localization, another embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the video recognition method based on cue localization of any embodiment of the present invention.

[0057] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the terminal device.

[0058] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0059] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0060] Based on the above-described method embodiments, another embodiment of the present invention provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the video recognition method based on cue localization described in any of the above-described method embodiments of the present invention.

[0061] The modules / units integrated in the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0062] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A video recognition method based on cue localization, characterized in that, include: The audio and video data to be processed and the query information of the target to be identified in the audio and video data are obtained, and the audio and video data are sampled in time sequence to obtain video frame sequence and audio data; The query information, the video frame sequence, and the audio data are input into a preset video recognition model to extract text features, visual features, and audio features and fuse them to obtain fused features; Based on the fusion features, the event temporal localization result and the object spatial localization result are obtained as multiple candidate recognition clues for the audio and video data; The type of the target to be identified is obtained based on the query information, and multiple candidate identification clues are classified into identification instances according to the type of the target to obtain a set of identification instances; The video recognition result of the audio and video data is output based on the set of recognition instances.

2. The video recognition method based on cue localization as described in claim 1, characterized in that, The step of performing time-series sampling on the audio and video data to obtain video frame sequences and audio data specifically involves: The video segments of the audio and video data are segmented to obtain an initial video frame sequence; Multiple target video frames are extracted from the initial video frame sequence according to a preset sampling interval, and the timestamp of each target video frame is recorded to obtain a video frame sequence; Audio data is obtained by extracting audio segments synchronized with each of the target video frames from the audio track of the audio and video data based on the timestamp.

3. The video recognition method based on cue localization as described in claim 1, characterized in that, The training process of the video recognition model includes: Obtain a preset recognition training dataset, and perform supervised fine-tuning of the initial model based on the recognition training dataset to obtain the first training model; The recognition training dataset is grouped according to the difficulty of the training task, and the first training model is subjected to group reinforcement learning by combining the grouping relative policy optimization algorithm to obtain the second training model. The recognition training dataset is sampled evenly according to the training task type to obtain a sampled training set. The second training model is then subjected to full-task reinforcement learning based on the sampled training set to obtain a trained video recognition model.

4. The video recognition method based on cue localization as described in claim 3, characterized in that, After obtaining the trained video recognition model, the following is also included: Obtain a recognition test dataset containing multiple audio and video test samples and manually labeled data; The recognition test dataset is input into the trained video recognition model to obtain the predicted recognition results of multiple audio and video test samples; wherein, the predicted recognition results include the predicted occurrence frequency and predicted location clues of multiple recognition targets in multiple audio and video test samples; Based on the predicted frequency of occurrence and the actual frequency of occurrence of the manually labeled data, the evaluation index of the recognition result of the trained video recognition model is calculated. Based on the predicted location clues and the manually labeled actual location clues, calculate the recognition process evaluation index of the trained video recognition model; The trained video recognition model is updated based on the recognition result evaluation index and the recognition process evaluation index to obtain an updated video recognition model, which is then used to perform video recognition on the audio and video data to be processed.

5. The video recognition method based on cue localization as described in claim 1, characterized in that, The step of obtaining event temporal localization results and object spatial localization results based on the fusion features, and using them as multiple candidate recognition clues for the audio and video data, specifically involves: Temporal boundary detection is performed on the fusion features to obtain each recognition event of the audio and video data and the start timestamp and end timestamp corresponding to each recognition event, so as to use the start timestamp and end timestamp as the temporal localization results of multiple events of the audio and video data; The fused features are used to detect objects and obtain the bounding box coordinates of each identified object in the audio and video data, so as to use the bounding box coordinates as the spatial localization results of multiple objects in the audio and video data. The event time-series localization results and object spatial localization results are verified, and the verified event time-series localization results and object spatial localization results are used as multiple candidate recognition clues for the audio and video data.

6. The video recognition method based on cue localization as described in claim 1, characterized in that, The step of classifying multiple candidate identification clues into identification instances based on the type of the identification target to obtain an identification instance set is specifically as follows: When the target to be identified is an event type, the time intervals of the same identification event among multiple candidate identification clues are aggregated into the same event instance to obtain several event instances; When the target is identified as an object, cross-frame association matching is performed on the identified objects in each of the candidate identification clues, and the detection results of the same identified object instance are merged into the same object instance to obtain several object instances. When the target to be identified is an attribute type, multiple objects in each of the candidate identification clues are spatially located, and the detection results of multiple objects are clustered and grouped according to the attribute category of the query information to obtain several attribute instances. A set of identified instances is obtained based on several event instances, several object instances, and several attribute instances.

7. The video recognition method based on cue localization as described in claim 1, characterized in that, The step of outputting the video recognition result of the audio and video data based on the set of recognition instances specifically includes: The recognition results of each recognition target in the audio and video data are output based on multiple recognition instances in the recognition instance set, and are used as video recognition results. Specifically, when the recognition target is an event, the occurrence frequency and time information of the corresponding event instance are output; when the recognition target is an object, the occurrence frequency of the corresponding object instance and the time and spatial information of its first appearance are output; when the recognition target is an attribute category, the occurrence frequency of the corresponding attribute instance and the spatial information and attribute category identifier of each object instance under each attribute instance are output.

8. A video recognition system based on clue localization, characterized in that, include: The module includes an initialization processing module, a fusion feature generation module, a candidate recognition clue acquisition module, a recognition instance classification module, and a video recognition result acquisition module. The initialization processing module is used to acquire the audio and video data to be processed and the query information of the target to be identified in the audio and video data, and to perform time-series sampling on the audio and video data to obtain video frame sequences and audio data; The fusion feature generation module is used to input the query information, the video frame sequence, and the audio data into a preset video recognition model to extract text features, visual features, and audio features and fuse them to obtain fusion features; The candidate identification clue acquisition module is used to acquire event temporal positioning results and object spatial positioning results based on the fusion features, so as to serve as multiple candidate identification clues for the audio and video data; The identification instance classification module is used to obtain the type of the identification target based on the query information, and classify multiple candidate identification clues into identification instances based on the type of the identification target to obtain a set of identification instances; The video recognition result acquisition module is used to output the video recognition result of the audio and video data based on the recognition instance set.

9. A terminal device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the video recognition method based on cue localization as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, include: A stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform the video recognition method based on cue localization as described in any one of claims 1-7.

Images