An intelligent video analysis system based on multi-modal AI

Abstract

This invention discloses an intelligent video analysis system based on multimodal AI, belonging to the field of video analysis technology. It includes: a video preprocessing module, which extracts analysis frames from the original video segment; an IFS and Tracking analysis module, which performs cross-frame tracking of the analysis frames and filters important frames; a task orchestration and collaboration module, which relies on Redis to achieve state control and progress quantification; if an analysis frame contains a package or a person, it triggers the corresponding module to perform package logo or face recognition; a logo recognition module, which filters the optimal frame from the analysis frames containing packages and determines the package logo type by combining object detection and GPT dual recognition; a face recognition module, which performs face detection on the analysis frames containing people and identifies the person; and a structured video summarization generation module, which generates a structured summary of the original video segment. This invention can efficiently process videos, improve the accuracy of target recognition in videos, and facilitate subsequent retrieval and searching.

Description

Technical Field

[0001] This invention belongs to the field of video analysis technology, specifically relating to an intelligent video analysis system based on multimodal AI. Background Technology

[0002] Current video analytics in access control systems often falls short of meeting the demands for intelligence, efficiency, and accuracy in practical applications, primarily due to the following shortcomings:

[0003] (1) Low processing efficiency: Existing video analysis often uses uniform frame extraction or simple difference method, which requires traversing all video frames, generating a large number of invalid calculations, resulting in resource waste and response delay, and cannot meet the requirements of real-time access control.

[0004] (2) Single analysis strategy: It relies on a single model or full-frame analysis and lacks windowed filtering strategy; the calling cost is high, the function control is inflexible, and there is a lack of clear confidence fusion method, which affects the recognition accuracy.

[0005] (3) Insufficient module collaboration: Each functional module operates independently, data flow is not smooth, there is a lack of unified status management and progress tracking, and data reuse and process automation cannot be achieved. Summary of the Invention

[0006] This invention provides an intelligent video analysis system based on multimodal AI, which can efficiently process videos, improve the recognition accuracy of targets in videos, and facilitate subsequent retrieval and searching.

[0007] This invention provides the following technical solution:

[0008] A multimodal AI-based intelligent video analysis system includes:

[0009] The video preprocessing module is used to acquire the original video segment to be analyzed and extract the analysis frames of the original video segment;

[0010] The IFS and Tracking analysis modules are used to perform target detection and cross-frame tracking on each analysis frame, filter important frames in the analysis frames, build an important frame list, and combine the important frame list, the analysis frame list, and the cross-frame tracking results to generate an analysis file.

[0011] The task orchestration and collaboration module is used to achieve state control and progress quantification based on Redis, and to determine the type of all detection targets in the analysis frame based on the analysis file stored in Redis. If the analysis frame shows that the detection target is a package or a person, the corresponding module is triggered to perform package logo recognition or face recognition.

[0012] The logo recognition module is used to group and filter the best frames from the analysis frames that show packages. It combines object detection and GPT dual recognition with conflict handling to determine the logo type of each package and finally integrates and outputs the logo recognition result file.

[0013] The face recognition module is used to perform face detection on the analysis frames in which people appear, in order to identify the person's identity, and combine the identified person's identity with the face image to generate a person file;

[0014] The structured video summarization module is used to generate structured summaries of original video segments based on analysis files, logo recognition result files, and person files.

[0015] Optionally, the video preprocessing module is used to acquire the original video segment to be analyzed and extract the analysis frames of the original video segment, specifically including:

[0016] Name the original video segments according to the set format and create a structured directory to save each file;

[0017] The original video segment is preprocessed and the analysis frames are extracted according to the configured frame interval. The sequence number and timestamp of the analysis frames are recorded.

[0018] Optionally, the IFS and Tracking analysis module is used to perform target detection and cross-frame tracking on each analysis frame, and to filter important frames in the analysis frames to construct an important frame list, specifically including:

[0019] The first target detection model is used to detect targets in each analysis frame, and the type of the detected target, the size of the detection box, the center coordinates, and the confidence score are output.

[0020] The IOU matching algorithm is used to calculate the overlap between the detection box in the current frame and the predicted historical trajectory box to construct the cost matrix, and a global allocation strategy that minimizes the cost matrix is ​​used for target association.

[0021] The VIT model is used to convert each analysis frame into a mathematical vector, and the importance score of each analysis frame is calculated according to the following formula. Analysis frames that exceed the set threshold are selected as important frames, and an important frame list is constructed.

[0022] ;

[0023] in, For the first The importance score of each analysis frame and These are the weighting coefficients for similarity and confidence, respectively. For the first Analyze the mathematical vectors of the frames. For the first Analysis frame and the first Analyze the cosine similarity of the mathematical vectors corresponding to the frames; This is the set maximum similarity value. For the first Analyze the average confidence of all detected targets in the frame. This is the maximum set confidence level.

[0024] Optionally, the logo recognition module is used to group and filter the optimal frames from the analysis frames containing packages, combine target detection and GPT dual recognition with conflict handling, determine the package logo type group by group, and finally integrate and output the logo recognition result file, specifically including:

[0025] All analysis frames containing the package will be grouped according to the set window.

[0026] For each group, the second target detection model is used to identify the logo region in each analysis frame within the group, and the detection box size, center coordinates and recognition confidence of each logo region are output.

[0027] Combining the detection box information of the identified logo area, the recognition confidence, and the package detection box information, the best analysis frame is selected from each group as the target frame for the current group, and the best logo area of ​​the target frame is output.

[0028] For each target frame, the GPT model is used to identify the logo type of the detection box area and the best logo area of ​​the package, and conflict resolution is performed on the two identification results to obtain the package logo type of the current target frame.

[0029] The logo type of each target frame output is combined to form a logo recognition result file.

[0030] Optionally, the step of selecting an optimal analysis frame from each group as the target frame for the current group and outputting the optimal logo area of ​​the target frame specifically includes:

[0031] For each analysis frame, the positions of all identified logo areas are analyzed sequentially. If the logo area overlaps with the package, it is taken as the package logo area. If the logo area overlaps with the person, it is taken as the person logo area. If the logo area overlaps with other targets, it is taken as the other target logo area.

[0032] If the current group has a package logo area, select the analysis frame containing the package logo area with the highest confidence as the target frame of the current group, and use the package logo area as the optimal logo area of ​​the target frame. If the current group does not have a package logo area, select the analysis frame containing the person logo area with the highest confidence as the target frame of the current group, and use the person logo area as the optimal logo area of ​​the target frame. If the current group does not have a person logo area, select the analysis frame containing any other target logo area with the highest confidence as the target frame of the current group, and use the other target logo area as the optimal logo area of ​​the target frame.

[0033] Optionally, for each target frame, the GPT model is used to identify the logo type of the detection box region and the optimal logo region of the package, and conflict resolution is performed on the two identification results to obtain the package logo type of the current target frame, specifically including:

[0034] The GPT model is used to identify the logo type in the detection box area and the best logo area of ​​the package respectively. The identification result of the package detection box area is used as the first identification type, and the identification result of the best logo area is used as the second identification type.

[0035] If the first identification type is unknown, then the second identification type is used as the package logo type of the current target frame; otherwise, the first identification type is used as the package logo type of the current target frame.

[0036] Optionally, the face recognition module is used to perform face detection on the analysis frame in which a person appears, in order to identify the person's identity, specifically including:

[0037] The InsightFace model is used to detect faces in the analysis frames where people appear, and the face bounding boxes and key points are extracted to form face feature vectors.

[0038] The system performs similarity matching between the facial feature vector and the known facial feature vectors in the known face database, and outputs the identity of the person after a successful match.

[0039] Optionally, it also includes a highlight video generation module, which is used to filter key frames from the list of important frames and generate a highlight video by overlapping and merging the video segments of each key frame after extraction.

[0040] Optionally, the highlight video generation module is used to select key frames from the list of important frames, and generate a highlight video by overlapping and merging the video segments of each key frame after extraction, specifically including:

[0041] Sort all important frames in the important frame list in descending order according to their importance scores, select the Top-K frames as key frames, and add them to the key frame set.

[0042] For each keyframe in the keyframe set, use the keyframe's timestamp. Centered on the original video segment, a fixed-length video clip is extracted. The time range of the video clip is as follows: ; To set the extraction time;

[0043] The following formula is used to determine whether adjacent video segments overlap. If they overlap, the two adjacent video segments are merged and used as a new video segment to check for overlap with other video segments until all video segments do not overlap. All non-overlapping video segments are then output as the highlighted video of the original video segment.

[0044] ;

[0045] ;

[0046] in, For video clips and video clips The overlap duration, and For video clips The start and end timestamps, and For video clips The start and end timestamps, To set the threshold for overlap duration.

[0047] Optionally, the highlight video generation module is further configured to correct the keyframe set, specifically:

[0048] The timestamp coverage of the current keyframe set is calculated using the following formula. If timestamp coverage If the set threshold is exceeded, the current keyframe set will not be processed; otherwise, a specified number of non-keyframes will be continuously selected from the sorted list of important frames and added to the keyframe set until the timestamp coverage exceeds the set threshold.

[0049] ;

[0050] in, The total duration of the original video segment. and These are the maximum and minimum timestamps in the keyframe set, respectively.

[0051] Compared with the prior art, the beneficial effects of the present invention are:

[0052] This invention, by extracting and analyzing frames and filtering important frames, can eliminate a large amount of redundant footage, processing only frames that retain key information. This significantly reduces the computational load on subsequent modules and improves processing speed. For logo recognition on packages, this invention constructs a dual verification mechanism. Target detection can quickly locate candidate regions, while the GPT model uses its built-in knowledge base for secondary identification, quickly outputting the specific logo type. Furthermore, a conflict handling mechanism effectively avoids false alarms. This complementarity of visual and semantic capabilities significantly improves recognition accuracy in complex scenarios. In addition, this invention relies on Redis for real-time task status management and progress quantification, combined with task orchestration and collaboration modules to coordinate the collaborative work of various modules. This gives the system high observability and fault tolerance, ensuring the stable and efficient operation of large-scale video analysis tasks. This invention generates analysis files, logo recognition result files, person files, and structured summaries of the original video segments, transforming the original video into searchable structured data assets for easy user retrieval. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the structure of the intelligent video analysis system based on multimodal AI of the present invention;

[0054] Figure 2 This is a flowchart of the IFS and Tracking analysis process of this invention;

[0055] Figure 3 This is a flowchart of the logo recognition process of this invention;

[0056] Figure 4 This is a flowchart of the face recognition process of the present invention;

[0057] Figure 5 This is a flowchart of the process for generating structured video summaries according to the present invention;

[0058] Figure 6 This is a flowchart illustrating the process of generating a highlighted video according to the present invention. Detailed Implementation

[0059] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be used to limit the scope of protection of the present invention. It should be noted that the term "comprising" and any variations thereof in the specification, claims and the above-mentioned drawings of the present invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or devices.

[0060] like Figure 1 As shown, a multimodal AI-based intelligent video analysis system is provided for use in access control systems. The system includes: a video preprocessing module, an IFS and Tracking analysis module, a task orchestration and collaboration module, a logo recognition module, a face recognition module, a structured video summarization generation module, and a highlight video generation module.

[0061] I. Video Preprocessing Module.

[0062] Used to acquire the original video segment to be analyzed and extract the analysis frames from the original video segment.

[0063] The input video can be from multiple sources. Specifically, it supports various input formats such as local paths and HTTP / HTTPS links. Local videos are directly transferred to the specified location, while remote videos are obtained through a network download mechanism. If a download error occurs, the Redis status is updated to "error" and logged. If the original video segment is too long, it needs to be segmented to obtain analyzable segments. The segmentation method follows existing technologies. For example, if the input video comes directly from a real-time monitoring video stream, the original video segments can be obtained directly based on the preset sampling time.

[0064] Extracting the analysis frames from the original video segment includes the following sub-steps:

[0065] S1.1: Creating the structure directory.

[0066] Name the original video segments according to the set format and create a structured directory to save each file.

[0067] If the input filename does not conform to the “renamed_xx_xx.mp4” specification, a unique new name will be generated by combining the system timestamp and the first few bits of the original file's MD5 hash. For example, the original file “video_sample.mp4” will be renamed to “renamed_1720000000_b7c8e9f0.mp4” under timestamp 1720000000.

[0068] Create an independent root directory for each original video segment ID, with subfolders such as analysis, faces, package, summary, and highlight.

[0069] S1.2: Analyze frame extraction.

[0070] The original video segment is preprocessed and the analysis frames are extracted according to the configured frame interval. The sequence number and timestamp of the analysis frames are recorded.

[0071] The preprocessing method follows existing technology. Based on the configured frame interval N, frames 0, N, 2N, 3N, etc., are extracted sequentially from the beginning of the original video segment until the entire length of the original video segment is covered, and the sequence number and timestamp of the extracted analysis frame are recorded.

[0072] II. IFS and Tracking Analysis Module.

[0073] This tool is used to perform target detection and cross-frame tracking on each analysis frame, filter important frames in the analysis frame, construct an important frame list, and combine the important frame list, the analysis frame list, and the cross-frame tracking results to generate an analysis file.

[0074] like Figure 2 As shown, in this embodiment, filtering and analyzing important frames specifically includes the following steps:

[0075] S2.1: Target detection.

[0076] The first target detection model is used to detect targets in each analysis frame, and the type of the detected target, the size of the detection box, the center coordinates, and the confidence level are output.

[0077] The first object detection model is usually one of the YOLO series models. The YOLO series models are a collection of models used for real-time object detection, including YOLOv1 to YOLOv12 models.

[0078] The types of objects detected include packages, people, and other objects. The first object detection model will output the type, bounding box size, center coordinates, and confidence score for each detected object.

[0079] S2.2: Cross-frame tracking.

[0080] The IOU matching algorithm is used to calculate the overlap between the current frame detection box and the historical trajectory prediction box to construct the cost matrix, and a global allocation strategy that minimizes the cost matrix is ​​used for target association.

[0081] Each detected target is assigned a unique ID. The method for calculating the overlap between the current frame detection box and the historical trajectory prediction box using the IOU matching algorithm is based on existing technologies. The constructed cost matrix formula is as follows:

[0082] ;

[0083] in, Detect target for the current frame With existing trajectory The cost matrix, This is the weighting coefficient for Interaction-to-Unit (IOU). For detection target With existing trajectory Interaction ratio (IOU) The weighting coefficients for the Euclidean distance to the target center point. and These are the center point coordinates of the target detection box and the trajectory prediction box, respectively. For the Euclidean norm, This is a normalization factor used to scale the Euclidean distance of the center point to the same dimension as the IOU term.

[0084] The global allocation strategy for minimizing the cost matrix is ​​based on existing techniques, such as the Hungarian algorithm.

[0085] S2.3: Important frame filtering.

[0086] The VIT model is used to convert each analysis frame into a mathematical vector, and the importance score of each analysis frame is calculated according to the following formula. Analysis frames that exceed the set threshold are selected as important frames, and an important frame list is constructed.

[0087] ;

[0088] in, For the first The importance score of each analysis frame and These are the weighting coefficients for similarity and confidence, respectively. For the first transformation through the VIT model Analyze the mathematical vectors of the frames. For the first Analysis frame and the first Analyze the cosine similarity of the mathematical vectors corresponding to the frames; This is the set maximum similarity value. For the first Analyze the average confidence of all detected targets in the frame. This is the maximum set confidence level.

[0089] The VIT (Vision Transformer) model encodes image patches for each analysis frame and processes the sequence of image patches in the input analysis frame through a multi-head self-attention mechanism to capture intra-frame feature dependencies.

[0090] S2.4: Analysis file generation.

[0091] The important frame list, the analysis frame list, and cross-frame tracking results are combined to generate an analysis file, which is then placed into the analysis subfile.

[0092] III. Task orchestration and collaboration module.

[0093] It is used to implement state control and progress quantification based on Redis, and to determine the type of all detection targets in the analysis frame based on the analysis file stored in Redis. If the analysis frame shows that the detection target is a package or a person, the corresponding module is triggered to perform package logo recognition or face recognition.

[0094] If the analysis frame detects a package as the target, the logo recognition module is triggered to identify the package logo. If the analysis frame detects a person as the target, the face recognition module is triggered to perform face recognition. Relying on Redis, the system achieves real-time management and progress quantification of task status. Combined with the task orchestration and collaboration module, the system coordinates the collaborative work of various modules, giving it high observability and fault tolerance, and ensuring the stable and efficient operation of large-scale video analysis tasks.

[0095] IV. Logo Recognition Module.

[0096] This is used to group and filter the best frames from the analysis frames that contain packages. It combines object detection and GPT dual recognition with conflict handling to determine the package logo type for each group, and finally integrates and outputs the logo recognition result file.

[0097] like Figure 3 As shown, package logo recognition specifically includes the following steps:

[0098] S4.1: Analyze frame groups.

[0099] All analysis frames containing the package will be grouped according to the set window. The grouping formula is as follows:

[0100] ;

[0101] in, For the first Groups, For the size of the grouped windows, Indicates the first A Boolean function to determine whether each analysis frame contains a wrapper.

[0102] S4.2: Logo area recognition.

[0103] For each group, the second target detection model is used to identify the logo region in each analysis frame within the group, and the detection box size, center coordinates and recognition confidence of each logo region are output.

[0104] The second object detection model is usually one of the YOLO series models.

[0105] S4.3: Target frame filtering.

[0106] By combining the detection frame information of the identified logo area, the recognition confidence, and the package detection frame information, the best analysis frame is selected from each group as the target frame for the current group, and the best logo area of ​​the target frame is output.

[0107] S4.3.1: Logo area location analysis.

[0108] For each analysis frame, the positions of all identified logo areas are analyzed sequentially. If a logo area overlaps with a package, it is considered a package logo area. If a logo area overlaps with a person, it is considered a person logo area. If a logo area overlaps with other targets, it is considered the other target logo area.

[0109] Based on the detection box size and center coordinates of the logo area, and the detection box size and center coordinates of the package, it can be determined whether the logo area position overlaps with the package position. Similarly, it can be determined whether the logo area position overlaps with the person position, and whether the logo area position overlaps with other target positions.

[0110] S4.3.2: Determining the target frame and the optimal logo area.

[0111] The target frame and the optimal logo area are determined according to the priority order of the package logo area, the person logo area, and other target logo areas.

[0112] If the current group has a package logo area, select the analysis frame containing the package logo area with the highest confidence as the target frame of the current group, and use the package logo area as the optimal logo area of ​​the target frame. If the current group does not have a package logo area, select the analysis frame containing the person logo area with the highest confidence as the target frame of the current group, and use the person logo area as the optimal logo area of ​​the target frame. If the current group does not have a person logo area, select the analysis frame containing any other target logo area with the highest confidence as the target frame of the current group, and use the other target logo area as the optimal logo area of ​​the target frame.

[0113] S4.4: Logo type determined.

[0114] For each target frame, the GPT model is used to identify the logo type of the detection box area and the optimal logo area of ​​the package, and conflict resolution is performed on the two identification results to obtain the package logo type of the current target frame.

[0115] In this embodiment, the logo type can be a specific brand or a specific courier company, etc.; the recognition process of the GPT (Generative Pre-trained Transformer) model can refer to existing technologies.

[0116] The conflict handling process is as follows: The GPT model is used to identify the logo type of the detection box area of ​​the package and the optimal logo area respectively. The identification result of the package detection box area is taken as the first identification type, and the identification result of the optimal logo area is taken as the second identification type. If the first identification type is unknown, the second identification type is taken as the package logo type of the current target frame; otherwise, the first identification type is taken as the package logo type of the current target frame.

[0117] S4.5: Generate logo recognition result file.

[0118] Combine the logo types of each output target frame to form a logo recognition result file, and put it in the package subfolder.

[0119] V. Face recognition module.

[0120] This is used to perform face detection on analysis frames containing people to identify their identities, and then combine the identified identities with the face images to generate a person file.

[0121] like Figure 4 As shown, facial recognition specifically includes the following steps:

[0122] S5.1: Facial Feature Extraction.

[0123] The InsightFace model was used to detect faces in the analysis frames containing people, and the face bounding boxes and key points were extracted to form face feature vectors.

[0124] S5.2: Face matching.

[0125] The system performs similarity matching between the facial feature vector and the known facial feature vectors in the known face database, and outputs the identity of the person after a successful match.

[0126] The formula for calculating similarity is: , for and If the similarity score is greater than the set threshold, the match is successful; otherwise, the match fails and the identification result is an unknown person. The feature vector extracted for face detection. The feature vector of a known face.

[0127] S5.3: Character file generation.

[0128] The identified person's identity and the captured facial image are combined to generate a person file, which is then placed in the faces subfolder.

[0129] VI. Structured Video Summarization Generation Module.

[0130] Used to generate structured summaries of original video segments based on analysis files, logo recognition result files, and person files.

[0131] like Figure 5 As shown, the GPT model is used to register subject descriptions, and a summary file containing subject and object associations is constructed by combining timeline and detection information. The generation process of structured video summarization refers to existing technologies, and can be regarded as a sequence generation problem based on multimodal context. In this process, the GPT model is used in a given multimodal context. Under the condition of (formed by the fusion of analysis, package, and faces files), structured summaries are automatically generated by maximizing the conditional probability of the summary sequence. When generating structured summaries, the GPT model used can be the same as the GPT model used in step S4.4 to determine the logo type, but in practice, two independent GPT models are configured.

[0132] VII. Highlight Video Generation Module.

[0133] It is used to filter keyframes from the list of important frames, and generate a highlighted video by overlapping and merging the video segments after extracting each keyframe.

[0134] like Figure 6 As shown, the generation of highlighted video specifically includes the following steps:

[0135] S7.1: Construct a set of keyframes.

[0136] Sort all important frames in the important frame list in descending order according to their importance scores, select the Top-K frames as key frames, and add them to the key frame set.

[0137] The keyframe set was modified as follows:

[0138] The timestamp coverage of the current keyframe set is calculated using the following formula. If timestamp coverage If the set threshold is exceeded, the current keyframe set will not be processed; otherwise, a specified number of non-keyframes will be continuously selected from the sorted list of important frames and added to the keyframe set until the timestamp coverage exceeds the set threshold.

[0139] ;

[0140] in, The total duration of the original video segment. and These are the maximum and minimum timestamps in the keyframe set, respectively.

[0141] S7.2: Keyframe video segment extraction.

[0142] For each keyframe in the keyframe set, use the keyframe's timestamp. Centered on the original video segment, a fixed-length video clip is extracted. The time range of the video clip is as follows: ; To set the extraction time.

[0143] The value can be set, with a single segment duration of 2. Ensure that the fragment contains the core events corresponding to the keyframes.

[0144] S7.3: Merging overlapping segments.

[0145] The following formula is used to determine whether adjacent video segments overlap. If they overlap, the two adjacent video segments are merged and used as a new video segment to check for overlap with other video segments until all video segments do not overlap. All non-overlapping video segments are then output as the highlighted video of the original video segment.

[0146] ;

[0147] ;

[0148] in, For video clips and video clips The overlap duration, and For video clips The start and end timestamps, and For video clips The start and end timestamps, To set the threshold for overlap duration.

[0149] The method for merging two overlapping video segments follows existing technology. After all video segments are no longer overlapping, the merged segment time range is called using the FFmpeg tool (Fast Forward MPEG, a commonly used audio and video processing tool) to generate a highlighted video file, which is then placed in the highlight subfolder.

[0150] In some other embodiments, Redis is used to monitor the overall progress. Specifically, the core processing steps of the entire process can be divided into five key stages: IFS and Tracking analysis, package logo recognition, face detection, structured video summarization, and intelligent highlighted video generation. Each stage has equal weight, and the overall progress is monitored. The following formula is used to quantify the calculation:

[0151] ;

[0152] in, This indicates the completion status of IFS and Tracking analysis. This indicates the completion status of logo recognition on the package. This represents the completion status of face detection. This indicates the completion status of structured video summarization. The corresponding completion status of intelligent highlight video generation , , , and Both 0 and 1 indicate completion. IFS and Tracking analysis, package logo recognition, face detection, structured video summarization, and intelligent highlight video generation can include only two states (0 or 1), or they can have more granular distinctions based on completion status. , , , and The range is The overall schedule range is .

[0153] Link and archive the package, analysis, faces, summary, and highlight files to ensure that the results of each module are traceable.

[0154] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Those skilled in the art will clearly understand that the technologies in the embodiments of this invention can be implemented using software and necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or certain parts of the embodiments of this invention.

[0155] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. An intelligent video analysis system based on multimodal AI, characterized in that, include: The video preprocessing module is used to acquire the original video segment to be analyzed and extract the analysis frames of the original video segment; The IFS and Tracking analysis modules are used to perform target detection and cross-frame tracking on each analysis frame, filter important frames in the analysis frames, build an important frame list, and combine the important frame list, the analysis frame list, and the cross-frame tracking results to generate an analysis file. The task orchestration and collaboration module is used to achieve state control and progress quantification based on Redis, and to determine the type of all detection targets in the analysis frame based on the analysis file stored in Redis. If the analysis frame shows that the detection target is a package or a person, the corresponding module is triggered to perform package logo recognition or face recognition. The logo recognition module is used to group and filter the best frames from the analysis frames that show packages. It combines object detection and GPT dual recognition with conflict handling to determine the logo type of each package and finally integrates and outputs the logo recognition result file. The face recognition module is used to perform face detection on the analysis frames in which people appear, in order to identify the person's identity, and combine the identified person's identity with the face image to generate a person file; The structured video summarization generation module is used to generate structured summaries of original video segments based on analysis files, logo recognition result files, and person files. The IFS and Tracking analysis module is used to perform target detection and cross-frame tracking on each analysis frame, and to filter important frames in the analysis frame to construct an important frame list, specifically including: The first target detection model is used to detect targets in each analysis frame, and the type of the detected target, the size of the detection box, the center coordinates, and the confidence score are output. The IOU matching algorithm is used to calculate the overlap between the detection box in the current frame and the predicted historical trajectory box to construct the cost matrix, and a global allocation strategy that minimizes the cost matrix is ​​used for target association. The VIT model is used to convert each analysis frame into a mathematical vector, and the importance score of each analysis frame is calculated according to the following formula. Analysis frames that exceed the set threshold are selected as important frames, and an important frame list is constructed. ； in, For the first The importance score of each analysis frame and These are the weighting coefficients for similarity and confidence, respectively. For the first Analyze the mathematical vectors of the frames. For the first Analysis frame and the first Analyze the cosine similarity of the mathematical vectors corresponding to the frames; The maximum set similarity value. For the first Analyze the average confidence of all detected targets in the frame. This is the maximum set confidence level; The logo recognition module is used to group and filter the optimal frames from the analysis frames containing packages, combining target detection and GPT dual recognition with conflict handling, to determine the package logo type group by group, and finally integrate and output the logo recognition result file, specifically including: All analysis frames containing the package will be grouped according to the set window. For each group, the second target detection model is used to identify the logo region in each analysis frame within the group, and the detection box size, center coordinates and recognition confidence of each logo region are output. Combining the detection box information of the identified logo area, the recognition confidence, and the package detection box information, the best analysis frame is selected from each group as the target frame for the current group, and the best logo area of ​​the target frame is output. For each target frame, the GPT model is used to identify the logo type of the detection box area and the best logo area of ​​the package, and conflict resolution is performed on the two identification results to obtain the package logo type of the current target frame. The logo type of each target frame output is combined to form a logo recognition result file.

2. The multi-modal AI based intelligent video analytics system, as claimed in claim 1, wherein, The video preprocessing module is used to acquire the original video segment to be analyzed and extract the analysis frames of the original video segment, specifically including: Name the original video segments according to the set format and create a structured directory to save each file; The original video segment is preprocessed and the analysis frames are extracted according to the configured frame interval. The sequence number and timestamp of the analysis frames are recorded. 3.The multi-modal Al based intelligent video analytics system of claim 1, wherein, The step of selecting the optimal analysis frame from each group as the target frame for the current group and outputting the optimal logo area of ​​the target frame specifically includes: For each analysis frame, the positions of all identified logo areas are analyzed sequentially. If the logo area overlaps with the package, it is taken as the package logo area. If the logo area overlaps with the person, it is taken as the person logo area. If the logo area overlaps with other targets, it is taken as the other target logo area. If the current group has a package logo area, select the analysis frame containing the package logo area with the highest confidence as the target frame of the current group, and use the package logo area as the optimal logo area of ​​the target frame. If the current group does not have a package logo area, select the analysis frame containing the person logo area with the highest confidence as the target frame of the current group, and use the person logo area as the optimal logo area of ​​the target frame. If the current group does not have a person logo area, select the analysis frame containing any other target logo area with the highest confidence as the target frame of the current group, and use the other target logo area as the optimal logo area of ​​the target frame.

4. The intelligent video analysis system based on multimodal AI according to claim 1, characterized in that, For each target frame, the GPT model is used to identify the logo type of the package's detection box region and the optimal logo region, respectively. Conflict resolution is performed between the two identification results to obtain the package logo type of the current target frame. Specifically, this includes: The GPT model is used to identify the logo type in the detection box area and the best logo area of ​​the package respectively. The identification result of the package detection box area is used as the first identification type, and the identification result of the best logo area is used as the second identification type. If the first identification type is unknown, then the second identification type is used as the package logo type of the current target frame; otherwise, the first identification type is used as the package logo type of the current target frame.

5. The multi-modal AI based intelligent video analytics system, as claimed in claim 1, wherein, The face recognition module is used to perform face detection on the analysis frames in which a person appears, in order to identify the person's identity, specifically including: The InsightFace model is used to detect faces in the analysis frames where people appear, and the face bounding boxes and key points are extracted to form face feature vectors. The system performs similarity matching between the facial feature vector and the known facial feature vectors in the known face database, and outputs the identity of the person after a successful match.

6. The multi-modal AI based intelligent video analytics system, as claimed in claim 1, wherein, It also includes a highlight video generation module, which is used to filter key frames from the list of important frames and generate highlight videos by overlapping and merging the video segments of each key frame after extraction.

7. The multi-modal AI based intelligent video analytics system according to claim 6, wherein, The highlighted video generation module is used to select key frames from the list of important frames, and generate a highlighted video by overlapping and merging the video segments of each key frame after extraction. Specifically, it includes: Sort all important frames in the important frame list in descending order according to their importance scores, select the Top-K frames as key frames, and add them to the key frame set. For each keyframe in the keyframe set, use the keyframe's timestamp. Centered on the original video segment, a fixed-length video clip is extracted. The time range of the video clip is as follows: ; To set the extraction time; The following formula is used to determine whether adjacent video segments overlap. If they overlap, the two adjacent video segments are merged and used as a new video segment to check for overlap with other video segments until all video segments do not overlap. All non-overlapping video segments are then output as the highlighted video of the original video segment. ； ； in, For video clips and video clips The overlap duration, and For video clips The start and end timestamps, and For video clips The start and end timestamps, To set the threshold for overlap duration.

8. The multi-modal AI based intelligent video analytics system, as claimed in claim 7, wherein, The highlighted video generation module is also used to correct the keyframe set, specifically: The timestamp coverage of the current keyframe set is calculated using the following formula. If timestamp coverage If the set threshold is exceeded, the current keyframe set will not be processed. Otherwise, a specified number of non-key frames are continuously selected from the sorted list of important frames and added to the key frame set until the timestamp coverage exceeds the set threshold. ； in, The total duration of the original video segment. and These are the maximum and minimum timestamps in the keyframe set, respectively.

Images