A multi-task AI video analysis detection method and system sharing a network mechanism

Abstract

The application relates to the technical field, in particular to a multi-task AI video analysis detection method and system of a shared network mechanism, the application effectively solves problems such as task conflict, model redundancy and inflexible reasoning by freezing CLIP-ResNet50 as a unified feature extractor, utilizing the structural advantages of a YOLOv8 detection head and combining an innovative MoE module (supporting temperature adjustment, intelligent routing, self-attention fusion and expert enhancement mechanism). The scheme is suitable for real application scenarios such as multi-task edge reasoning and video stream concurrent detection, has strong practical value and technical advancement, and secondly, the application does not adopt multi-label merging, but establishes an independent detection head for each task to avoid interference between tasks; meanwhile, the MoE module dynamically selects the detection head to realize task switching or sharing in the reasoning stage, so that the scheme is suitable for scenarios with strong task segmentation and obvious feature conflict.

Description

Technical Field

[0001] The present invention relates to the technical field, and in particular to a multi-task AI video analysis and detection method and system using a shared network mechanism. Background Art

[0002] In real-world scenarios like multi-camera, multi-scene intelligent surveillance and industrial safety testing, multiple tasks (such as helmet detection, work uniform recognition, and violation analysis) often require parallel or switchable detection. The traditional approach is to deploy a separate object detection model for each task, resulting in a significant waste of redundant computing resources and, in particular, low efficiency in inference on edge devices or with multiple video streams.

[0003] In the existing technology, a multi-task YOLO detection solution is generally adopted. For example, in the Ultralytics YOLOv5 / YOLOv8 framework, a common approach to implementing multi-task target detection is to merge multiple task categories into the same detection head and train a shared model. For example, the categories for detecting "safety helmets" and "work clothes" can be incorporated into the same YOLO model, and a unified detector can be trained using multi-label data. The implementation process of this method is: merge task categories → build a unified dataset → use the YOLO structure for training → directly predict all tasks simultaneously during the inference phase, but the disadvantage is that the conflict between category features of different tasks can easily interfere with the training effect. For example, work clothes and safety helmets have different spatial structures and semantic patterns. It is difficult for a unified detection head to take into account the characteristics of the tasks, which may lead to a decrease in detection accuracy, especially in the case of data imbalance or different task domains. In addition, this solution cannot achieve dynamic task scheduling during inference. All tasks are always output in parallel and lack task control capabilities. In addition, it is very troublesome to merge data from multiple tasks. For example, dataset A is a safety helmet and dataset B is a work uniform. If they are merged, all the parts about people, dataset A must mark the work uniforms, and dataset B must mark the safety helmets. The workload is very large, especially when merging more than 10 tasks, all datasets must be complete with all labels.

[0004] There are also hybrid expert network approaches that use general features and MoE. For example, approaches like Google's Switch Transformer or GShard propose using sparsely activated MoE modules to achieve efficient computation for large models. In image tasks, M6-MoE has also been applied to large-scale visual text matching. These technical solutions are implemented by constructing multiple sparse expert subnetworks, then using a gated network to select some activations, and then fusing the expert outputs to execute the task. However, these solutions are primarily used for classification, language modeling, or general image-text matching tasks, rather than object detection tasks with dynamic routing in the task detection head. Their complex structure and high computational resource requirements make them unsuitable for lightweight inference deployment. Furthermore, existing MoE solutions are often deeply coupled with the backbone network, with expert selection and backbone training performed simultaneously, making it difficult to independently schedule frozen pre-trained features. Furthermore, their complex structure and tight coupling make them typically used in highly complex models like the Transformer, making them unsuitable for deployment in lightweight object detection frameworks such as YOLOv8. Their expert modules are often deeply coupled to the backbone network, preventing the backbone from freezing and using pre-trained models, thus losing model transferability. More importantly, this type of solution lacks the expert scheduling capability of “task granularity” and is suitable for sparse distribution of features, but not for sending one feature into multiple parallel detection heads. Summary of the Invention

[0005] The purpose of the present invention is to provide a multi-task AI video analysis detection method and system with a shared network mechanism to solve the above-mentioned problems in the prior art.

[0006] The present invention is achieved through the following technical solutions:

[0007] A multi-task AI video analysis and detection method with a shared network mechanism, comprising:

[0008] Obtain images for use by multiple task detection heads, extract image features of the images through CLIP Backbone, and generate feature maps;

[0009] Establish a dynamic weight pool, output several feature map-based weights through the MoE module and the dynamic weight pool, train the MoE module based on the weights, obtain one of the optimal feature maps corresponding to the detection head as the target detection head, obtain the target feature map of the target detection head, and perform weighted summation of the losses of each detection head;

[0010] Based on YOLOv8, the target feature map output by the target detection head is used to generate a detection frame and corresponding label, and the detection frame and corresponding label are output as the final detection result.

[0011] Preferably, the extracting the image features of the image by CLIP Backbone includes:

[0012] Get the X dimension of the current image and perform normalization to obtain the standardized target image;

[0013] Build a ResNet50 neural network model, input the target image into the Backbone module of the ResNet50 neural network model, perform feature extraction, and obtain several feature maps represented by multi-dimensional features;

[0014] Select the first key feature map, the second key feature map and the third key feature map, input the third key feature map into the SPPF module for spatial pyramid pooling to obtain the target key feature map;

[0015] Upsample the target key feature map twice and fuse the corresponding Backbone features each time;

[0016] Stepwise downsampling from the first resolution to the third key feature map is used to generate multi-scale predictions and finally output three scales;

[0017] The Detect layer makes predictions for the three scales and outputs the prediction results.

[0018] Preferably, obtaining the X dimension of the current image includes:

[0019]

[0020] The normalization process includes:

[0021]

[0022] Where, is the X dimension of the current image, is the height of the image, is the width of the image, is the result of normalization processing, is the mean vector of each channel, is the standard deviation of each channel, is the set of real numbers.

[0023] Preferably, inputting the target image to the Backbone module of the ResNet50 neural network model for feature extraction comprises:

[0024]

[0025] The selecting of the first key feature graph, the second key feature graph, and the third key feature graph includes:

[0026]

[0027]

[0028]

[0029] Where, is the feature map output by Backbone, 、 and are the height, width and number of channels of the feature map, respectively. is the first key feature map, is the second key feature map, is the third key feature map;

[0030] Inputting the third key feature map into the SPPF module for spatial pyramid pooling includes:

[0031]

[0032] Where, is the target key feature map.

[0033] Preferably, the upsampling from the target key feature graph twice and fusing the corresponding Backbone features each time includes:

[0034]

[0035]

[0036]

[0037]

[0038] Where, To map the target key features The intermediate feature map after upsampling, represents the upsampling operation, for With the second key feature map The splicing result, Indicates the channel-wise splicing operation, It is the output feature map after the splicing result is processed by the intermediate convolution module. Indicates the use Convolutional module, is the convolution path function, for The upsampled feature map is used for subsequent fusion with the first key feature map.

[0039] Preferably, the stepwise downsampling from the first resolution to the third key feature map comprises:

[0040]

[0041]

[0042]

[0043]

[0044]

[0045]

[0046]

[0047]

[0048] Where, For the first key feature map The intermediate feature map obtained by downsampling, is a convolution operation with a stride for downsampling, for and The output feature map obtained after splicing and convolution fusion, for and The concatenated feature map, for The output after processing by the intermediate level C2f convolution module, To represent the convolution path function, For the general The feature map after downsampling, for and The result after splicing, is the final large target feature map output by the deep C2f convolution module, Its convolution function, is the convolution structure function for the small target detection path, for and The result after splicing;

[0049] The final output of three scales includes:

[0050]

[0051]

[0052]

[0053] In the formula, in the formula, is the output feature map for the first target detection, is the output feature map for the second target detection, It is the output feature map of the third target detection.

[0054] Preferably, the predicting of three scales by the Detect layer includes:

[0055]

[0056] in, For the final prediction result, is the target detection module, is the total number of all predicted anchor points at three scales, is the number of categories in the target detection task.

[0057] Preferably, the establishing of a dynamic weight pool includes:

[0058] Set up weight pools and build dynamic weighted models;

[0059]

[0060] Where, is the weight vector output by the MoE module, is a function that normalizes the scoring results of all experts so that the sum of the weights is 1. is the fully connected layer parameter matrix that maps weights to the expert space, is the first layer weight matrix of the input features, is the image semantic vector obtained by global average pooling of the feature map output by the backbone network, 、 are the corresponding bias terms, is the number of candidate detection heads (experts).

[0061] Preferably, the constructing of a dynamic weighted model includes:

[0062] Set a trainable temperature coefficient parameter , make adjustments and allocations;

[0063]

[0064] Where, The dynamic weight vector after introducing the temperature adjustment parameter is used to control the smoothness of the weight distribution between different experts. is a trainable temperature coefficient parameter, belonging to the positive real number domain ;

[0065] By changing the trainable temperature coefficient parameter ,adjust the degree of selecting multiple experts and perform hard selection;

[0066]

[0067] Where, is the dynamic weight vector, The index is The prediction result output by the detection head is is the final target detection output.

[0068] In a second aspect, the present invention also provides a multi-task AI video analysis and detection system with a shared network mechanism, comprising

[0069] an image acquisition module configured to acquire images for use by multiple task detection heads, extract image features of the images through the CLIP Backbone, and generate feature maps;

[0070] The selection module is configured to establish a dynamic weight pool, output several feature map-based weights through the MoE module and the dynamic weight pool, select the corresponding detection head of one of the feature maps as the target detection head based on the weights, obtain the target feature map, and perform weighted summation of the losses of each detection head;

[0071] The output module is configured to generate a detection frame and corresponding label for the target feature map output by the target detection head based on YOLOv8, and output the detection frame and corresponding label as the final detection result.

[0072] The technical solution of the present invention has at least the following advantages and beneficial effects:

[0073] This paper utilizes CLIP-ResNet50 as a unified feature extractor, leverages the structural advantages of the YOLOv8 detection head, and combines it with an innovative MoE module (supporting temperature regulation, intelligent routing, self-attention fusion, and expert enhancement mechanisms) to effectively address issues such as task conflict, model redundancy, and inflexible reasoning. This solution is suitable for real-world applications such as multi-task edge reasoning and concurrent video stream detection, demonstrating strong practical value and technological advancement.

[0074] Combining the visual perception capabilities of large models with the dynamic expert selection mechanism of MoE to achieve the sharing and scheduling of task detection heads is a key technical requirement in current multi-task target detection. Secondly, this application does not use multi-label merging, but instead establishes an independent detection head for each task to avoid interference between tasks. At the same time, the dynamic selection of detection heads by the MoE module enables task switching or sharing during the inference phase, making it suitable for scenarios with strong task segmentation and obvious feature conflicts. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0076] Figure 1 It is a schematic diagram of the process of the present invention;

[0077] Figure 2 Schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0078] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0079] The division of modules in this application is a logical division. In actual application, there may be other division methods. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0080] Independently described modules or submodules may or may not be physically separate; they may be implemented in software or hardware. Some modules or submodules may be implemented in software, with the processor invoking the software to implement the functionality of these modules or submodules, while other modules or submodules may be implemented in hardware, such as hardware circuits. Furthermore, some or all of the modules may be selected based on actual needs to achieve the objectives of the present application.

[0081] Please refer to Figure 1-Figure 2 The present invention provides a multi-task AI video analysis and detection method with a shared network mechanism, comprising:

[0082] S101: Acquire images for use by multiple task detection heads, extract image features of the images through CLIP Backbone, and generate feature maps;

[0083] The CLIP Backbone is specifically embodied in the image encoding block of CLIP, which performs comparative learning on a large number of image-text pairs and has rich feature extraction capabilities. Specifically, it is implemented as Resmet50. ResNet50 is a deep convolutional neural network architecture. Backbone refers to the core part of a deep neural network responsible for extracting multi-scale and multi-level features from the input image.

[0084] S102: Establish a dynamic weight pool, output several weights based on feature maps through the MoE module and the dynamic weight pool, train the MoE module based on the weights, obtain one of the optimal feature maps corresponding to the detection head as the target detection head, obtain the target feature map of the target detection head, and perform weighted summation on the losses of each detection head;

[0085] During training, there are two steps. Each detection head needs to calculate the loss and train each detection head. Then they are combined together and only the MoE part is trained. During inference, the weights output by the MoE module are used to select the optimal detection head from multiple detection heads.

[0086] S103: Generate a detection frame and corresponding label for the target feature map output by the target detection head based on YOLOv8, and output the detection frame and corresponding label as the final detection result.

[0087] This paper utilizes CLIP-ResNet50 as a unified feature extractor, leverages the structural advantages of the YOLOv8 detection head, and combines it with an innovative MoE module (supporting temperature regulation, intelligent routing, self-attention fusion, and expert enhancement mechanisms) to effectively address issues such as task conflict, model redundancy, and inflexible reasoning. This solution is suitable for real-world applications such as multi-task edge reasoning and concurrent video stream detection, demonstrating strong practical value and technological advancement.

[0088] Combining the visual perception capabilities of large models with the dynamic expert selection mechanism of MoE to achieve sharing and scheduling of task detection heads is a key technical requirement in current multi-task target detection. Secondly, this application does not use multi-label merging, but instead establishes an independent detection head for each task to avoid interference between tasks; at the same time, the MoE module dynamically selects the detection head to achieve task switching or sharing in the inference stage, making it suitable for scenarios with strong task segmentation and obvious feature conflicts.

[0089] In an exemplary embodiment of the present invention, extracting image features of the image using CLIP Backbone includes:

[0090] Get the X dimension of the current image and perform normalization to obtain the standardized target image;

[0091] Build a ResNet50 neural network model, input the target image into the Backbone module of the ResNet50 neural network model, perform feature extraction, and obtain several feature maps represented by multi-dimensional features;

[0092] Select the first key feature map, the second key feature map and the third key feature map, input the third key feature map into the SPPF module for spatial pyramid pooling to obtain the target key feature map;

[0093] Upsample the target key feature map twice and fuse the corresponding Backbone features each time;

[0094] Stepwise downsampling from the first resolution to the third key feature map is used to generate multi-scale predictions and finally output three scales;

[0095] The Detect layer makes predictions for the three scales and outputs the prediction results.

[0096] Specifically, obtaining the X dimension of the current image includes:

[0097]

[0098] The normalization process includes:

[0099]

[0100] Where, is the X dimension of the current image, is the height of the image, is the width of the image, is the result of normalization processing, is the mean vector of each channel, is the standard deviation of each channel, is the set of real numbers.

[0101] Secondly, the target image is input to the Backbone module of the ResNet50 neural network model for feature extraction, including:

[0102]

[0103] The selecting of the first key feature graph, the second key feature graph, and the third key feature graph includes:

[0104]

[0105]

[0106]

[0107] Where, is the feature map output by Backbone, 、 and are the height, width and number of channels of the feature map, respectively. is the first key feature map, is the second key feature map, is the third key feature map;

[0108] Inputting the third key feature map into the SPPF module for spatial pyramid pooling includes:

[0109]

[0110] Where, is the target key feature map.

[0111] Preferably, the upsampling from the target key feature graph twice and fusing the corresponding Backbone features each time includes:

[0112]

[0113]

[0114]

[0115]

[0116] Where, To map the target key features The intermediate feature map after upsampling, represents the upsampling operation, for With the second key feature map The splicing result, Indicates the channel-wise splicing operation, It is the output feature map after the splicing result is processed by the intermediate convolution module. Indicates the use Convolutional module, is the convolution path function, for The upsampled feature map is used for subsequent fusion with the first key feature map.

[0117] Specifically, the step of downsampling from the first resolution to the third key feature map, wherein the first resolution is a low resolution, and the feature map is a low resolution if it is smaller in spatial dimension (i.e., width × height) than the input image, includes:

[0118]

[0119]

[0120]

[0121]

[0122]

[0123]

[0124]

[0125]

[0126] Where, For the first key feature map The intermediate feature map obtained by downsampling, is a convolution operation with a stride for downsampling, for and The output feature map obtained after splicing and convolution fusion, for and The concatenated feature map, for The output after processing by the intermediate level C2f convolution module, To represent the convolution path function, For the general The feature map after downsampling, for and The result after splicing, is the final large target feature map output by the deep C2f convolution module, Its convolution function, is the convolution structure function for the small target detection path, for and The result after splicing;

[0127] The final output of three scales includes:

[0128]

[0129]

[0130]

[0131] In the formula, in the formula, is the output feature map for the first target detection, is the output feature map for the second target detection, It is the output feature map of the third target detection.

[0132] Among them, specifically, is the output feature map for small target detection, corresponding to the feature layer with higher resolution (such as ); is the output feature map for medium target detection, corresponding to the intermediate resolution feature layer (such as ); is the output feature map for large target detection, corresponding to the low-resolution but most semantically powerful feature layer (such as ). The three are input into the Detect layer together to generate the final target detection result.

[0133] Secondly, the prediction of three scales through the Detect layer includes:

[0134]

[0135] in, is the final prediction result, Detect() represents the target detection module, which is used to transform the multi-scale feature map Integrate and output candidate target boxes; is the total number of all predicted anchors at three scales, is the number of categories in the target detection task, output dimension The 4 in represents the position information of each box (center point coordinates and width and height).

[0136] It also includes using a multi-task dataset for training. Each task (such as work clothes, hard hats) has a label and requires the use of a corresponding detection head. The loss function for each task includes classification loss and regression loss:

[0137]

[0138] in, is the classification loss, is the box regression loss, Set the target box loss weight coefficient, load the CLIP pre-trained weights, and freeze the backbone. Initialize multiple YOLOv8 detection heads, making their parameters updateable during training. CLIP has strong feature representation capabilities after undergoing text and image encoding, and can be transferred to various tasks with zero-shot accuracy.

[0139] In an exemplary embodiment of the present invention, a traditional MoE network only selects an optimal expert (i.e., a detection head) through a gating network. To enhance the capabilities of the MoE, dynamic expert selection can be introduced, where multiple experts are dynamically selected and fused based on the input.

[0140] In this method, the gating network not only outputs the weight of each expert, but also considers the combination of multiple experts and selects different experts based on image features. In particular, a dynamic weight pool is introduced to adaptively adjust the weight of each expert.

[0141] The establishing of a dynamic weight pool comprises:

[0142] Set up weight pools and build dynamic weighted models;

[0143]

[0144] Where, The weight vector output by the MoE module; represents the function that normalizes the scoring results of all experts so that the sum of the weights is 1; is the fully connected layer parameter matrix that maps weights to the expert space, is the first layer weight matrix of the input features, is the image semantic vector obtained by global average pooling of the feature map output by the backbone network; 、 are the corresponding bias terms respectively; Indicates the number of candidate detection heads (experts).

[0145] Preferably, the constructing of a dynamic weighted model includes:

[0146] Set a trainable temperature coefficient parameter , make adjustments and allocations;

[0147]

[0148] Where, The dynamic weight vector after introducing the temperature adjustment parameter is used to control the smoothness of the weight distribution between different experts. is a trainable temperature coefficient parameter, belonging to the positive real number domain ;

[0149] By changing the training temperature coefficient parameter ,adjust the degree of selecting multiple experts and perform hard selection;

[0150]

[0151] Where, is the dynamic weight vector The detection head index corresponding to the maximum value in represents the optimal task branch for the current image; The index is The prediction result output by the detection head is used as the final target detection output .

[0152] It should be noted that the training is divided into two steps. The first step is to train each output head separately to obtain the final prediction result. The above steps are the second part, which is to combine the output results of multiple detection heads to obtain the final target detection output .

[0153] Based on the above solution, when it is actually used, the target detection head of YOLOv8 is responsible for generating detection boxes and category information from the feature map extracted by Backbone. Corresponding to a task (such as work clothes or helmet detection), its output is:

[0154]

[0155] in, is the number of anchor points for each task, For the task The number of categories.

[0156] The connection between the YOLOv8 detection head and the CLIP Backbone is completed by splicing the feature maps: Backbone output feature map After multiple convolution and pooling operations, it is finally fused with the output of the task detection head.

[0157] The connection between the detection head and the Backbone can be expressed as:

[0158]

[0159] Train the MoE branch and use the trained detection heads at the same time. The weights of the gating network output Determine which task's detection head should be used for the current input image. Perform a weighted summation of the losses of each task head:

[0160]

[0161] in, is the total loss, It is the regularization loss of the MoE network, controlling the balance of expert selection.

[0162] During inference, image input is extracted through Backbone MoE routing gated network based on Output weight , decide which detection head to use. Generate target box and category information.

[0163] The final test results are:

[0164] , use the selected detection head.

[0165] In a second aspect, the present invention also provides a multi-task AI video analysis and detection system with a shared network mechanism, comprising

[0166] an image acquisition module configured to acquire images for use by multiple task detection heads, extract image features of the images through the CLIP Backbone, and generate feature maps;

[0167] The selection module is configured to establish a dynamic weight pool, output several feature map-based weights through the MoE module and the dynamic weight pool, select the corresponding detection head of one of the feature maps as the target detection head based on the weights, obtain the target feature map, and perform weighted summation of the losses of each detection head;

[0168] The output module is configured to generate a detection frame and corresponding label for the target feature map output by the target detection head based on YOLOv8, and output the detection frame and corresponding label as the final detection result.

[0169] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0170] If the integrated unit is implemented as a software functional unit and sold or used as a standalone product, it can be stored on a computer-readable storage medium. This computer software product, stored on a storage medium, includes instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0171] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A multi-task AI video analysis and detection method with a shared network mechanism, characterized in that: include: Obtain images for use by multiple task detection heads, extract image features of the images through CLIP Backbone, and generate feature maps; Establish a dynamic weight pool, output several feature map-based weights through the MoE module and the dynamic weight pool, train the MoE module based on the weights, obtain one of the optimal feature maps corresponding to the detection head as the target detection head, obtain the target feature map of the target detection head, and perform weighted summation of the losses of each detection head; Generate detection frames and corresponding labels based on the target feature map output by the target detection head based on YOLOv8, and output the detection frames and corresponding labels as the final detection results; The establishing of a dynamic weight pool comprises: Set up weight pools and build dynamic weighted models; Where, is the weight vector output by the MoE module, is a function that normalizes the scoring results of all experts so that the sum of the weights is 1. is the fully connected layer parameter matrix that maps weights to the expert space, is the first layer weight matrix of the input features, is the image semantic vector obtained by global average pooling of the feature map output by the backbone network, 、 are the corresponding bias terms, is the number of candidate detection heads.

2. The multi-task AI video analysis and detection method based on a shared network mechanism according to claim 1 is characterized in that: Extracting the image features of the image by CLIP Backbone includes: Get the X dimension of the current image and perform normalization to obtain the standardized target image; Build a ResNet50 neural network model, input the target image into the Backbone module of the ResNet50 neural network model, perform feature extraction, and obtain several feature maps represented by multi-dimensional features; Select the first key feature map, the second key feature map and the third key feature map, input the third key feature map into the SPPF module for spatial pyramid pooling to obtain the target key feature map; Upsample the target key feature map twice and fuse the corresponding Backbone features each time; Stepwise downsampling from the first resolution to the third key feature map is used to generate multi-scale predictions and finally output three scales; The Detect layer makes predictions for the three scales and outputs the prediction results.

3. The multi-task AI video analysis and detection method of a shared network mechanism according to claim 2 is characterized in that: Obtaining the X dimension of the current image includes: The normalization process includes: Where, is the X dimension of the current image, is the height of the image, is the width of the image, is the result of normalization processing, is the mean vector of each channel, is the standard deviation of each channel, is the set of real numbers.

4. The multi-task AI video analysis and detection method of a shared network mechanism according to claim 3 is characterized in that: The target image is input to the Backbone module of the ResNet50 neural network model for feature extraction, which includes: The selecting of the first key feature graph, the second key feature graph, and the third key feature graph includes: Where, is the feature map output by Backbone, 、 and are the height, width and number of channels of the feature map, respectively. is the first key feature map, is the second key feature map, is the third key feature map; Inputting the third key feature map into the SPPF module for spatial pyramid pooling includes: Where, is the target key feature map.

5. The multi-task AI video analysis and detection method of a shared network mechanism according to claim 4 is characterized in that: The target key feature map is upsampled twice, and the corresponding Backbone features are fused each time, including: Where, To map the target key features The intermediate feature map after upsampling, represents the upsampling operation, for With the second key feature map The splicing result, Indicates the channel-wise splicing operation, It is the output feature map after the splicing result is processed by the intermediate convolution module. Indicates the use Convolutional module, is the first convolution path function, for The upsampled feature map is used for subsequent fusion with the first key feature map.

6. The multi-task AI video analysis and detection method with a shared network mechanism according to claim 5 is characterized in that: The stepwise downsampling from the first resolution to the third key feature map comprises: Where, for and The result after splicing, for and The output feature map obtained after splicing and convolution fusion, is the convolutional structure function used in the target detection path, For The intermediate feature map obtained by downsampling, is a convolution operation with a stride for downsampling, for and The concatenated feature map, for The output after processing by the C2f convolution module, is the second convolution path function, For the general The feature map after downsampling, for and The result after splicing, is the final target feature map output by the C2f convolution module, is the convolution function; The final output of three scales includes: Where, is the output feature map for the first target detection, is the output feature map for the second target detection, It is the output feature map of the third target detection.

7. The multi-task AI video analysis and detection method with a shared network mechanism according to claim 6 is characterized in that: The prediction of three scales through the Detect layer includes: in, For the final prediction result, is the target detection module, is the total number of all predicted anchor points at three scales, is the number of categories in the target detection task.

8. The multi-task AI video analysis and detection method with a shared network mechanism according to claim 7 is characterized in that: The construction of the dynamic weighted model includes: Set a trainable temperature coefficient parameter , make adjustments and allocations; Where, The dynamic weight vector after introducing the temperature adjustment parameter is used to control the smoothness of the weight distribution between different experts. is a trainable temperature coefficient parameter, belonging to the positive real number domain ; By changing the trainable temperature coefficient parameter ,adjust the degree of selecting multiple experts and perform hard selection; Where, is the dynamic weight vector, The index is The prediction result output by the detection head is is the final target detection output.

9. A multi-task AI video analysis and detection system with a shared network mechanism, characterized in that: include an image acquisition module configured to acquire images for use by multiple task detection heads, extract image features of the images through the CLIP Backbone, and generate feature maps; The selection module is configured to establish a dynamic weight pool, output a plurality of weights based on feature maps through the MoE module and the dynamic weight pool, select a detection head corresponding to one of the feature maps as a target detection head based on the weights, obtain a target feature map, and perform weighted summation on the losses of each detection head, wherein establishing the dynamic weight pool includes: setting the weight pool and constructing a dynamic weighted model; Where, is the weight vector output by the MoE module, is a function that normalizes the scoring results of all experts so that the sum of the weights is 1. is the fully connected layer parameter matrix that maps weights to the expert space, is the first layer weight matrix of the input features, is the image semantic vector obtained by global average pooling of the feature map output by the backbone network, 、 are the corresponding bias terms, is the number of candidate detection heads; The output module is configured to generate a detection frame and corresponding label for the target feature map output by the target detection head based on YOLOv8, and output the detection frame and corresponding label as the final detection result.

Images