Long tail target detection method and device based on visual association scene and computer equipment

Abstract

The application relates to a long-tail target detection method and device based on a visual association scene and computer equipment, wherein the method comprises the following steps: acquiring each frame of image and a corresponding scene label; inputting each frame of image and the corresponding scene label into a pre-trained detection model to obtain a detection result; the detection model comprises a feature extraction module, a feature fusion module and a dynamic feature pyramid network; the feature extraction module is used for performing feature extraction on each frame of image to obtain general features and scene features; the scene features are obtained by processing a scene-specific feature head in the feature extraction module; the scene-specific feature head corresponds to a scene type of the visual association scene; the feature fusion module is used for fusing the general features and the scene features to obtain enhanced features; and the dynamic feature pyramid network is used for detecting the enhanced features to obtain the detection result. Through the application, the problems of low long-tail target detection accuracy and insufficient generalization ability of a model in the related art are solved.

Description

Technical Field

[0001] This application relates to the field of visual network technology, and in particular to a method, apparatus and computer equipment for long-tail target detection based on visual network scenes. Background Technology

[0002] With the rapid development of video network technology, fast and accurate long-tail object detection from massive amounts of video data plays a crucial role in scene analysis and decision support in fields such as smart cities, public safety, and industrial manufacturing. An efficient and stable long-tail object detection technology for video network scenarios will directly impact the accuracy and generalization ability of video content understanding.

[0003] Current methods for long-tail object detection involve training a model on a general dataset and then using 3D convolutional feature extraction strategies and traditional evaluation metrics to detect long-tail objects in various visual association scenes. The drawback of this approach is that, due to significant differences in target features across different scenes and the low proportion of long-tail scene samples, general feature extraction strategies cannot capture scene-specific information, resulting in low accuracy in long-tail object detection and insufficient model generalization ability.

[0004] There are currently no effective solutions to the problems of low accuracy in long-tail target detection and insufficient generalization ability of the models in related technologies. Summary of the Invention

[0005] This embodiment provides a method, apparatus, computer device, and storage medium for long-tail target detection based on visual scenes, in order to solve the problems of low accuracy and insufficient generalization ability of long-tail target detection in related technologies.

[0006] Firstly, this embodiment provides a long-tail target detection method based on visual association scenes, including:

[0007] Obtain each frame image and its corresponding scene label;

[0008] Each frame of the image and its corresponding scene label are input into a pre-trained detection model to obtain the detection result;

[0009] The detection model includes a feature extraction module, a feature fusion module, and a dynamic feature pyramid network;

[0010] The feature extraction module is used to extract features from each frame of the image to obtain general features and scene features; the scene features are obtained by processing a scene-specific feature head in the feature extraction module; the scene-specific feature head corresponds to the scene type of the visual association scene;

[0011] The feature fusion module is used to fuse the general features and the scene features to obtain enhanced features;

[0012] The dynamic feature pyramid network is used to detect the enhanced features and obtain the detection results.

[0013] In some embodiments, the feature extraction module further includes a basic feature extraction network;

[0014] The basic feature extraction network is used to extract features from each frame of the image to obtain the general features.

[0015] In some of these embodiments, the scene-specific feature headers include kitchen scene feature headers, transportation scene feature headers, security scene feature headers, and industrial scene feature headers;

[0016] The kitchen scene feature heads include a chef's hat detector, a mask detector, a smoking detector, and a trash can detector;

[0017] The traffic scene feature header includes vehicle detectors, pedestrian detectors, and traffic sign detectors;

[0018] The security scene feature header includes an intrusion detector, a loitering detector, and a fight detector;

[0019] The industrial scenario feature header includes equipment operation detectors, safety violation detectors, and equipment fault detectors.

[0020] In some embodiments, the method further includes: in the feature fusion module, fusing the enhanced features based on dynamically adjusted fusion weights, the general features, and the scene features; the dynamically adjusted fusion weights are determined by the scene type.

[0021] In some of these embodiments, the dynamic feature pyramid network includes a feature pyramid network, a scene gating network, and a detection head network;

[0022] The feature pyramid network is used to extract multi-scale features based on the enhanced features;

[0023] The scene gating network is used to adaptively adjust the multi-scale features to obtain gating features;

[0024] The detection head network is used to detect the gating features and obtain the detection results.

[0025] In some embodiments, the method further includes optimizing the gating features based on a scene-specific attention mechanism after obtaining the gating features.

[0026] In some embodiments, the method further includes: constructing a dedicated evaluation system for the visual association scene based on the video data of the visual association scene and the corresponding scene category set;

[0027] The test results are evaluated based on the dedicated evaluation system.

[0028] Secondly, this embodiment provides a long-tail target detection device based on visual association scene, including: a processing module and a detection module;

[0029] The processing module is used to acquire each frame of image and its corresponding scene label;

[0030] The detection module is used to input each frame of the image and the corresponding scene label into a pre-trained detection model to obtain the detection result;

[0031] The detection model includes a feature extraction module, a feature fusion module, and a dynamic feature pyramid network;

[0032] The feature extraction module is used to extract features from each frame of the image to obtain general features and scene features; the scene features are obtained by processing a scene-specific feature head in the feature extraction module; the scene-specific feature head corresponds to the scene type of the visual association scene;

[0033] The feature fusion module is used to fuse the general features and the scene features to obtain enhanced features;

[0034] The dynamic feature pyramid network is used to detect the enhanced features and obtain the detection results.

[0035] Thirdly, this embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the long-tail target detection method based on visual scenes described in the first aspect above.

[0036] Fourthly, this embodiment provides a storage medium storing a computer program that, when executed by a processor, implements the long-tail target detection method based on visual scenes described in the first aspect above.

[0037] Compared with related technologies, the long-tail target detection method, apparatus, and computer equipment based on visual association scenarios provided in this embodiment acquire each frame of image and its corresponding scene label; input each frame of image and its corresponding scene label into a pre-trained detection model to obtain detection results; the detection model includes a feature extraction module, a feature fusion module, and a dynamic feature pyramid network; the feature extraction module is used to extract features from each frame of image to obtain general features and scene features; the scene features are processed by scene-specific feature heads in the feature extraction module; the scene-specific feature heads correspond to the scene type of the visual association scenario; the feature fusion module is used to fuse general features and scene features to obtain enhanced features; the dynamic feature pyramid network is used to detect the enhanced features to obtain detection results. This solves the problems of low accuracy and insufficient generalization ability of long-tail target detection in related technologies. By configuring scene-specific feature heads for different scenarios and extracting scene features, it improves the accuracy of long-tail target detection and enhances the generalization ability of the model.

[0038] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0039] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0040] Figure 1 This is a hardware structure block diagram of a terminal device for a long-tail target detection method based on visual scenes provided in an embodiment of this application;

[0041] Figure 2 This is a flowchart of a long-tail target detection method based on visual association scene provided in an embodiment of this application;

[0042] Figure 3 yes Figure 2 Flowchart of the dynamic feature pyramid network in step S220;

[0043] Figure 4 This is a schematic diagram of the process for evaluating test results using a dedicated evaluation system, provided in one embodiment of this application.

[0044] Figure 5 This is a structural block diagram of a long-tail target detection method based on visual scenes provided in an embodiment of this application.

[0045] In the diagram: 102, processor; 104, memory; 106, transmission device; 108, input / output device; 210, processing module; 220, detection module. Detailed Implementation

[0046] To better understand the purpose, technical solution, and advantages of this application, the application is described and explained below in conjunction with the accompanying drawings and embodiments.

[0047] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0048] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal for the long-tail target detection method based on visual scenes in this embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0049] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the long-tail target detection method based on visual scenes in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0050] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0051] This embodiment provides a long-tail target detection method based on visual association scenes. Figure 2 This is a flowchart of the long-tail target detection method based on visual association scene in this embodiment, as follows: Figure 2 As shown, the process includes the following steps:

[0052] Step S210: Obtain each frame image and its corresponding scene label.

[0053] Specifically, in practical applications, the methods for obtaining images and scene tags include, but are not limited to, obtaining images and scene tags that meet the above requirements from a pre-stored database; downloading images and scene tags that meet the requirements from a network platform; or generating corresponding images and scene tags according to needs. This application embodiment does not limit the method of obtaining images and scene tags. It should be noted that each frame of an image has a corresponding scene tag.

[0054] For example, each frame of image data can be obtained by decomposing multi-scene video data collected by video network terminal devices. Scene tags refer to the scene type to which an image belongs. For example, an image obtained from a restaurant kitchen surveillance video is tagged as a kitchen scene, an image obtained from a road surveillance video is tagged as a traffic scene, an image obtained from a residential area surveillance video is tagged as a security scene, and an image obtained from a factory workshop surveillance video is tagged as an industrial scene.

[0055] Step S220: Input each frame image and its corresponding scene label into the pre-trained detection model to obtain the detection result. The detection model includes a feature extraction module, a feature fusion module, and a dynamic feature pyramid network. The feature extraction module is used to extract features from each frame image to obtain general features and scene features. The scene features are processed by the scene-specific feature head in the feature extraction module. The scene-specific feature head corresponds to the scene type of the visual scene. The feature fusion module is used to fuse the general features and scene features to obtain enhanced features. The dynamic feature pyramid network is used to detect the enhanced features to obtain the detection result.

[0056] Specifically, a pre-trained detection model refers to a dedicated model for long-tail object detection in visual-connected scenes, which is pre-trained and whose internal modules work together to complete feature processing and object detection. It consists of three parts: a feature extraction module, a feature fusion module, and a dynamic feature pyramid network.

[0057] The feature extraction module is the foundational feature processing module of the detection model. It extracts features from each frame of image data, focusing on representative key information that can be used for subsequent detection tasks, ultimately yielding general features and scene features. Scene features are targeted visual information reflecting the unique attributes of targets within a specific scene, suitable for personalized feature descriptions of corresponding scenes. For example, in a kitchen scene, the state of a chef's hat and the diffusion pattern of smoke; in a traffic scene, the trajectory of vehicles and the markings of traffic signs; in a security scene, intrusion actions and loitering patterns of personnel; and in an industrial scene, the texture of equipment malfunctions and the degree of operational standardization. Scene features are obtained by processing scene-specific feature heads in the feature extraction module. These scene-specific feature heads correspond to the scene type of the visual association scene and can perform standardized feature extraction on image data of specific scenes, specifically capturing the unique target attributes of the scene.

[0058] The feature fusion module is a multi-dimensional feature integration module of the detection model. It is used to fuse general features and scene features. By dynamically adjusting the fusion weights, the two types of features are weighted and integrated to balance the proportion of general basic information and scene-specific attributes. Finally, the enhanced features are obtained that retain the basic visual information of the image and highlight the key target information of the scene.

[0059] The Dynamic Feature Pyramid Network is the long-tail target localization and recognition module of the detection model. It is used to detect enhanced features and output detection results, including information such as the category, location coordinates, and confidence score of the long-tail target. Long-tail targets refer to special targets that appear less frequently in the visual network but have important value, such as kitchen violations, traffic accidents, and industrial safety hazards.

[0060] Through the above steps, the problems of low accuracy in long-tail target detection and insufficient generalization ability of the model in related technologies have been solved. It not only retains the basic adaptability to various scenarios, but also enhances the feature localization and recognition accuracy of long-tail targets in specific scenarios, effectively improving the detection performance and generalization ability of the model in visual association scenarios.

[0061] The above steps are explained in detail below:

[0062] In some embodiments, the feature extraction module further includes a basic feature extraction network for extracting features from each frame of the image to obtain general features.

[0063] Specifically, general features refer to common feature descriptions that reflect basic visual information such as edges, textures, and colors of an image, applicable to various scenarios. For example, in a kitchen or traffic scene, basic features such as the outlines of objects and changes in lighting can be represented by general features. The basic feature extraction network can be ResNet50, ResNeXt-50, or ResNeSt-50, etc., without limitation. This network extracts abstract features of the image layer by layer through multiple convolutional and pooling operations.

[0064] This embodiment utilizes a basic feature extraction network to obtain general features, providing stable basic visual information support for subsequent fusion with scene features, thereby improving the efficiency and versatility of feature extraction.

[0065] In some of these embodiments, the scene-specific feature heads in the feature extraction module include kitchen scene feature heads, transportation scene feature heads, security scene feature heads, and industrial scene feature heads;

[0066] Specifically, the kitchen scene feature head includes a chef's hat detector, a mask detector, a smoking detector, and a trash can detector, which respectively extract target features related to safety regulations in the kitchen scene: the chef's hat detector focuses on the hat shape outline and wearing status features of the head area, the mask detector captures the coverage area and fit features of the face, the smoking detector identifies the shape of smoke and the light spot features of fire, and the trash can detector extracts the position coordinates of the container and the internal overflow status features.

[0067] The traffic scene feature head includes vehicle detectors, pedestrian detectors, and traffic sign detectors, which are adapted to the target detection needs of traffic scenes: the vehicle detector extracts the contour structure and driving trajectory features of different vehicle types, the pedestrian detector captures the limb movements and movement path features of pedestrians on the road, and the traffic sign detector identifies the geometric shape and surface marking information features of the signs.

[0068] The security scene feature head includes intrusion detectors, loitering detectors, and fighting detectors, designed for abnormal behavior monitoring in security scenes: the intrusion detector extracts the characteristics of personnel entering unauthorized areas and changes in their positions; the loitering detector captures the characteristics of repeated movement trajectories and dwell time in specific areas; and the fighting detector identifies the amplitude of movements and contact posture characteristics of physical conflicts.

[0069] The industrial scenario feature head includes equipment operation detectors, safety violation detectors, and equipment fault detectors, focusing on the safety and operation and maintenance needs of industrial scenarios: the equipment operation detector extracts the standardized features of the interaction between personnel and equipment, the safety violation detector identifies the features of the exposed areas of the body without protective equipment, and the equipment fault detector captures abnormal textures, subtle deformations, and abnormal light spots on the surface of the equipment.

[0070] This embodiment uses a specific feature head to extract scene-specific features, which can accurately identify key target attributes in different scenes, providing targeted scene feature inputs for subsequent feature fusion and effectively improving the accuracy of long-tail target detection in specific scenes.

[0071] In some embodiments, the long-tail target detection method based on visual association scenes further includes the following steps:

[0072] In the feature fusion module, enhanced features are obtained by fusing based on dynamically adjusted fusion weights, general features, and scene features; the dynamically adjusted fusion weights are determined by the scene type.

[0073] Specifically, the enhanced features are obtained by fusing dynamically adjusted fusion weights, general features, and scene features. These enhanced features can be flexibly selected according to scene requirements, and there are multiple implementation methods. One method is to achieve weighted fusion through mathematical formulas. This involves integrating general features and scene features with dynamic weights using a pre-defined calculation formula, ensuring that the enhanced features retain basic visual information while highlighting key scene attributes. If the formula method is used, the calculation formula for the enhanced features is:

[0074]

[0075] Among them, F base F represents the general features extracted from the image by the basic feature extraction network. scene This represents the scene features extracted from the scene-specific feature head, where α is the scene-adaptive fusion weight, and its value is dynamically adjusted according to the scene type.

[0076] In other embodiments, the implementation may be feature matrix concatenation and fusion, or attention mechanism-guided fusion, and there are no limitations on this.

[0077] In this embodiment, the enhanced features retain the basic visual information of the image while highlighting the key attributes of scene-specific targets, providing more targeted input features for the dynamic feature pyramid network and effectively improving the feature representation capability in visual association scenarios.

[0078] In some embodiments, the dynamic feature pyramid network includes a feature pyramid network, a scene gating network, and a detection head network. Figure 3 This is a flowchart of a dynamic feature pyramid network, such as... Figure 3 As shown, the process also includes the following steps:

[0079] Step S211, Feature Pyramid Network, used to extract multi-scale features based on enhanced features;

[0080] Step S212, scene gating network, used to adaptively adjust multi-scale features to obtain gating features;

[0081] Step S213: The detection head network is used to detect the gated features and obtain the detection results.

[0082] Specifically, the Dynamic Feature Pyramid Network achieves accurate detection of long-tailed targets in visual scenes through the synergistic effect of the feature pyramid network, scene gating network, and detection head network. The feature pyramid network is a feature extraction and fusion structure in a target detection system used to aggregate multi-scale feature information. It extracts multi-scale features based on enhanced features, using the basic feature pyramid network structure to extract multi-scale features {F1, F2, F3, F4, F5} from the enhanced features, adapting to long-tailed targets of different sizes, such as small-scale smoke and large-scale kitchen utensils in a kitchen scene, or subtle fault features and the overall outline of equipment in an industrial scene. The scene gating network is a structure that solves cross-scene feature interference. It adaptively adjusts multi-scale features through a scene-specific gating mechanism, filtering out noise features irrelevant to the current scene and strengthening the core attributes of long-tailed targets within the scene to obtain gated features. There are multiple ways to obtain gating features, one of which is through a scene-specific gating function F. i_gated =Gate i (F i ) F i For each scale feature F i Adjustments, including Gate i The first network is a dedicated gating network for scene i. The second network, the detection head network, is a structure that transforms the optimized gating features into specific detection results. Through the collaborative work of classification and localization branches, it outputs the core information of long-tail targets, completing the transformation from abstract features to concrete decisions. The detection head network classifies and localizes the optimized features, outputting information such as the target's category, location coordinates, and confidence level, thereby achieving accurate recognition of long-tail targets in the visual scene.

[0083] This embodiment utilizes a feature pyramid network to extract multi-scale features, adaptive adjustments to the scene gating network, and the output of the detection head network to improve the model's detection performance for low-frequency long-tailed targets of different scales in complex visual scenes.

[0084] In some of these embodiments, the long-tail target detection method based on visual scenes further includes the following steps: after obtaining the gating features in the dynamic feature pyramid network, the gating features are optimized based on a scene-specific attention mechanism.

[0085] Specifically, the scene-specific attention mechanism designs dedicated attention modules for the target characteristics of different visual association scenes, using formula F. i_attended =Attention i (F i_gatedThe optimized features are obtained. This mechanism focuses on long-tail target regions of the scene by calculating the similarity weights between features: for example, in a kitchen scene, the attention module will concentrate the weights on features related to smoking behavior, such as smoke diffusion areas and lighter light spots, while weakening irrelevant background features such as kitchenware reflections and wall textures; in an industrial scene, it will focus on strengthening the weights of fault features such as abnormal textures on equipment surfaces and loose bolts, while reducing the feature ratio of non-target areas such as normal equipment shells and work surfaces.

[0086] In this embodiment, the optimized features enhance the core visual information of long-tailed targets (such as the grayscale gradient changes of smoke and the edge contours of fault textures) and weaken irrelevant background interference (such as the decorative textures of kitchen walls and the shadows on the ground in industrial scenes). This provides more discriminative input for the subsequent classification and localization of the detection head network, enabling the detection head network to more clearly distinguish target categories during classification and to more accurately define target areas during localization, effectively improving the accuracy of target detection in long-tailed scenes.

[0087] In some embodiments, the long-tail target detection method based on visual association scenes further includes the following steps:

[0088] Based on the video data of the VisionLink scene and the corresponding scene category set, a dedicated evaluation system for the VisionLink scene is constructed;

[0089] The test results are evaluated based on a dedicated evaluation system.

[0090] Specifically, such as Figure 4 As shown, firstly, video data is acquired by collecting videos from multiple scenarios from terminals such as smart city cameras and industrial monitoring equipment. The set of video-linked scene categories is defined as S={kitchen,traffic,security,industrial}, and a weight matrix W is constructed for each scene category.

[0091] Secondly, to address the varying detection requirements across different scenarios, specific evaluation parameters are set for each scenario category, including the IoU threshold T. iou and confidence threshold T conf .

[0092] In kitchen scenarios, due to the small size or occlusion of targets (such as chef's hats and masks), setting the IoU threshold to 0.5 can capture more potential targets, while a confidence threshold of 0.7 ensures high detection accuracy and avoids a large number of false alarms. Using a higher IoU threshold (such as 0.7) may cause many real targets to be missed due to insufficient overlap with the ground truth bounding boxes. In traffic scenarios, the outlines of targets such as vehicles and traffic signs are relatively clear, but a balance needs to be struck between detection completeness and accuracy. Therefore, an IoU threshold of 0.6 and a confidence threshold of 0.8 are set. In security scenarios, the detection requirements for behaviors such as intrusion and loitering are usually very high, requiring an extremely low false alarm rate. Therefore, an IoU threshold of 0.7 and a confidence threshold of 0.9 are set to ensure that only highly accurate detection results are adopted, thereby improving the reliability of the security system. In industrial scenarios, long-tail targets such as equipment malfunctions need to balance detailed recognition and false alarm control. Therefore, an IoU threshold of 0.6 and a confidence threshold of 0.8 are set.

[0093] Finally, the prediction results are filtered using scene-specific IoU and confidence thresholds, and then a weighted average accuracy is calculated to evaluate the detection results. The specific formula for the weighted average accuracy is as follows:

[0094]

[0095] Where W[i][j] represents the weighted influence of scene i on scene j, and AP[i] is the average precision of scene i. It represents the number of scenes in the visual association scene category set (here) =4).

[0096] This embodiment constructs a dedicated evaluation system for visual association scenarios and uses this system to evaluate detection results. The evaluation results better reflect detection performance in practical applications and are more aligned with the actual application needs of visual association scenarios, providing more valuable feedback for model optimization and system deployment. Ultimately, this achieves accurate matching of detection results with actual needs in different scenarios, reducing the problems of missed detections and false alarms caused by general evaluation standards.

[0097] This embodiment provides a long-tail target detection device based on visual scene association. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. The terms "module," "unit," "subunit," etc., used below refer to combinations of software and / or hardware that perform a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0098] Figure 5 This is a structural block diagram of the long-tail target detection device based on visual scenes in this embodiment, as shown below. Figure 5As shown, the device includes: a processing module 210 and a detection module 220;

[0099] Processing module 210 is used to acquire each frame of image and its corresponding scene label;

[0100] The detection module 220 is used to input each frame of image and its corresponding scene label into a pre-trained detection model to obtain detection results. The detection model includes a feature extraction module, a feature fusion module, and a dynamic feature pyramid network. The feature extraction module is used to extract features from each frame of image to obtain general features and scene features. Scene features are obtained by processing scene-specific feature heads in the feature extraction module. Scene-specific feature heads correspond to the scene type of the visual association scene. The feature fusion module is used to fuse general features and scene features to obtain enhanced features. The dynamic feature pyramid network is used to detect the enhanced features to obtain detection results.

[0101] The aforementioned device solves the problems of low accuracy in long-tail target detection and insufficient generalization ability of the model in related technologies, thereby improving the accuracy of long-tail target detection and enhancing the generalization ability of the model.

[0102] In some embodiments, the feature extraction module further includes a basic feature extraction network; the basic feature extraction network is used to extract features from each frame of the image to obtain general features.

[0103] In some embodiments, the scene-specific feature headers also include kitchen scene feature headers, transportation scene feature headers, security scene feature headers, and industrial scene feature headers;

[0104] The kitchen scene feature heads include a chef's hat detector, a mask detector, a smoking detector, and a trash can detector;

[0105] Traffic scene feature heads include vehicle detectors, pedestrian detectors, and traffic sign detectors;

[0106] Security scene feature heads include intrusion detectors, loitering detectors, and fighting detectors;

[0107] The industrial scene feature headers include equipment operation detectors, safety violation detectors, and equipment fault detectors.

[0108] In some embodiments, the detection module 220 is further configured to fuse features in the feature fusion module based on dynamically adjusted fusion weights, general features, and scene features to obtain enhanced features; the dynamically adjusted fusion weights are determined by the scene type.

[0109] In some of these embodiments, the dynamic feature pyramid network includes a feature pyramid network, a scene gating network, and a detection head network;

[0110] Feature pyramid network, used to extract multi-scale features based on enhanced features;

[0111] Scene gating networks are used to adaptively adjust multi-scale features to obtain gated features;

[0112] The detection head network is used to detect gated features and obtain detection results.

[0113] In some embodiments, the detection module 220 is also configured to optimize the gating features based on a scene-specific attention mechanism after obtaining the gating features.

[0114] In some embodiments, the detection module 220 is also used to construct a dedicated evaluation system for the visual association scene based on the video data of the visual association scene and the corresponding scene category set;

[0115] The test results are evaluated based on a dedicated evaluation system.

[0116] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0117] This embodiment provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0118] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0119] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0120] S1, obtain each frame image and its corresponding scene label;

[0121] S2 inputs each frame of image and its corresponding scene label into a pre-trained detection model to obtain the detection result. The detection model includes a feature extraction module, a feature fusion module, and a dynamic feature pyramid network. The feature extraction module is used to extract features from each frame of image to obtain general features and scene features. Scene features are obtained by processing scene-specific feature heads in the feature extraction module. Scene-specific feature heads correspond to the scene type of the visual association scene. The feature fusion module is used to fuse general features and scene features to obtain enhanced features. The dynamic feature pyramid network is used to detect the enhanced features to obtain the detection result.

[0122] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0123] Furthermore, in conjunction with the long-tail target detection method based on visual scenes provided in the above embodiments, this embodiment can also provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the long-tail target detection methods based on visual scenes in the above embodiments.

[0124] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0125] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0126] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0127] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A method for detecting long-tailed targets based on visual association scenes, characterized in that, include: Obtain each frame image and its corresponding scene label; Each frame of the image and its corresponding scene label are input into a pre-trained detection model to obtain the detection result; The detection model includes a feature extraction module, a feature fusion module, and a dynamic feature pyramid network; The feature extraction module is used to extract features from each frame of the image to obtain general features and scene features; The scene features are obtained by processing the scene-specific feature head in the feature extraction module; The scene-specific feature header corresponds to the scene type of the visual association scene; the general feature is a description of the common features of various visual association scenes; The feature fusion module is used to fuse the general features and the scene features to obtain enhanced features; In the feature fusion module, the enhanced features are obtained by fusing the dynamically adjusted fusion weights, the general features, and the scene features. The dynamically adjusted fusion weights are determined by the scene type; The dynamic feature pyramid network is used to detect the enhanced features and obtain the detection results.

2. The long-tail target detection method based on visual association scene according to claim 1, characterized in that, The feature extraction module also includes a basic feature extraction network; The basic feature extraction network is used to extract features from each frame of the image to obtain the general features.

3. The long-tail target detection method based on visual association scene according to claim 1, characterized in that, The scene-specific feature headers include kitchen scene feature headers, transportation scene feature headers, security scene feature headers, and industrial scene feature headers; The kitchen scene feature heads include a chef's hat detector, a mask detector, a smoking detector, and a trash can detector; The traffic scene feature header includes vehicle detectors, pedestrian detectors, and traffic sign detectors; The security scene feature header includes an intrusion detector, a loitering detector, and a fight detector; The industrial scenario feature header includes equipment operation detectors, safety violation detectors, and equipment fault detectors.

4. The long-tail target detection method based on visual association scene according to claim 1, characterized in that, The dynamic feature pyramid network includes a feature pyramid network, a scene gating network, and a detection head network; The feature pyramid network is used to extract multi-scale features based on the enhanced features; The scene gating network is used to adaptively adjust the multi-scale features to obtain gating features; The detection head network is used to detect the gating features and obtain the detection results.

5. The long-tail target detection method based on visual association scene according to claim 4, characterized in that, The method further includes: After obtaining the gating features, the gating features are optimized based on a scene-specific attention mechanism.

6. The long-tail target detection method based on visual association scene according to claim 1, characterized in that, The method further includes: Based on the video data of the aforementioned visual association scene and the corresponding scene category set, a dedicated evaluation system for the visual association scene is constructed. The test results are evaluated based on the dedicated evaluation system.

7. A long-tail target detection device based on visual-connected scenes, characterized in that, include: Processing module and detection module; The processing module is used to acquire each frame of image and its corresponding scene label; The detection module is used to input each frame of the image and the corresponding scene label into a pre-trained detection model to obtain the detection result; The detection model includes a feature extraction module, a feature fusion module, and a dynamic feature pyramid network; The feature extraction module is used to extract features from each frame of the image to obtain general features and scene features; The scene features are obtained by processing the scene-specific feature head in the feature extraction module; The scene-specific feature header corresponds to the scene type of the visual association scene; the general feature is a description of the common features of various visual association scenes; The feature fusion module is used to fuse the general features and the scene features to obtain enhanced features; In the feature fusion module, the enhanced features are obtained by fusing the dynamically adjusted fusion weights, the general features, and the scene features. The dynamically adjusted fusion weights are determined by the scene type; The dynamic feature pyramid network is used to detect the enhanced features and obtain the detection results.

8. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the steps of the long-tailed target detection method based on visual scenes as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the long-tail target detection method based on visual scenes as described in any one of claims 1 to 6.

Images