Open-world object detection method and system based on attention mechanism

Abstract

The application relates to the technical field of computer vision, and particularly discloses an open-world target detection method and system based on an attention mechanism. The application obtains a to-be-detected feature map by processing a to-be-detected video stream image, detects the to-be-detected feature map to obtain an initial detection result, determines a possible region of an unknown-class target in the to-be-detected feature map through the initial detection result, adjusts the attention weight of the region to strengthen the feature weight of the region, and finally detects the to-be-detected image according to the attention weight to obtain a target recognition result. When there is an unknown-class target in the image, the attention weight of the region can be strengthened, the model is prompted to analyze the potential unknown-class target more deeply, the model does not need to be retrained for the unknown-class target, and the processing efficiency of the target detection task is improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an open-world object detection method and system based on an attention mechanism. Background Technology

[0002] In the field of object detection technology, traditional object detection methods typically rely on labeled categories in the training data to identify and locate objects in images or video frames. These methods perform well on the categories included in the training set, but due to the lack of adaptability and generalization ability of traditional models to new situations, they cannot effectively learn from limited data and generalize to new categories. When encountering unknown or unlabeled new categories of objects, they usually fail to perform object detection tasks correctly. With the development of deep learning technology, especially the application of convolutional neural networks in image recognition, object detection technology has made significant progress. However, most deep learning models require large amounts of labeled data for training and still cannot complete object detection tasks when faced with unknown categories in open-world environments. Although some solutions have been proposed, such as using metric learning, generative adversarial networks (GANs), or self-supervised learning methods to enhance the model's generalization ability to unknown categories, these methods often require complex training processes, resulting in low processing efficiency for object detection tasks. Therefore, how to improve the processing efficiency of object detection tasks has become an urgent problem to be solved. Summary of the Invention

[0003] This application provides an open-world object detection method and system based on an attention mechanism to improve the processing efficiency of object detection tasks.

[0004] Firstly, this application provides an open-world object detection method based on an attention mechanism, the method comprising:

[0005] Acquire the image of the video stream to be detected;

[0006] Based on a preset target detection model, the video stream image to be detected is processed to obtain a feature map to be detected, and target detection is performed on the feature map to obtain an initial detection result.

[0007] Based on the initial detection results, the region where the unknown category target is located in the feature map to be detected is determined;

[0008] Based on the target detection model, the attention weights of the region where the unknown category target is located are adjusted, and based on the attention weights, target detection is performed on the feature map to be detected to obtain the recognition result of the unknown category target.

[0009] Secondly, this application also provides a computer device, the computer device including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the open-world object detection method based on the attention mechanism as described above.

[0010] Thirdly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the attention-based open-world object detection method described above.

[0011] This application discloses an open-world object detection method and system based on an attention mechanism. The method involves acquiring a video stream image to be detected; processing the video stream image to be detected based on a preset object detection model to obtain a feature map to be detected; performing object detection on the feature map to obtain an initial detection result; determining the region containing an unknown category object in the feature map based on the initial detection result; adjusting the attention weight of the region containing the unknown category object based on the object detection model; and performing object detection on the feature map based on the attention weight to obtain the recognition result of the unknown category object. This application processes the video stream image to be detected to obtain a feature map to be detected, then performs detection on the feature map to obtain an initial detection result. The initial detection result determines the possible region containing an unknown category object in the feature map. The attention weight of this region is then adjusted to strengthen its feature weight. Finally, object detection is performed on the image to be detected based on the attention weight to obtain the object recognition result. When an unknown category of target exists in an image, the attention weight of that region can be enhanced, prompting the model to analyze potential unknown category targets more deeply. This eliminates the need to retrain the model for unknown category targets, thus improving the processing efficiency of the target detection task. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic flowchart of a first embodiment of an attention-based open-world object detection method provided in this application;

[0014] Figure 2This is a schematic flowchart of a second embodiment of an open-world object detection method based on an attention mechanism provided in this application;

[0015] Figure 3 This is a schematic flowchart of a third embodiment of an attention-based open-world object detection method provided in this application;

[0016] Figure 4 This is a schematic flowchart of a fourth embodiment of an attention-based open-world object detection method provided in this application;

[0017] Figure 5 A schematic block diagram of an attention-based open-world object detection system provided for embodiments of this application;

[0018] Figure 6 A schematic block diagram of the structure of a computer device provided for an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0021] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0022] It should also be understood that the term "and / or" as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0023] This application provides an open-world object detection method and system based on an attention mechanism. This attention-based open-world object detection method can be applied to a server. By enhancing the attention weight of the region containing an unknown category of object in an image, it encourages the model to analyze potential unknown category objects more deeply, eliminating the need to retrain the model for unknown category objects and improving the processing efficiency of the object detection task. The server can be a standalone server or a server cluster.

[0024] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0025] Please see Figure 1 , Figure 1 This is a schematic flowchart of an open-world object detection method based on an attention mechanism provided in an embodiment of this application.

[0026] like Figure 1 As shown, the open-world object detection method based on the attention mechanism specifically includes steps S101 to S104.

[0027] S101. Obtain the image of the video stream to be detected;

[0028] In one embodiment, a real-time video stream captured by a camera is received as input data, the video stream providing continuous images frame by frame as images of the video stream to be detected.

[0029] S102. Based on a preset target detection model, the video stream image to be detected is processed to obtain a feature map to be detected, and target detection is performed on the feature map to be detected to obtain an initial detection result.

[0030] Further, the step of processing the video stream image to be detected based on the preset target detection model to obtain the feature map to be detected includes: processing the video stream image to be detected based on the target detection model to obtain a set of multi-size feature maps; extracting features from each multi-size feature map in the set of multi-size feature maps to obtain feature information of the target in each multi-size feature map; and determining the feature map to be detected in the set of multi-size feature maps based on the feature information.

[0031] In one embodiment, multi-scale feature maps are processed using MSHEA (Multi-Scale Hollow Explicit Attention). MSHEA consists of four attention heads with atrium rates {1, 2, 4, 8}, each corresponding to a different scale of feature extraction to capture feature information from small to large targets. Specifically, the first attention head sets a low atrium rate to finely capture detailed features of small targets; each subsequent head gradually increases the atrium rate to accommodate increasingly larger target sizes, until the fourth head, configured with the highest atrium rate, is used to identify and locate large targets in the image.

[0032] In a specific embodiment, the MSHEA module consists of four attention heads, each designed with a specific dilation rate of 1, 2, 4, and 8. This dilation rate setting allows the corresponding attention head to focus on capturing features of targets of different sizes. The head with a dilation rate of 1 captures the finest features, while the head with a dilation rate of 8 focuses on larger target features. The dilated convolutional kernel size of the MSHEA module is matched to its respective dilation rate. This configuration allows the module to comprehensively capture target features of varying scales in the image, from small to significant, while maintaining computational efficiency. Furthermore, the MSHEA module employs a feature fusion strategy to integrate the feature maps of four different scales into a single feature representation, thereby providing rich multi-scale contextual information for subsequent target detection tasks and significantly improving the accuracy of target detection.

[0033] Further, determining the feature map to be detected in the multi-scale feature map set based on the feature information includes: calculating the gradient magnitude and counting the activation values ​​of each multi-scale feature map based on the feature information, and obtaining the gradient magnitude value and activation value information corresponding to each multi-scale feature map respectively; and determining the feature map to be detected in each multi-scale feature map based on the gradient magnitude value and the activation value information.

[0034] In one embodiment, an adaptive feature selection mechanism processes feature information, which may include pixels, activation values, etc. This mechanism includes an advanced algorithm that dynamically selects the optimal feature map by comprehensively applying gradient magnitude evaluation, activation value distribution analysis, and confidence evaluation. Specifically, the mechanism first performs pixel-by-pixel differentiation on the multi-scale feature map using a feature map gradient calculation unit, quantifies the intensity of local changes using a gradient magnitude evaluator, and then constructs a histogram and calculates statistics using an activation value distribution analyzer to evaluate information richness. Simultaneously, a confidence evaluator analyzes the distribution of the model's output probability to determine the confidence level of the prediction. The feature selection decision engine integrates these indicators, uses weighted decision rules to generate a comprehensive score, and intelligently selects key feature maps. Furthermore, parameters are dynamically adjusted based on performance feedback to optimize the selection process. The entire algorithm is implemented using an efficient numerical computation library, ensuring real-time performance and accuracy, thereby enabling the target detection model to flexibly adapt to different scenarios and improve detection accuracy and efficiency.

[0035] In a specific embodiment, the detailed steps of the above algorithm are as follows:

[0036] 1. Uncertainty Assessment: The algorithm first assesses the uncertainty of the model's predictions. This can be achieved by calculating the entropy of the probability distribution of the predicted categories, where a high entropy value indicates greater prediction uncertainty, requiring further analysis of the model.

[0037] 2. Gradient Calculation: For each feature map, the algorithm calculates the gradient magnitude for each pixel. The first derivative of the gradient can be approximated using the Sobe or Scharr operator to identify regions of significant change in the image.

[0038] 3. Activation value statistical analysis: The algorithm analyzes the distribution characteristics of all activation values ​​on the feature map, including calculating statistical measures such as mean, variance, and skewness, in order to identify the possible locations of targets on the feature map.

[0039] 4. Selection Criteria Formulation: Based on the above statistical data, the algorithm formulates selection criteria, selecting feature maps with high gradient magnitudes and specific activation value distributions. These feature maps may contain important information about the target object.

[0040] 5. Feature Map Weighting: The algorithm weights the feature maps according to the selection criteria. Regions with high uncertainty and large gradient magnitudes will receive higher weights to enhance the importance of these regions in subsequent processing.

[0041] 6. Feature Map Pooling: Through pooling operations, such as max pooling or average pooling, the algorithm reduces the dimensionality of the feature map while retaining the most important features, providing a more refined input for the deep learning model.

[0042] 7. Feedback Adjustment: The adaptive feature selection mechanism also includes a feedback adjustment step, which dynamically adjusts the selection criteria and weighting factors based on the final result of object detection to optimize the future feature selection process.

[0043] In one embodiment, this mechanism can be integrated as a standalone module into the object detection system, working in conjunction with other modules such as feature extraction, attention enhancement, object detection, and classification to form an efficient detection workflow. Through this adaptive feature selection mechanism, the object detection system of this invention can adaptively focus on key features in the image, ignoring unimportant or noisy information, thereby significantly improving the accuracy of object detection, especially when detecting unknown or rare object categories.

[0044] S103. Based on the initial detection results, determine the region where the unknown category target is located in the feature map to be detected;

[0045] Further, step S103 includes: obtaining probability distribution information of target categories in the feature map to be detected based on the initial detection result; calculating the uncertainty of each region in the feature map to be detected based on the probability distribution information; comparing the uncertainty with a preset uncertainty threshold; and when the uncertainty is greater than the uncertainty threshold, taking the region as the region where the unknown category target is located.

[0046] In one embodiment, the initial detection result includes the predicted probability of each target category in the output layer of the target detection model. Uncertainty is quantified by calculating the standard deviation of the predicted probability distribution for each target category in the model's output layer. This uncertainty is then compared to a preset uncertainty threshold. When the uncertainty of a certain region exceeds the uncertainty threshold, that region is considered the location of an unknown category target. The uncertainty threshold can be set by the user according to specific circumstances.

[0047] S104. Based on the target detection model, the attention weights of the region where the unknown category target is located are adjusted, and based on the attention weights, target detection is performed on the feature map to be detected to obtain the recognition result of the unknown category target.

[0048] In one embodiment, the attention weights of the regions containing unknown category targets are adjusted by increasing the weight factor in the attention module or by applying a specific algorithm to enhance the feature representation of these regions.

[0049] In one embodiment, the attention adjustment process is iterative, with each iteration re-evaluating and updating the weights based on the latest prediction uncertainty to ensure that the model continues to focus on the most uncertain regions that may contain unknown categories of targets.

[0050] In one embodiment, the attention enhancement strategy can be implemented as a standalone module, integrated and implemented at any critical stage of the target detection process to improve the model's detection accuracy for new categories of targets.

[0051] In a specific embodiment, the attention enhancement strategy is a dynamic adjustment process designed to increase the model's attention to regions of high prediction uncertainty, thereby enhancing its ability to identify targets of unknown categories. The implementation of this strategy involves the following key steps:

[0052] 1. Uncertainty Quantification: First, the system quantifies the uncertainty of the prediction. This can be achieved in various ways, such as calculating the standard deviation of the probability distribution of the predicted categories.

[0053] 2. Attention weight initialization: In the attention mechanism, the initial attention weights for each feature map are usually determined by a trainable neural network layer that learns how to allocate weights to maximize object detection performance.

[0054] 3. Dynamic Weight Adjustment: When the system's uncertainty quantification index indicates high prediction uncertainty in a certain region, the system automatically adjusts the attention weight for that region. This is typically achieved through an adjustment factor (a), which dynamically scales the attention weight based on the magnitude of the uncertainty.

[0055] 4. Attention Weight Update: The updated attention weight can be calculated using the following formula: w new =w old +α×(uncertainty quantification index), where w old It is the original attention weight, w new These are the updated weights, and α is a learnable, adjustable parameter used to control the degree to which uncertainty affects the attention weights.

[0056] 5. Feature Enhancement: Using updated attention weights, the system weights the feature map to enhance the feature representation of regions with high uncertainty. This can be achieved by multiplying each channel of the feature map by the corresponding weighting factor.

[0057] 6. Deep Network Ensemble: This strategy is integrated into deep learning models, typically implemented in the attention module between feature extraction and object detection. This allows the model to automatically focus on regions of high uncertainty during training and inference.

[0058] 7. Iterative Optimization: During training, the adjustment factor and other relevant network parameters are continuously updated using the backpropagation algorithm to minimize the loss function for object detection, while improving the ability to identify unknown categories of objects. Through this attention-enhancing strategy, the system can more effectively process regions where the model is uncertain, thereby improving the detection accuracy of unknown or rare objects.

[0059] In a specific embodiment, the object detection method proposed in this application includes an object detection and classification engine. This engine is a comprehensive processing unit responsible for accurately locating objects in an image and determining their categories. The following is the specific process by which the engine performs object detection and classification:

[0060] 1. Feature Map Integration: The engine first receives enhanced feature maps from MSHEA and the adaptive feature selection mechanism. These feature maps contain rich representations of the target object, highlighting its key features.

[0061] 2. Attention Weighting: This combines enhanced feature maps with attention weights. These weights reflect the importance of different regions, with regions of higher uncertainty receiving higher weights. Weighting is achieved by element-wise multiplication of the feature map with the corresponding attention weight vector.

[0062] 3. RPN (Region Proposal Network): For feature maps, the engine uses a region proposal network to predict the location of potential targets. RPN generates region proposals at different scales using sliding windows or anchor boxes.

[0063] 4. Bounding Box Regression: For each candidate region, the engine performs bounding box regression to fine-tune the target's position. This typically involves predicting the offset of each candidate region to obtain a more accurate target bounding box.

[0064] 5. Category Classification: For each adjusted bounding box, the engine uses a Softmax classifier to determine the target's category. The Softmax function converts the classifier's output into a probability distribution, representing the probability that the target belongs to each category.

[0065] 6. NMS (Non-Maximum Suppression): To remove overlapping predictions and duplicate detections, the engine applies the NMS algorithm. NMS cleans up the final detection results by selectively retaining the prediction with the highest probability and suppressing other overlapping predictions.

[0066] 7. Confidence threshold: The engine sets a confidence threshold. Only when the predicted class probability exceeds this threshold will the detected target be considered trustworthy.

[0067] 8. Output Results: Finally, the engine outputs the class probability and corresponding bounding box coordinates for each target, completing the target detection and classification. Through this continuous process, the target detection and classification engine can accurately identify and locate targets in images, while determining their categories, providing a powerful and accurate solution for open-world target detection.

[0068] In one embodiment, open world is a multi-faceted concept. In the field of artificial intelligence, open world refers to the ability of an AI system to adapt to and learn from a variety of environments and tasks, rather than being limited to specific, closed scenarios, such as autonomous driving, intelligent monitoring, and robot vision.

[0069] Please see Figure 2 , Figure 2 This is a schematic flowchart of an open-world object detection method based on an attention mechanism provided in an embodiment of this application.

[0070] like Figure 2 As shown, the open-world object detection method based on the attention mechanism specifically includes steps S201 to S204.

[0071] S201. Obtain the language description information of the unknown category target, and based on the feature information of the unknown category target, obtain the visual features of the unknown category target;

[0072] S202. Perform natural language processing on the language description information to obtain the language features corresponding to the language description information, which are used as the language features of the unknown category target;

[0073] S203. The visual features and language features are fused to obtain the fused features;

[0074] S204. Based on the attention weights and the fusion features, target detection is performed on the feature map to be detected to obtain the recognition result of the target of the location category.

[0075] In one embodiment, the fusion of visual and linguistic features is achieved through a cross-modal feature fusion technique. This technique enhances the ability to identify targets of unknown categories by combining the linguistic descriptions provided by the system with visual features extracted from the video stream. Specifically, the technique first converts the system's linguistic descriptions into numerical feature vectors using natural language processing (NLP). Then, it calculates the semantic similarity between these feature vectors and the visual features, typically using metrics such as cosine similarity. Simultaneously, it analyzes the spatial consistency between the spatial information of the target in the linguistic description and the spatial consistency of the target in the visual feature map, ensuring the correspondence between the linguistic description and the visual data in the real-world scenario. Next, using specific algorithms, such as weighted averaging or feature concatenation, the linguistic description feature vectors, based on semantic similarity and spatial consistency matching, are fused with the visual features to form a comprehensive feature representation. This representation provides richer contextual information, helping the model better understand and classify targets. Finally, this fusion technique, as an integrated module, is applied to the target detection process, significantly improving the model's detection accuracy and generalization ability for new target categories.

[0076] In a specific embodiment, the cross-modal feature fusion framework is an advanced processing unit responsible for deeply fusing visual features with linguistic descriptions from the system to improve the understanding and recognition of target semantics. The following are the specific implementation details of this framework:

[0077] 1. Language Feature Extraction: The framework first utilizes NLP (Natural Language Processing) techniques to extract language features from the system's language description. Various pre-trained word embedding models can be used to convert text into numerical vectors.

[0078] 2. Semantic Embedding Space: The extracted linguistic features are mapped to a semantic embedding space, which matches the representation space of the visual features, so that the features of the two modalities can be compared and fused on the same dimension.

[0079] 3. Feature Alignment: By calculating the similarity or distance between visual and linguistic features, the frame determines the correspondence between them. This can be achieved using cosine similarity, Euclidean distance, or more complex metrics.

[0080] 4. Feature weighting: Based on semantic similarity, the framework weights visual and linguistic features to strengthen the visual features most relevant to the linguistic description.

[0081] 5. Feature Fusion: Weighted features are combined using specific fusion strategies. These can include simple weighted averaging, feature concatenation, or more advanced fusion techniques.

[0082] 6. Context integration: The fused features not only include visual information, but also incorporate contextual information provided by language description, thus providing a richer semantic representation for each target.

[0083] 7. Attention Modulation: The fused feature representations are further modulated through an attention mechanism to highlight the feature parts most relevant to the target category and suppress irrelevant or noisy features.

[0084] 8. Deep Network Training: Cross-modal fusion of feature representations serves as input to the deep learning model and is continuously optimized during the training process to minimize the loss function for object detection and classification.

[0085] 9. End-to-end optimization: The entire framework is designed as an end-to-end trainable system, allowing the entire process from visual feature extraction to linguistic feature fusion, and then to object detection and classification to be jointly optimized during training.

[0086] 10. Enhanced Detection Results Output: Finally, the fusion framework outputs enhanced target detection results, which are based not only on visual information but also take into account the semantic context provided by the linguistic description. Through this cross-modal feature fusion framework, the target detection system of this invention can more accurately identify and classify targets, especially when processing images with complex backgrounds or blurred visual features, where the system's performance is significantly improved.

[0087] Please see Figure 3 , Figure 3 This is a schematic flowchart of an open-world object detection method based on an attention mechanism provided in an embodiment of this application.

[0088] like Figure 3 As shown, the open-world object detection method based on the attention mechanism specifically includes steps S301 to S304.

[0089] S301. When there is a target with a confidence level lower than a preset confidence level threshold in the initial detection results, the target is marked as a new category target;

[0090] S302. Based on the identification result of the unknown category target, obtain the category label of the new category target;

[0091] S303. Based on the new category target and the category label, generate a new category target dataset;

[0092] S304. Based on the preset formula and the new category target dataset, update the parameters of the target detection model to update the target detection model.

[0093] Furthermore, the preset formula is as follows:

[0094]

[0095] Where, θ t These are the parameters of the model at the t-th iteration, and η is the learning rate. It is a loss function The gradient with respect to parameter θ, It is a new category target dataset.

[0096] In one embodiment, when an unknown category of target is identified, an incremental learning mechanism is used to automatically update the model, intelligently adapting to new target categories. This mechanism includes a data collection module that is automatically triggered when the model identifies targets with low prediction confidence, collecting and labeling data based on the target's category label and location information. The collected data is stored in an incremental data storage module, providing material for model updates. At the core of the mechanism is a model update algorithm that employs an online learning strategy, allowing the model to progressively integrate new data and update parameters during continuous operation to adapt to new categories. This update strategy is designed to be incremental, adjusting only a small portion of the model's parameters each time to minimize the impact on existing performance and maintain model stability. Furthermore, model performance is continuously monitored throughout the process, and the data collection and update strategy is dynamically adjusted through a feedback loop to ensure the model maintains high detection accuracy for known target categories while continuously learning. By implementing model version control, parameter changes and update history can be tracked, allowing for regression to older models when necessary. This mechanism enables the target detection method to maintain long-term effectiveness and adaptability in open-world environments, continuously evolving to identify newly emerging target categories.

[0097] In a specific embodiment, the incremental learning algorithm includes the following steps:

[0098] 1. Data Collection: When the object detection engine identifies a target with a prediction confidence level below a preset threshold, the system automatically marks these targets as potential new category instances and triggers a data collection mechanism. These instances are then sent to a temporary buffer for further processing.

[0099] 2. Online data annotation: For the collected data, the system uses automated tools to annotate the data based on the final identification results of the unknown category targets, so as to annotate the category labels of the new category targets.

[0100] 3. Mini-batch learning: To avoid large-scale updates to model parameters and thus maintain model stability, the algorithm employs a mini-batch learning strategy. This means that only a small portion of newly collected data is used for each model update.

[0101] 4. Parameter Update Strategy: Incremental learning algorithms employ cautious parameter update strategies, such as using a small learning rate or an adaptive learning rate strategy. Parameter updates can be expressed by the following formula: Where, θ t These are the parameters of the model at the t-th iteration, and η is the learning rate. It is a loss function The gradient with respect to parameter θ, It is a new category target dataset.

[0102] 5. Model Evaluation and Selection: After each update, the system evaluates the model performance to ensure that the detection accuracy for known categories has not significantly decreased. If performance degrades, the system may revert to the previous model version.

[0103] 6. Experience replay: In order to further improve the model’s ability to generalize to new categories, incremental learning algorithms can adopt an experience replay mechanism, that is, periodically retrain the model to include a combination of old and new data.

[0104] In one embodiment, to avoid catastrophic failure (i.e., the model forgets previously learned knowledge), the algorithm may employ techniques such as appropriately increasing the proportion of old data or using the Elastic Weights Combined (EWC) algorithm to protect important old knowledge.

[0105] In one embodiment, model saving and version control are implemented so that the system can revert to an earlier model version when necessary. Through this incremental learning algorithm, the object detection system of this invention can adapt to new target categories while maintaining high detection accuracy for known categories, achieving continuous evolution and learning of the model.

[0106] Please see Figure 4 , Figure 4 This is a schematic flowchart of an open-world object detection method based on an attention mechanism provided in an embodiment of this application.

[0107] like Figure 4 As shown, the open-world object detection method based on the attention mechanism specifically includes steps S401 to S403.

[0108] S401. Obtain historical feature maps to be detected and historical target detection results;

[0109] S402. Based on a preset pre-trained model, target detection is performed on the historical feature map to be detected to obtain the detection accuracy and detection confidence of the target detection.

[0110] S403. When the detection accuracy is less than a preset accuracy threshold and / or the detection confidence is less than a preset confidence threshold, the model parameters are adjusted until the detection accuracy is not less than the preset accuracy threshold and the detection confidence is not less than the preset confidence threshold, and then the pre-trained model is determined as the target detection model.

[0111] In one embodiment, a probabilistic calibration method is used to improve the accuracy of model prediction probabilities. This method combines historical data and model performance, using a calibration algorithm to precisely adjust the probabilities output by the model. This algorithm first uses maximum likelihood estimation to determine initial estimates of model parameters, then applies Bayesian updates, continuously refining these parameter estimates by incorporating prior knowledge and newly collected data. Next, the predicted probabilities are modeled by fitting an appropriate statistical distribution, such as a Gaussian or Beta distribution, to reflect the confidence level of predictions for different categories. Based on this model, the original predicted probabilities are adjusted to correct biases and improve accuracy. Furthermore, performance evaluation metrics, such as precision, recall, and F1 score, are defined to quantitatively measure the calibration effect. New data is collected after each round of target detection, and parameter estimation is performed again to achieve iterative optimization. The entire process is managed by an automated system, ensuring real-time application of the calibration method and improving the accuracy and reliability of model predictions. By integrating this probabilistic calibration module, the confidence level and overall detection performance of the target detection model when handling unknown categories of targets are significantly improved.

[0112] In one embodiment, the probabilistic calibration subsystem is a sophisticated tuning tool specifically designed to improve the accuracy and confidence of the output probabilities of an object detection model. This subsystem relies on two data-driven approaches: Maximum Likelihood Estimation (MLE) and Bayesian methods. The subsystem first collects historical data, including predictions and actual annotations from previous object detection tasks. Using this data, the probabilistic calibration subsystem performs Maximum Likelihood Estimation, a parameter estimation method that estimates model parameters by selecting parameter values ​​that maximize the probability of observed data. In this process, MLE helps determine the initial shape of the model's predicted probability distribution. Subsequently, the probabilistic calibration subsystem further calibrates the parameters obtained from MLE using Bayesian methods. Bayesian methods update model parameters by combining prior knowledge and observed data, allowing the probabilistic calibration subsystem to dynamically adjust and refine predicted probabilities as new data becomes available. This approach not only considers the model's current predictive performance but also incorporates prior beliefs about the class distribution, thereby improving the model's confidence in detecting unknown or rare class targets. During the calibration process, the probabilistic calibration subsystem may also employ other statistical techniques or use more advanced techniques to handle uncertainties in multi-class settings. Ultimately, the probabilistic calibration subsystem outputs adjusted probabilities that more accurately reflect the model's confidence level in the target class. Through this calibration, the target detection system can provide more reliable and accurate detection results when facing complex scenes and unknown classes.

[0113] Please see Figure 5 , Figure 5 This application provides a schematic block diagram of an attention-based open-world object detection system, which is used to execute the aforementioned attention-based open-world object detection method. The attention-based open-world object detection system includes:

[0114] The image acquisition module is used to acquire images of the video stream to be detected;

[0115] The feature selection module is used to process the video stream image to be detected based on a preset target detection model to obtain a feature map to be detected, and to perform target detection on the feature map to obtain an initial detection result.

[0116] The region determination module is used to determine the region where the unknown category target is located in the feature map to be detected based on the initial detection results;

[0117] The target detection module is used to adjust the attention weight of the region where the unknown category target is located based on the target detection model, and to perform target detection on the feature map to be detected based on the attention weight, so as to obtain the recognition result of the unknown category target.

[0118] In one embodiment, the feature selection module includes:

[0119] The feature map set acquisition unit is used to process the video stream image to be detected based on the target detection model to obtain a multi-size feature map set;

[0120] The feature map determination unit is used to extract features from each multi-size feature map in the multi-size feature map set, obtain feature information of the target in each multi-size feature map, and determine the feature map to be detected in the multi-size feature map set based on the feature information.

[0121] In one embodiment, the feature selection module further includes:

[0122] The statistics and calculation unit is used to perform gradient magnitude calculation and activation value statistics on each of the multi-scale feature maps based on the feature information, so as to obtain the gradient magnitude value and activation value information corresponding to each of the multi-scale feature maps respectively.

[0123] The feature map determination unit is used to determine the feature map to be detected from each of the multi-size feature maps based on the gradient magnitude value and the activation value information.

[0124] In one embodiment, the region determination module includes:

[0125] The probability distribution information acquisition unit is used to obtain the probability distribution information of the target category in the feature map to be detected based on the initial detection result;

[0126] An uncertainty calculation unit is used to calculate the uncertainty of each region in the feature map to be detected based on the probability distribution information.

[0127] An uncertainty comparison unit is used to compare the uncertainty with a preset uncertainty threshold.

[0128] A region determination unit is used to determine the region as the location of the unknown category target when the uncertainty is greater than the uncertainty threshold.

[0129] In one embodiment, the target detection module includes:

[0130] A visual feature acquisition unit is used to acquire the linguistic description information of the unknown category target, and acquire the visual features of the unknown category target based on the feature information of the unknown category target;

[0131] The language feature acquisition unit is used to perform natural language processing on the language description information to obtain the language features corresponding to the language description information, which are used as the language features of the unknown category target.

[0132] A fusion feature acquisition unit is used to fuse the visual features and the linguistic features to obtain the fused features;

[0133] The recognition result acquisition unit is used to perform target detection on the feature map to be detected based on the attention weight and the fusion feature, and obtain the recognition result of the target of the location category.

[0134] In one embodiment, the attention-based open-world object detection system further includes a model update module, which comprises:

[0135] The new category target determination unit is used to mark the target as a new category target when there is a target with a confidence level less than a preset confidence level threshold in the initial detection results;

[0136] The category label acquisition unit is used to acquire the category label of the new category target based on the recognition result of the unknown category target;

[0137] A dataset generation unit is used to generate a new category target dataset based on the new category target and the category label;

[0138] The model update unit is used to update the parameters of the target detection model based on a preset formula and the new category target dataset, so as to update the target detection model.

[0139] In one embodiment, the preset formula is:

[0140]

[0141] Where, θ t These are the parameters of the model at the t-th iteration, and η is the learning rate. It is a loss function The gradient with respect to parameter θ, It is a new category target dataset.

[0142] In one embodiment, the attention-based open-world object detection system further includes a probability calibration subsystem, which comprises:

[0143] The historical data acquisition module is used to acquire historical feature maps to be detected and historical target detection results;

[0144] The accuracy acquisition module is used to perform target detection on the historical feature map to be detected based on a preset pre-trained model, and to obtain the detection accuracy and detection confidence of the target detection.

[0145] The model determination module is used to adjust the model parameters when the detection accuracy is less than a preset accuracy threshold and / or the detection confidence is less than a preset confidence threshold, until the detection accuracy is not less than the preset accuracy threshold and the detection confidence is not less than the preset confidence threshold, and then determine the pre-trained model as the target detection model.

[0146] In one embodiment, the integration and management of the attention-based open-world object detection system involves building a highly integrated software platform designed to coordinate and synchronize various hardware components and data processing flows to ensure the efficient operation of the object detection system. This software platform is meticulously designed to run on electronic devices equipped with high-performance processors, including multi-core CPUs and high-performance GPUs optimized for parallel processing and graphics rendering. To support rapid data processing and response, the system is equipped with high-speed RAM to prevent data corruption and ensure stability, while also featuring large-capacity SSDs for storing the operating system, applications, model parameters, and log data. Furthermore, the system integrates multiple communication interfaces, including a high-speed Ethernet interface for stable wired network connectivity, Wi-Fi 6 or Bluetooth 5.0 modules for wireless connectivity, and, possibly, a cellular network module for data exchange with external devices or networks. These communication interfaces support rapid data transmission, allowing the system to exchange and communicate with cameras, information systems, and other external devices in real time. In terms of system management, the software platform provides a centralized user interface, which can be a graphical user interface (GUI) or a command-line interface (CLI), enabling users to easily configure system parameters, monitor system status, start and stop target detection tasks, view detection results, and manage data streams. Furthermore, the system includes logging, error detection, and recovery mechanisms to ensure system reliability and robustness. Security is also a key part of system integration and management, including implementing security measures such as data encryption, access control lists (ACLs), user authentication, and audit trails to protect the system from unauthorized access and potential network threats. Through this comprehensive system integration and management, the target detection system of this invention achieves a high degree of automation, intelligence, and user-friendliness, while maintaining operational flexibility and data security.

[0147] In one embodiment, the attention-based open-world object detection system also includes a user interface, i.e., an intuitive and feature-rich front-end system that provides users with a direct way to integrate and manage the object detection system. This interface, as a graphical user interface (GUI), runs on a high-resolution touchscreen and provides a range of tools and options, enabling users to easily interact with the system. The main control panel displays a system status overview, including CPU and GPU usage, memory consumption, storage space, and network connectivity. Users can access more detailed system monitoring data through simple touch or mouse clicks. In the configuration module, users can set system parameters such as image acquisition resolution, frame rate, selection of preprocessing algorithms, parameter adjustments for deep learning models, sensitivity of the attention mechanism, and learning rate of incremental learning algorithms. These parameters are set through an easy-to-understand and operate form containing controls such as drop-down menus, sliders, and input boxes. The task control center allows users to start, pause, or stop object detection tasks. Users can perform detection by selecting predefined task configurations or creating new task configurations, and can set priorities and resource allocations for each task. The performance monitoring tool provides real-time performance feedback, including key performance indicators such as processing time, detection accuracy, and system throughput. This data is presented in the form of charts and digital dashboards, allowing users to quickly grasp the system's performance status. The results viewer provides users with a visual interface to view target detection results. Users can browse the list of detected targets and view the bounding box, category label, and confidence score for each target. Furthermore, the interface supports interactive operations such as zooming, panning, and tagging, helping users to perform detailed analysis of the results. Security and privacy settings allow users to manage security-related configurations such as data encryption, access permissions, and user authentication, ensuring system security and data confidentiality. The help and support center provides users with abundant documentation resources, frequently asked questions (FAQs), online tutorials, and customer service contact information so that users can get help when needed. The entire user interface design prioritizes user experience, ensuring intuitive and convenient operation, allowing both professional and non-professional users to quickly get started and efficiently manage and use the target detection system.

[0148] In one embodiment, the attention-based open-world target detection system further includes a security and privacy protection subsystem: multi-layered security functions are integrated into the design to ensure data security and privacy. First, the system uses Advanced Encryption Standard (AES) to encrypt all sensitive data, ensuring confidentiality during transmission and storage. Second, a Role-Based Access Control (RBAC) system is implemented to ensure that only authorized users can access system resources at the appropriate level. Furthermore, the system deploys an audit log function to record all user actions and system events for post-event tracking and analysis. These comprehensive measures together constitute a robust security protection system, effectively preventing unauthorized access and data leakage, and protecting the security of system and user data.

[0149] In one embodiment, the attention-based open-world object detection system also includes a system evaluation and optimization tool: this tool is executed automatically on a regular basis to comprehensively evaluate the performance of the object detection system. It measures system performance using a series of quantitative indicators, such as detection accuracy, response time, and resource utilization, and compares it with preset performance benchmarks. The evaluation results generate detailed reports that include not only current performance data but also system optimization suggestions. These suggestions may involve adjusting algorithm parameters, upgrading hardware configurations, or improving data processing flows. Furthermore, the tool has real-time monitoring capabilities, enabling it to issue timely warnings when system performance degrades and provide quick recovery or optimization solutions to ensure the system maintains maximum efficiency in dynamically changing application environments.

[0150] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the system and each module described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0151] The above-described system can be implemented as a computer program, which can be used in, for example... Figure 6 It runs on the computer device shown.

[0152] Please see Figure 6 , Figure 6 This is a schematic block diagram illustrating the structure of a computer device according to an embodiment of this application. The computer device may be a server.

[0153] See Figure 6 The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0154] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any attention-based open-world object detection method.

[0155] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0156] Internal memory provides an environment for the execution of computer programs in non-volatile storage media. When the computer program is executed by the processor, it enables the processor to execute any open-world object detection method based on an attention mechanism.

[0157] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0158] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0159] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:

[0160] Acquire the image of the video stream to be detected;

[0161] Based on a preset target detection model, the video stream image to be detected is processed to obtain a feature map to be detected, and target detection is performed on the feature map to obtain an initial detection result.

[0162] Based on the initial detection results, the region where the unknown category target is located in the feature map to be detected is determined;

[0163] Based on the target detection model, the attention weights of the region where the unknown category target is located are adjusted, and based on the attention weights, target detection is performed on the feature map to be detected to obtain the recognition result of the unknown category target.

[0164] In one embodiment, when the processor processes the video stream image to be detected based on a preset target detection model to obtain a feature map to be detected, it is used to:

[0165] Based on the target detection model, the video stream image to be detected is processed to obtain a set of feature maps of multiple sizes;

[0166] Feature extraction is performed on each multi-size feature map in the multi-size feature map set to obtain the feature information of the target in each multi-size feature map, and based on the feature information, the feature map to be detected is determined in the multi-size feature map set.

[0167] In one embodiment, when the processor determines the feature map to be detected from the multi-size feature map set based on the feature information, it is configured to:

[0168] Based on the feature information, gradient magnitude calculation and activation value statistics are performed on each of the multi-scale feature maps to obtain the gradient magnitude value and activation value information corresponding to each of the multi-scale feature maps.

[0169] Based on the gradient magnitude value and the activation value information, the feature map to be detected is determined in each of the multi-size feature maps.

[0170] In one embodiment, when the processor determines the region containing an unknown category target in the feature map to be detected based on the initial detection result, it is configured to:

[0171] Based on the initial detection results, the probability distribution information of the target category in the feature map to be detected is obtained;

[0172] The uncertainty of each region in the feature map to be detected is calculated based on the probability distribution information.

[0173] The uncertainty is compared with a preset uncertainty threshold;

[0174] When the uncertainty is greater than the uncertainty threshold, the region is designated as the region where the unknown category target is located.

[0175] In one embodiment, when the processor adjusts the attention weights of the region where the unknown category target is located based on the target detection model, and performs target detection on the feature map to be detected based on the attention weights to obtain the recognition result of the unknown category target, it is configured to:

[0176] Obtain the linguistic description information of the unknown category target, and based on the feature information of the unknown category target, obtain the visual features of the unknown category target;

[0177] Natural language processing is performed on the language description information to obtain the language features corresponding to the language description information, which are used as the language features of the unknown category target.

[0178] The visual features and linguistic features are fused to obtain the fused features;

[0179] Based on the attention weights and the fusion features, target detection is performed on the feature map to be detected to obtain the recognition result of the target of the location category.

[0180] In one embodiment, after the processor adjusts the attention weights of the region where the unknown category target is located based on the target detection model, and performs target detection on the feature map to be detected based on the attention weights to obtain the recognition result of the unknown category target, it is further configured to implement:

[0181] If there are targets with a confidence level lower than a preset confidence threshold in the initial detection results, the targets are marked as new category targets;

[0182] Based on the identification results of the unknown category target, obtain the category label of the new category target;

[0183] Based on the new category target and the category label, generate a new category target dataset;

[0184] Based on the preset formula and the new category target dataset, the parameters of the target detection model are updated to update the target detection model.

[0185] In one embodiment, the preset formula is:

[0186]

[0187] Where, θ t These are the parameters of the model at the t-th iteration, and η is the learning rate. It is a loss function The gradient with respect to parameter θ, It is a new category target dataset.

[0188] In one embodiment, before the processor processes the video stream image to be detected based on a preset target detection model to obtain a feature map to be detected, and performs target detection on the feature map to obtain an initial detection result, it is also configured to:

[0189] Obtain historical feature maps to be detected and historical target detection results;

[0190] Based on a pre-trained model, target detection is performed on the historical feature map to be detected, and the detection accuracy and detection confidence of the target are obtained.

[0191] When the detection accuracy is less than a preset accuracy threshold and / or the detection confidence is less than a preset confidence threshold, the model parameters are adjusted until the detection accuracy is not less than the preset accuracy threshold and the detection confidence is not less than the preset confidence threshold, at which point the pre-trained model is determined as the target detection model.

[0192] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the attention-based open-world object detection methods provided in the embodiments of this application.

[0193] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD), flash memory card, etc., provided on the computer device.

[0194] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An open-world object detection method based on an attention mechanism, characterized in that, include: Acquire the image of the video stream to be detected; Based on a preset target detection model, the video stream image to be detected is processed to obtain a feature map to be detected, and target detection is performed on the feature map to obtain an initial detection result. Based on the initial detection results, the region where the unknown category target is located in the feature map to be detected is determined; Based on the target detection model, the attention weights of the region where the unknown category target is located are adjusted, and based on the attention weights, target detection is performed on the feature map to be detected to obtain the recognition result of the unknown category target; The step of determining the region containing the unknown category target in the feature map to be detected based on the initial detection result includes: Based on the initial detection results, the probability distribution information of the target category in the feature map to be detected is obtained; The uncertainty of each region in the feature map to be detected is calculated based on the probability distribution information. The uncertainty is compared with a preset uncertainty threshold; When the uncertainty is greater than the uncertainty threshold, the region is designated as the region where the unknown category target is located. The step of adjusting the attention weights of the region where the unknown category target is located based on the target detection model, and performing target detection on the feature map to be detected based on the attention weights to obtain the recognition result of the unknown category target, includes: Obtain the linguistic description information of the unknown category target, and based on the feature information of the unknown category target, obtain the visual features of the unknown category target; Natural language processing is performed on the language description information to obtain the language features corresponding to the language description information, which are used as the language features of the unknown category target. The visual features and language features are fused to obtain fused features; Based on the attention weights and the fusion features, target detection is performed on the feature map to be detected to obtain the recognition result of the unknown category target.

2. The open-world object detection method based on attention mechanism according to claim 1, characterized in that, The process of processing the video stream image to be detected based on the preset target detection model to obtain the feature map to be detected includes: Based on the target detection model, the video stream image to be detected is processed to obtain a set of feature maps of multiple sizes; Feature extraction is performed on each multi-size feature map in the multi-size feature map set to obtain the feature information of the target in each multi-size feature map, and based on the feature information, the feature map to be detected is determined in the multi-size feature map set.

3. The open-world object detection method based on an attention mechanism according to claim 2, characterized in that, The step of determining the feature map to be detected from the multi-size feature map set based on the feature information includes: Based on the feature information, gradient magnitude calculation and activation value statistics are performed on each of the multi-size feature maps to obtain the gradient magnitude value and activation value information corresponding to each of the multi-size feature maps. Based on the gradient magnitude value and the activation value information, the feature map to be detected is determined in each of the multi-size feature maps.

4. The open-world object detection method based on attention mechanism according to claim 1, characterized in that, After adjusting the attention weights of the region where the unknown category target is located based on the target detection model, and performing target detection on the feature map to be detected based on the attention weights to obtain the recognition result of the unknown category target, the method further includes: If there are targets with a confidence level lower than a preset confidence threshold in the initial detection results, the targets are marked as new category targets; Based on the identification results of the unknown category target, obtain the category label of the new category target; Based on the new category target and the category label, generate a new category target dataset; Based on the preset formula and the new category target dataset, the parameters of the target detection model are updated to update the target detection model.

5. The open-world object detection method based on an attention mechanism according to claim 4, characterized in that, The preset formula is: in, These are the parameters of the model at the t-th iteration. It's the learning rate. It is a loss function For parameters gradient, It is a new category target dataset.

6. The open-world object detection method based on an attention mechanism according to any one of claims 1-5, characterized in that, Before processing the video stream image to be detected based on the preset target detection model to obtain a target feature map, and performing target detection on the target feature map to obtain the initial detection result, the method further includes: Obtain historical feature maps to be detected and historical target detection results; Based on a pre-trained model, target detection is performed on the historical feature map to be detected, and the detection accuracy and detection confidence of the target are obtained. When the detection accuracy is less than a preset accuracy threshold and / or the detection confidence is less than a preset confidence threshold, the model parameters are adjusted until the detection accuracy is not less than the preset accuracy threshold and the detection confidence is not less than the preset confidence threshold, at which point the pre-trained model is determined as the target detection model.

7. A computer device, characterized in that, The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, in executing the computer program, implement the open-world object detection method based on the attention mechanism as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the open-world object detection method based on the attention mechanism as described in any one of claims 1 to 6.

Images