Article detection method and device in monitoring scene based on open vocabularies

Abstract

The embodiment of the invention provides an article detection method and device in a monitoring scene based on open vocabularies, and the method comprises the steps: extracting video data of the monitoring scene to obtain image data, carrying out the labeling of a to-be-detected article, generating a labeled json file, carrying out the data integration according to a data normal form, generating standardized text data and standardized image data, and carrying out the detection of the to-be-detected article. The method comprises the following steps: firstly obtaining a text embedding expression and an image embedding expression, further obtaining a text embedding expression and an image embedding expression, then calculating the similarity of the text embedding expression and the image embedding expression, obtaining a classification alignment score based on the similarity, calculating a loss function through the classification alignment score, and performing training or fine tuning by utilizing the loss function to obtain an article detection model. According to the object detection method and device, the objects appearing in the training process can be detected, the objects which do not appear can also be detected, the scene migration capability is high, and the objects can be migrated to different monitoring view angle scenes.

Description

Technical Field

[0001] The present application relates to the technical field of object detection, and in particular to an object detection method and device in a monitoring scenario based on an open vocabulary. Background Art

[0002] In the field of modern security monitoring and object detection, traditional object detection methods such as Fast R-CNN, Faster R-CNN, Mask R-CNN and DETR have been widely used. These methods are usually trained based on data sets with closed categories, so they perform well in identifying targets within predefined categories. However, this limitation makes it difficult for them to effectively handle new types or unknown objects that did not appear in the training phase, which poses a major challenge to practical applications. To address this problem, Open Vocabulary Detection (OVD) was proposed as an emerging technology and quickly gained widespread attention in academia and industry. The core of OVD is to enable the system to use category names to identify any type of objects, even if these objects have never appeared in the training process. This technology breaks through the limitations of traditional detection methods and greatly expands the scope of application and flexibility of the system.

[0003] Some existing patent documents show different methods and technologies to improve the detection performance in specific scenarios. For example, WO2021073370A1 describes a security inspection image processing solution, which converts the scanned image into a grayscale image, performs denoising, background removal and feature extraction steps, and finally uses a lightweight neural network model to determine whether the image contains contraband. The invention aims to improve the efficiency and accuracy of automatic detection during security inspections. CN109829429B provides a security monitoring solution based on the YOLOv3 architecture, which is particularly suitable for detecting sensitive items such as knives and guns. This method includes collecting relevant sample images and annotating them, and then implementing a data enhancement strategy to optimize the training effect of the YOLOv3 model. The trained model can efficiently and accurately identify specified objects from the video stream. WO2020010921A1 introduces an object recognition system that combines gas chromatography and ion migration tube technology. The system first concentrates and samples the sample molecules and performs preliminary separation, and then generates a fingerprint spectrum that can be used to identify the type of object. This method not only improves the detection speed and accuracy, but also reduces the false alarm rate.

[0004] Although the above technologies and methods have achieved certain successes in their respective application scopes, they still have significant shortcomings when facing complex and changing real-world environments. In particular, when cross-scene migration or reliable detection of unseen objects is required, current methods often fail to achieve the desired performance level. Therefore, developing a more general and powerful open object recognition technology has become one of the key issues that need to be addressed. Summary of the invention

[0005] The present application provides an object detection method and device in a monitoring scenario based on an open vocabulary, which can not only detect objects that appear in the training process, but also detect objects that have not appeared, and has a high scene migration capability and can migrate to different monitoring perspective scenes.

[0006] In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0007] In a first aspect, an embodiment of the present application provides an object detection method in a monitoring scenario based on an open vocabulary, the method comprising:

[0008] Use the camera to extract the collected surveillance scene video data according to the frame rate to obtain image data, mark the objects to be detected in the image data to generate a marked json file, and save the image data and json file;

[0009] Perform data processing on the image data and the json file respectively to obtain the processed data;

[0010] Perform data integration on the processed data according to the data paradigm to generate standardized text data and standardized image data;

[0011] Using a text encoder to convert text information of standardized text data to obtain text embedding representation, and using an image encoder to convert image information of standardized image data to obtain image embedding representation;

[0012] Calculate the similarity between the text embedding representation and the image embedding representation, and obtain the classification alignment score based on the similarity;

[0013] The loss function is calculated through classification alignment scores, and the object detection model is obtained by training or fine-tuning the loss function.

[0014] The object detection model is used to evaluate the real labeled data and object detection data of the received test data to obtain corresponding evaluation indicators.

[0015] In a possible implementation, the step of calculating the similarity between the text embedding representation and the image embedding representation and obtaining a classification alignment score based on the similarity includes:

[0016] Calculate the similarity between the text embedding representation and the image embedding representation, and filter out the image text embedding by similarity;

[0017] The image text embedding is processed using a transformer-based decoder to obtain the decoder output result;

[0018] The decoder output is input into the classification layer to predict the target box category and regress the bounding box to obtain the classification alignment score.

[0019] In a possible implementation, the step of calculating a loss function by using the classification alignment score and using the loss function to train or fine-tune an object detection model includes:

[0020] The classification loss is calculated using the classification alignment score and the true classification value, and the regression loss is calculated using the regression bounding box and the bounding box truth value;

[0021] Use classification loss and regression loss to form a loss function;

[0022] The object detection model is obtained by training or fine-tuning the loss function. The training hyperparameters include batch size, training iters, optimizerAdamW, weight decay, learning rate of image-text encoder, and hidden layer dimension.

[0023] In a possible implementation, the steps of respectively processing the image data and the json file to obtain processed data include:

[0024] Performing data screening on the image data and the json file respectively to obtain screened data, wherein the data screening includes fuzzy data and data with incorrect annotations;

[0025] The filtered data is cleaned and enhanced to obtain the processed data.

[0026] In one possible implementation, the text encoder includes a traditional encoder, an embedding encoder, and a context-aware encoder, and the image encoder includes a CNN-based image encoder, a Transformer-based image encoder, a generative adversarial network-based image encoder, and an autoencoder.

[0027] In a second aspect, the present application further proposes an object detection device in a monitoring scenario based on an open vocabulary, the device comprising:

[0028] The data acquisition module is used to use the camera to extract the collected surveillance scene video data according to the frame rate to obtain image data, annotate the objects to be detected in the image data to generate annotated json files, and save the image data and json files;

[0029] A data processing module is used to process the image data and the json file to obtain processed data;

[0030] A data integration module is used to integrate the processed data according to the data paradigm to generate standardized text data and standardized image data;

[0031] A text image encoding module, used to convert text information of standardized text data using a text encoder to obtain a text embedding representation, and to convert image information of standardized image data using an image encoder to obtain an image embedding representation;

[0032] The text-image fusion alignment module is used to calculate the similarity between the text embedding representation and the image embedding representation, and obtain the classification alignment score based on the similarity;

[0033] The model training optimization module is used to calculate the loss function through the classification alignment score, and use the loss function to train or fine-tune the object detection model;

[0034] The model evaluation module is used to use the object detection model to evaluate the real labeled data and object detection data of the received test data to obtain corresponding evaluation indicators.

[0035] In a possible implementation, the text-image fusion alignment module is further used for:

[0036] Calculate the similarity between the text embedding representation and the image embedding representation, and filter out the image text embedding by similarity;

[0037] The image text embedding is processed using a transformer-based decoder to obtain the decoder output result;

[0038] The decoder output is input into the classification layer to predict the target box category and regress the bounding box to obtain the classification alignment score.

[0039] In a possible implementation, the model training optimization module is further used to:

[0040] The classification loss is calculated using the classification alignment score and the true classification value, and the regression loss is calculated using the regression bounding box and the bounding box truth value;

[0041] Use classification loss and regression loss to form a loss function;

[0042] The object detection model is obtained by training or fine-tuning the loss function. The training hyperparameters include batch size, training iters, optimizerAdamW, weight decay, learning rate of image-text encoder, and hidden layer dimension.

[0043] In a possible implementation manner, the data processing module is further used to:

[0044] Performing data screening on the image data and the json file respectively to obtain screened data, wherein the data screening includes fuzzy data and data with incorrect annotations;

[0045] The filtered data is cleaned and enhanced to obtain the processed data.

[0046] In one possible implementation, the text encoder includes a traditional encoder, an embedding encoder, and a context-aware encoder, and the image encoder includes a CNN-based image encoder, a Transformer-based image encoder, a generative adversarial network-based image encoder, and an autoencoder.

[0047] The above-mentioned main scheme of the present application and its further options can be freely combined to form multiple schemes, all of which are schemes that can be adopted and claimed for protection in the present application; and in the present application, (non-conflicting options) options and other options can also be freely combined. After understanding the scheme of the present application, those skilled in the art can understand that there are multiple combinations based on the prior art and common knowledge, all of which are technical schemes to be protected by the present application, and they are not exhaustively listed here.

[0048] Compared with the prior art, the embodiments of the present application provide a method and device for object detection in a monitoring scenario based on an open vocabulary, extract the monitoring scene video data to obtain image data, annotate the objects to be detected to generate annotated json files, and then perform data processing, perform data integration according to the data paradigm, generate standardized text data and standardized image data, further obtain text embedding representation and image embedding representation, and then calculate the similarity between the text embedding representation and the image embedding representation, and obtain a classification alignment score based on the similarity, calculate the loss function through the classification alignment score, use the loss function to train or fine-tune to obtain an object detection model, and use the object detection model to evaluate and obtain corresponding evaluation indicators. It can not only detect objects that appear in the training process, but also detect objects that have not appeared, and has high scene migration capability and can migrate to different monitoring perspective scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1A flow chart of an object detection method in a monitoring scenario based on an open vocabulary proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0052] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0053] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0054] Traditional object detection methods, such as Fast R-CNN, Faster R-CNN, Mask R-CNN, DETR, etc., are usually trained on datasets with closed-set categories, which limits their ability to detect objects outside of predefined categories, which is a significant limitation for real-world applications. To address this limitation, a new task called Open Vocabulary Detection (OVD) was proposed, which requires the ability to detect any object using class names, even objects that have never been encountered during training.

[0055] The existing technology has average effect in the direction of object detection under the monitoring perspective, and has poor scene migration ability and weak detection ability for unknown objects.

[0056] Therefore, in order to solve the above-mentioned technical problems, the embodiment of the present application proposes a method and device for object detection in a monitoring scenario based on an open vocabulary, which has high accuracy, strong scene migration capability, and strong generalization ability. In addition to target detection of known objects, it has a certain ability to detect unknown objects, which is described in detail below.

[0057] Please refer to Figure 1 , Figure 1 A flow chart of an object detection method in a monitoring scenario based on an open vocabulary proposed in an embodiment of the present application is shown, and the method includes:

[0058] Step S1: Use a camera to extract the collected surveillance scene video data according to the frame rate to obtain image data, mark the objects to be detected in the image data to generate a marked json file, and save the image data and the json file.

[0059] Use multiple cameras (such as security cameras) to cover different monitoring perspectives, extract one frame of image per second from the video stream of each camera to reduce storage space requirements while retaining enough information, extract the video data collected by each camera at a frequency of one frame per second, and generate a series of static images. These images can be saved on disk in common image formats (such as JPEG or PNG).

[0060] Use image annotation tools (such as LabelImg, VGG Image Annotator (VIA), etc.) to annotate the objects to be detected in the image, draw a bounding box for each target object, and assign it a category label. For example, if the target is "person", "table" or "chair", add these labels in the corresponding bounding box, and export all annotation information to JSON format files. Each JSON file contains information such as image path, bounding box coordinates, and corresponding category labels.

[0061] Save the extracted images in a specified folder, make sure the file names are unique and in order, save the generated JSON files together with the corresponding image files in the same directory, or manage them in another separate folder.

[0062] Step S2: perform data processing on the image data and the json file respectively to obtain processed data.

[0063] Step S2 includes:

[0064] Performing data screening on the image data and the json file respectively to obtain screened data, wherein the data screening includes fuzzy data and data with incorrect annotations;

[0065] The filtered data is cleaned and enhanced to obtain the processed data.

[0066] Data screening is to remove data that does not meet the requirements from the original data set. Since image data involves identifying and removing blurry or incorrectly labeled images, the data in the JSON file needs to be checked for logical errors, format errors, etc., and these problematic data items need to be removed.

[0067] The data is then further cleaned to improve its quality, including correcting minor errors that can be automatically fixed by rules (such as unifying the date format), filling in missing values, or standardizing numerical features so that they are on the same scale.

[0068] After preparing a clean and effective training set, we can increase sample diversity and improve the generalization ability of the model through data augmentation. Images can be rotated, flipped, scaled, and noise can be added.

[0069] Step S3: perform data integration on the processed data according to the data paradigm to generate standardized text data and standardized image data.

[0070] First, convert data from different sources and formats into a common format. For image and text data, a standard data structure needs to be defined that is compatible with all input types. For example, for each record, it should contain three main parts: an image, a set of target boxes (if applicable), and related text information. For the target boxes in the image-text pair, their coordinate values ​​are adjusted to a proportional representation of the target image size, so that no matter how the original image size changes, its annotations are still valid, ensuring that all images are adjusted to the same resolution or aspect ratio for consistency in subsequent processing.

[0071] Based on a given template (such as a photo of {label}), a descriptive text is generated for each detected object, where {label} will be replaced with the actual object category name, ensuring that each sample has an accurate and consistent text description corresponding to it, and finally obtaining a complete and uniformly formatted dataset.

[0072] Step S4: using a text encoder to convert text information of the standardized text data to obtain a text embedding representation, and using an image encoder to convert image information of the standardized image data to obtain an image embedding representation.

[0073] Text encoders include traditional encoders, embedded encoders, and context-aware encoders. Image encoders include CNN-based image encoders, Transformer-based image encoders, generative adversarial network-based image encoders, and autoencoders.

[0074] The text encoder converts natural language text into a numerical form that can be understood and used by machine learning or deep learning models, mapping the text to a high-dimensional space where each dimension represents a certain degree of semantic features. In this way, even very complex text information can be simplified and presented in a way that is easier to analyze. The image encoder extracts key features from the image and converts them into a fixed-length vector form. After training on large-scale image datasets, it can recognize many basic and even high-level visual patterns, and based on this, it generates a unique feature vector for each image.

[0075] The text encoders in the embodiments of the present application include traditional encoders (including bag-of-words models, TF-IDF models), embedded encoders (including Word2Vec, GloVe, FastText, etc.), context-aware encoders (including ELMo, BERT, RoBERTa, etc.), and the image encoders in the embodiments of the present application include CNN-based image encoders (such as VGG, ResNet, Inception, etc.), Transformer-based image encoders (such as Vision Transformer and SwinTransformer), generative adversarial network-based image encoders and autoencoders, etc.

[0076] Preferably, the embodiment of the present application can use the Bert model as the text encoder and the Swin Transformer encoder, and the model used is mainly the Swin-T model. Swin Transformer is based on the Transformer structure, mainly using the self-attention mechanism for feature extraction. Unlike traditional CNN, it can more effectively capture long-range dependencies. The main components are Patch Embedding, Swin TransformerBlock, LayerNormalization, feature fusion, output layer, etc.

[0077] Step S5: Calculate the similarity between the text embedding representation and the image embedding representation, and obtain a classification alignment score based on the similarity.

[0078] Use the embedded representations of text and images to find matching image-text pairs. A Transformer decoder framework with self-attention and cross-attention mechanisms is used to integrate the data of these two modalities. Category recognition and location positioning are performed based on the decoder results, and quality assessment is performed by comparing the correlation between the text content and the prediction results again.

[0079] Step S5 includes:

[0080] Calculate the similarity between the text embedding representation and the image embedding representation, and filter out the image text embedding by similarity;

[0081] The image text embedding is processed using a transformer-based decoder to obtain the decoder output result;

[0082] The decoder output is input into the classification layer to predict the target box category and regress the bounding box to obtain the classification alignment score.

[0083] The text embedding vector generated by the text encoder (such as BERT) and the image embedding vector generated by the image encoder (such as SwinTransformer) are then used to calculate the similarity between each text embedding and all image embeddings. The most relevant image-text pairs are screened out according to preset thresholds or other strategies (such as selecting the top N most similar pairs) and processed using the Transformer decoder. The Transformer decoder includes a self-attention layer, a cross-attention layer, and a feedforward layer. The self-attention layer captures the information flow within the sequence, and the cross-attention layer enables the decoder to notice the features generated by the encoder (image-text embedding), thereby combining the information of the two modalities. The feedforward layer introduces nonlinear transformations to the model to enhance the expressiveness, and finally obtains a new representation (decoder output result) that integrates image and text information, which includes both visual features and reflects the language context.

[0084] The decoder output result is input into the classification layer to predict the category O to which the object in the target box belongs. The regression layer estimates the regression bounding box B based on the output result of the decoder. The degree of consistency is measured by comparing the similarity between the category (O) output by the classification layer and the original text embedding, that is, the classification alignment score S is obtained, which is used to evaluate the accuracy of the model in understanding the text description and locating the corresponding image area.

[0085] Step S6: Calculate the loss function through the classification alignment score, and use the loss function to train or fine-tune to obtain the object detection model.

[0086] The target boxes in the input image are classified and located. For each target box, the model outputs a classification score S (indicating the probability that the object in the box belongs to a certain category) and a predicted bounding box B (indicating the location coordinates of the object). The classification loss is calculated based on the difference between the classification score S predicted by the model and the actual correct category label. If the prediction is very accurate, this part of the loss is small; otherwise, the loss is large. The regression loss is measured based on the difference between the target box position B predicted by the model and the actual target box position. The closer the position is to the actual value, the smaller the loss.

[0087] The above two parts of loss are added together to get the total loss, and the model parameters are adjusted through the back propagation algorithm to minimize this total loss. After multiple iterations, the model can complete the tasks of target detection (positioning) and classification.

[0088] Step S6 includes:

[0089] The classification loss is calculated using the classification alignment score and the true classification value, and the regression loss is calculated using the regression bounding box and the bounding box truth value;

[0090] Use classification loss and regression loss to form a loss function;

[0091] The object detection model is obtained by training or fine-tuning the loss function. The training hyperparameters include batch size, training iters, optimizerAdamW, weight decay, learning rate of image-text encoder, and hidden layer dimension.

[0092] The standardized data format of (image, target box set, text) is used as input, and features of the image and text are extracted respectively through the image encoder and text encoder. After text-image fusion alignment, the target box category and coordinates are output. The loss of the model mainly consists of two parts, including classification loss and regression loss. The classification loss of the model is calculated by using the predicted classification alignment score S and the true classification value, and the regression loss is calculated by using the regression bounding box B and the true value of the bounding box.

[0093] Step S7: Use the object detection model to evaluate the real labeled data and the object detection data of the received test data to obtain corresponding evaluation indicators.

[0094] By receiving the real labeled data and object detection data of the test data, evaluating the two, and calculating the corresponding evaluation indicators, we can fully understand the performance of the model based on the evaluation indicators, such as which category it performs better or worse in recognition, and whether it has achieved the expected effect overall.

[0095] Evaluation indicators include AP, mAP, mAP@0.5 and AR, where AP (Average Precision) is the average accuracy. For each category, the area under the Precision-Recall curve is calculated. mAP (mean of Average Precision) is the average of the AP values ​​of all categories to obtain a single number that comprehensively evaluates the performance of the model. mAP@0.5 is calculated when only the case where IOU is greater than or equal to 0.5 is considered. AR (Average Recall) is the average recall rate, which indicates the proportion of all objects correctly identified by the model to the total number.

[0096] During the model usage phase, the model is fine-tuned based on the pre-object detection model to obtain a fine-tuned model for a specific monitoring scenario. Each frame of the monitoring video is used as the input for the test, and the result of object detection is returned after the model output, thereby realizing object detection in the monitoring scene.

[0097] In a possible embodiment, the data collected from the monitoring perspective in the conference room is used as fine-tuning data to implement the technical implementation of the scheme of the present invention. First, Objects365 and GoldG are selected as pre-training data sets. Objects365 is the largest target detection data set at present, with a total of 365 categories and more than 40,000 images in the data set, covering different scenes and environments. The object categories of the data set include humans, animals, vehicles, daily necessities, etc. The main purpose of the GoldG data set is to provide a high-quality benchmark data set for computer vision tasks such as image classification, object detection and instance segmentation, covering a variety of object categories, such as humans, animals, vehicles, daily necessities, etc.

[0098] After obtaining the data, the sample data of each dataset is preprocessed, mainly including: First, remove noise: exclude irrelevant or low-quality images. Second, check annotations: ensure the accuracy of annotations and correct incorrect or inconsistent annotations. Third, data enhancement (geometric transformation: such as rotation, translation, scaling and flipping to increase the diversity of data. Color transformation: adjust brightness, contrast, saturation, etc. Cutting: randomly crop parts of the image. Add noise: such as Gaussian noise, salt and pepper noise, to increase the robustness of the model) Fourth, image scaling: adjust the image to a uniform size to meet the input requirements of the model.

[0099] The deep learning framework Detectron was used for model optimization. Table 1 shows the initial parameter settings:

[0100] Table 1

[0101]

[0102]

[0103] After the pre-training data is processed, the classification alignment score S and the true classification value are obtained, and the classification loss L is calculated cls , use the regression bounding box B and the bounding box ground truth to calculate the regression loss L box , we get the loss function: L = L cls +L box .

[0104] The training process is as follows: First, use the stochastic gradient descent method to calculate the updated weight and bias values ​​of the neural network. Second, after 10,000 training iterations, adjust the learning rate to 1e-4 and continue training. Third, stop training after 50,000 iterations and save the trained model.

[0105] The data set used in the model fine-tuning stage comes from data collected from the monitoring perspective in the conference room, which is to collect pictures from each frame of the monitoring video to obtain the original picture data to form the data set Data1.

[0106] After acquiring the data, the sample data of each data set is preprocessed. In addition to the data processing method in the model training stage, there are also screening and elimination of blurred images and screening of images blocked by objects to be detected.

[0107] The data set is then split into 24,570 images in a ratio of 8:1:1 to obtain training set, test set, and validation set.

[0108] Data labeling: Download and install LabelMe and configure it. The labeled items include notebooks, special pens, special spoons, tissues, tissue boxes, and A4 paper. Manually label the preprocessed images and generate json files.

[0109] Model optimization: The pre-trained model is fine-tuned using LoRA-based fine-tuning technology. This fine-tuning method adjusts the behavior of the model by introducing a low-rank matrix to improve the performance of the model on new tasks. Specifically, LoRA adds two bypass matrices A and B to the original pre-trained model. The dimensions of these two matrices are much smaller than the input and output dimensions of the original model, thereby achieving efficient fine-tuning of parameters.

[0110] The model optimization process is similar to the optimization process during model training. Table 2 shows the parameter changes:

[0111] Table 2

[0112]

[0113]

[0114] During the model usage phase, the saved optimal fine-tuning model is read and each frame image under the monitoring perspective is used as input. The network resizes each small input image to 640*640 size and outputs six basic parameters: prediction box category and coordinates (x, y, w, h, confidence, l). Among them, (x, y) is the center coordinate of the bounding box, (w, h) is the width and height of the bounding box, confidence is the confidence, and l is the prediction category.

[0115] Read the saved optimal fine-tuning model, use the validation set as input, and obtain the overall evaluation indicators of the model shown in Table 3:

[0116] Table 3

[0117] Evaluation Metrics value mAP 0.823 mAP@0.5 0.987 AR 0.870

[0118] The generalization test of the model was carried out on items that did not appear in the fine-tuning dataset, such as paper cups and masks. Image data was randomly downloaded from the Internet as model input, and the predicted box category and coordinates were output for visualization. The experiment showed that it also had good detection capabilities for these items that did not appear.

[0119] First, pre-train the deep learning model on a large-scale dataset containing a large number of images. Use a dedicated data acquisition system to obtain image or video data from actual surveillance environments (such as conference rooms, security cameras, etc.). Manually annotate the target objects in these images to create the label data required for training and testing. Increase data diversity through methods such as rotation, scaling, and brightness adjustment to make the model more robust. Remove poor quality or incorrectly annotated data to ensure the quality of training data. Save the processed annotation information for subsequent use. Use previously collected and processed data in a specific field (such as data from a surveillance perspective) to fine-tune the pre-trained model. After all training and fine-tuning are completed, the final model is applied to the actual surveillance video stream. Specifically, each frame of the surveillance video is used as input, and the model outputs the object detection results in each frame.

[0120] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0121] First, traditional object detection models usually need to be trained for a specific set of object categories, which limits their ability to recognize new or unseen objects. The open vocabulary-based approach can overcome this problem to a certain extent by learning more generalized feature representations to make more accurate predictions for objects that do not appear in the training set.

[0122] Second, it improves the performance of the model across different application scenarios (such as from conference rooms to security environments), reducing the need to retrain the entire model for each new scenario. This is because the system can better understand and adapt to changes in various background information, thereby improving cross-scenario performance.

[0123] Third, when processing surveillance video streams, a challenge often encountered is how to effectively identify smaller target objects in the picture. The present application has made improvements in this regard, so that small objects can be more accurately located and classified even in complex backgrounds.

[0124] Fourth, by utilizing pre-training on large-scale datasets and subsequent data fine-tuning strategies for specific tasks, it not only maintains a high detection accuracy but also exhibits good generalization characteristics - this means that it can not only complete predefined tasks well, but also has the ability to handle zero-sample problems to a certain extent, that is, it can be directly applied to new types of objects that have never been seen before.

[0125] A possible implementation of an object detection device in a monitoring scenario based on an open vocabulary is given below, which is used to execute the various execution steps and corresponding technical effects of the object detection method shown in the above embodiment and possible implementation. The device includes:

[0126] The data acquisition module is used to use the camera to extract the collected surveillance scene video data according to the frame rate to obtain image data, annotate the objects to be detected in the image data to generate annotated json files, and save the image data and json files;

[0127] A data processing module is used to process the image data and the json file to obtain processed data;

[0128] A data integration module is used to integrate the processed data according to the data paradigm to generate standardized text data and standardized image data;

[0129] A text image encoding module, used to convert text information of standardized text data using a text encoder to obtain a text embedding representation, and to convert image information of standardized image data using an image encoder to obtain an image embedding representation;

[0130] The text-image fusion alignment module is used to calculate the similarity between the text embedding representation and the image embedding representation, and obtain the classification alignment score based on the similarity;

[0131] The model training optimization module is used to calculate the loss function through the classification alignment score, and use the loss function to train or fine-tune the object detection model;

[0132] The model evaluation module is used to use the object detection model to evaluate the real labeled data and object detection data of the received test data to obtain corresponding evaluation indicators.

[0133] In a possible implementation, the text-image fusion alignment module is further used for:

[0134] Calculate the similarity between the text embedding representation and the image embedding representation, and filter out the image text embedding by similarity;

[0135] The image text embedding is processed using a transformer-based decoder to obtain the decoder output result;

[0136] The decoder output is input into the classification layer to predict the target box category and regress the bounding box to obtain the classification alignment score.

[0137] In a possible implementation, the model training optimization module is further used to:

[0138] The classification loss is calculated using the classification alignment score and the true classification value, and the regression loss is calculated using the regression bounding box and the bounding box truth value;

[0139] Use classification loss and regression loss to form a loss function;

[0140] The object detection model is obtained by training or fine-tuning the loss function. The training hyperparameters include batch size, training iters, optimizerAdamW, weight decay, learning rate of image-text encoder, and hidden layer dimension.

[0141] In a possible implementation manner, the data processing module is further used to:

[0142] Performing data screening on the image data and the json file respectively to obtain screened data, wherein the data screening includes fuzzy data and data with incorrect annotations;

[0143] The filtered data is cleaned and enhanced to obtain the processed data.

[0144] In one possible implementation, the text encoder includes a traditional encoder, an embedding encoder, and a context-aware encoder, and the image encoder includes a CNN-based image encoder, a Transformer-based image encoder, a generative adversarial network-based image encoder, and an autoencoder.

[0145] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for detecting objects in a monitoring scenario based on an open vocabulary, characterized in that: The method comprises: Use the camera to extract the collected surveillance scene video data according to the frame rate to obtain image data, mark the objects to be detected in the image data to generate a marked json file, and save the image data and json file; Perform data processing on the image data and the json file respectively to obtain the processed data; Perform data integration on the processed data according to the data paradigm to generate standardized text data and standardized image data; Using a text encoder to convert text information of standardized text data to obtain text embedding representation, and using an image encoder to convert image information of standardized image data to obtain image embedding representation; Calculate the similarity between the text embedding representation and the image embedding representation, and obtain the classification alignment score based on the similarity; The loss function is calculated by the classification alignment score, and the object detection model is obtained by training or fine-tuning using the loss function; The object detection model is used to evaluate the real labeled data and object detection data of the received test data to obtain corresponding evaluation indicators.

2. The object detection method according to claim 1, characterized in that: The steps of calculating the similarity between the text embedding representation and the image embedding representation and obtaining the classification alignment score based on the similarity include: Calculate the similarity between the text embedding representation and the image embedding representation, and filter out the image text embedding by similarity; The image text embedding is processed using a transformer-based decoder to obtain the decoder output result; The decoder output is input into the classification layer to predict the target box category and regress the bounding box to obtain the classification alignment score.

3. The object detection method according to claim 2, characterized in that: The steps of calculating the loss function through the classification alignment score and using the loss function to train or fine-tune the object detection model include: The classification loss is calculated using the classification alignment score and the true classification value, and the regression loss is calculated using the regression bounding box and the bounding box truth value; Use classification loss and regression loss to form a loss function; The object detection model is obtained by training or fine-tuning using the loss function. The training hyperparameters include batch size, training iters, optimizerAdamW, weight decay, learning rate of the image-text encoder, and hidden layer dimension.

4. The object detection method according to claim 1, characterized in that: The steps of processing the image data and the json file to obtain the processed data include: Performing data screening on the image data and the json file respectively to obtain screened data, wherein the data screening includes fuzzy data and data with incorrect annotations; The filtered data is cleaned and enhanced to obtain the processed data.

5. The object detection method according to claim 1, characterized in that: Text encoders include traditional encoders, embedded encoders, and context-aware encoders. Image encoders include CNN-based image encoders, Transformer-based image encoders, generative adversarial network-based image encoders, and autoencoders.

6. An object detection device in a monitoring scenario based on an open vocabulary, characterized in that: The device comprises: The data acquisition module is used to use the camera to extract the collected surveillance scene video data according to the frame rate to obtain image data, annotate the objects to be detected in the image data to generate annotated json files, and save the image data and json files; A data processing module is used to process the image data and the json file to obtain processed data; A data integration module is used to integrate the processed data according to the data paradigm to generate standardized text data and standardized image data; A text image encoding module, used to convert text information of standardized text data using a text encoder to obtain a text embedding representation, and to convert image information of standardized image data using an image encoder to obtain an image embedding representation; The text-image fusion alignment module is used to calculate the similarity between the text embedding representation and the image embedding representation, and obtain the classification alignment score based on the similarity; The model training optimization module is used to calculate the loss function through the classification alignment score, and use the loss function to train or fine-tune the object detection model; The model evaluation module is used to use the object detection model to evaluate the real labeled data and object detection data of the received test data to obtain corresponding evaluation indicators.

7. The object detection device according to claim 6, characterized in that: The text-image fusion alignment module is also used for: Calculate the similarity between the text embedding representation and the image embedding representation, and filter out the image text embedding by similarity; The image text embedding is processed using a transformer-based decoder to obtain the decoder output result; The decoder output is input into the classification layer to predict the target box category and regress the bounding box to obtain the classification alignment score.

8. The object detection device according to claim 7, characterized in that: The model training optimization module is also used for: The classification loss is calculated using the classification alignment score and the true classification value, and the regression loss is calculated using the regression bounding box and the bounding box truth value; Use classification loss and regression loss to form a loss function; The object detection model is obtained by training or fine-tuning the loss function. The training hyperparameters include batch size, training iters, optimizerAdamW, weight decay, learning rate of image-text encoder, and hidden layer dimension.

9. The object detection device according to claim 6, characterized in that: The data processing module is also used to: Performing data screening on the image data and the json file respectively to obtain screened data, wherein the data screening includes fuzzy data and data with incorrect annotations; The filtered data is cleaned and enhanced to obtain the processed data.

10. The object detection device according to claim 6, characterized in that: Text encoders include traditional encoders, embedded encoders, and context-aware encoders. Image encoders include CNN-based image encoders, Transformer-based image encoders, generative adversarial network-based image encoders, and autoencoders.

Images