Luggage open set detection model training method, detection method and equipment

Abstract

The invention discloses a luggage open set detection model training method, a detection method and equipment, and belongs to the field of artificial intelligence. The detection method comprises the following steps: inputting a to-be-detected luggage image into a trained open set detection model to obtain a luggage bounding box and a category label; the detection model comprises a candidate region generation module used for sequentially performing zero sample instance segmentation and filtering operation on an original luggage image to obtain candidate regions; and the positioning and classification identification network module is used for performing feature extraction by utilizing the candidate region and a text with known category names, performing bidirectional cross-modal multi-head attention fusion on extracted visual features and text features, optimizing bounding box coordinates of the candidate region through a positioning head based on the fusion features, and outputting the bounding box coordinates. And similarity between the fusion features and the text features is obtained through a classification head, and a category label is output based on a comparison result of the similarity and a classification threshold value. And luggage in a conveyor belt or a field can be quickly and accurately positioned.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, and specifically relates to a training method, detection method and equipment for baggage open set detection model. Background Technology

[0002] In automated scenarios such as airport logistics sorting, automated baggage transfer, and unmanned warehouse management, achieving rapid and accurate positioning of baggage on conveyor belts or in designated areas is a crucial foundation for improving overall operational efficiency and automation levels. However, current technological solutions face numerous significant challenges in practical applications.

[0003] First, the concept of luggage is broad, with diverse shapes, types, and colors, making it difficult for traditional detection models to obtain unified and robust feature representations. Second, the highly diverse appearances and materials of luggage, as well as their dense stacking and occlusion, all interfere with stable and accurate feature extraction. Finally, existing object detection methods heavily rely on large amounts of precisely labeled bounding box data, while providing pixel-level or box-level annotations for luggage with diverse shapes requires enormous manpower and time costs, severely restricting the widespread application and performance generalization of models in diverse scenarios. Summary of the Invention

[0004] To address the above problems, this invention provides a method for training a baggage open set detection model, a detection method, and an apparatus.

[0005] This invention provides a method for training a baggage open set detection model, comprising: Construct an initial baggage open set detection model and training dataset, and design the overall loss function; The initial baggage open set detection model is trained using the training dataset. During the training process, the loss value calculated by the total loss function is used as the optimization objective. The model parameters are iteratively updated through backpropagation until the model converges, resulting in a well-trained baggage open set detection model. The initial baggage open set detection model includes a candidate region generation module, which sequentially performs zero-shot instance segmentation and filtering based on geometric rules and prior knowledge on the original baggage image to obtain candidate regions. The feature extraction module is used to extract features from candidate regions and known category name text to obtain visual and text features. The localization prediction module is used to perform localization prediction using visual features, and obtain the predicted bounding box and localization confidence. The adaptive localization optimization module is used to update the localization threshold of the current training round using the localization reliability, and to select the region of interest from the candidate regions based on the localization reliability and the updated localization threshold; The multimodal fusion module is used to find the visual features of the region of interest from the visual features. It performs bidirectional cross-modal multi-head attention fusion based on text features and the visual features of the region of interest to obtain fused features. The similarity calculation module is used to calculate the similarity between the fused features and the text features.

[0006] Furthermore, the adaptive localization optimization module is specifically used to update the localization threshold of the current training round using the localization confidence method through the exponential moving average method, and to select candidate regions whose localization confidence is greater than the updated localization threshold as regions of interest.

[0007] Furthermore, the multimodal fusion module is specifically used to fuse text features and visual features of regions of interest using a bidirectional attention mechanism and a multi-head attention machine to obtain fused features; wherein, the bidirectional attention mechanism is a visual-to-text attention mechanism and a text-to-visual attention mechanism, and the multi-head attention includes visual-to-text multi-head attention and text-to-visual multi-head attention. The similarity calculation module is used to calculate the cosine similarity between the fused features and the text feature vector.

[0008] Furthermore, the total loss function is constructed based on the localization loss function, fairness loss function, alignment loss function, and triplet loss function. The localization loss function is used to optimize the model's accuracy in locating the bounding box of luggage objects. The fairness loss function is used to alleviate class imbalance and ensure the fairness of the model's recognition results. The alignment loss function is used to compare similarity with the true matching relationship. The triplet loss function is used to bring similar samples closer together and push dissimilar samples further apart in the feature space.

[0009] Furthermore, the localization loss function is:

[0010] in, The number of images in each training batch. The weighted localization loss function for a single image is given by the following formula:

[0011] Among them, subscript Representing the The first image One candidate region; The location reliability of the candidate region; For the first The first image The weights of each candidate region; For the smoothed L1 loss function, the input is the predicted bounding box coordinates. and the actual bounding box coordinates The specific formula is as follows:

[0012] in, .

[0013] This invention also provides a large model-driven method for high-precision open set detection of luggage, comprising: Input the baggage image to be detected into the trained open set detection model; The trained baggage open set detection model is used to detect baggage and obtain baggage bounding boxes and category labels; The trained open set detection models include: The candidate region generation module is used to perform zero-shot instance segmentation and filtering based on geometric rules and prior knowledge on the original luggage image to obtain candidate regions. The feature extraction module is used to extract features from candidate regions and known category name text to obtain visual and text features. The localization prediction module is used to perform localization prediction using visual features and output the predicted bounding box. The multimodal fusion module is used to perform bidirectional cross-modal multi-head attention fusion based on text features and visual features, and the fused features are sent to the classification decision module. The classification decision module calculates the similarity to known category text features based on fused features and utilizes a classification threshold. Make decisions and output category labels based on the decision results.

[0014] Furthermore, the classification threshold is calculated during the training phase based on the similarity distribution of known categories on the validation set.

[0015] The present invention also provides an electronic device, comprising: Memory, which stores execution instructions; and The processor executes the instructions stored in memory, causing the processor to perform the methods described above.

[0016] The present invention also provides a readable storage medium storing executable instructions, which are executed by a processor to implement the above-described method.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0018] Compared with the prior art, the present invention has the following advantages: 1) Based on the fine initial contour values ​​provided by the zero-shot instance segmentation model and the adaptive optimization strategy for localization in this invention, the localization accuracy is improved because the strategy can filter high-quality localization samples through dynamic thresholds and give higher learning weights to irregular objects. The localization of bounding boxes for various types of luggage objects is more accurate, especially when objects are stuck together. Through appearance-driven classification and fairness learning, the robustness of recognition is significantly enhanced. The model can stably recognize luggage objects of different materials, shapes and colors, and is friendly to sparse categories. The model performs stably in real industrial environments. The final deployed model is lightweight and has high computational efficiency, which can meet the needs of real-time processing in pipelines.

[0019] 2) Based on the cross-modal fusion classification and adaptive threshold decision-making mechanism of this invention, the model can not only accurately identify baggage of known categories, but also robustly detect "unknown" categories not seen during training (such as special equipment, new types of baggage, etc.), realizing true open set detection and significantly improving the practicality and robustness of the system in dynamic airport environments.

[0020] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This diagram illustrates the structure and data processing flow of the baggage open set detection model during the training phase according to an embodiment of the present invention. Figure 2 A set of candidate regions according to an embodiment of the present invention is shown; Figure 3 This diagram illustrates the structure and data processing flow of an open set baggage detection model during the inference phase according to an embodiment of the present invention. Figure 4 A diagram showing the baggage detection and identification results according to an embodiment of the present invention is provided. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] like Figure 1 As shown, this is a schematic diagram of the baggage open set detection model and data processing flow during the training phase of an embodiment of the present invention. The baggage open set detection model includes: a candidate region generation module, a feature extraction and localization prediction module, an adaptive localization optimization module, and a multimodal fusion module. Figure 1 The multimodal fusion module includes similarity calculation functionality, or a separate similarity calculation module can be defined; an initial baggage open set detection model and training dataset are constructed, and the corresponding total loss function is designed; The candidate region generation module is used to sequentially perform zero-shot instance segmentation on the original luggage image using a zero-shot instance segmentation model and filtering based on geometric rules and prior knowledge to obtain candidate regions. The feature extraction module is used to extract features from candidate regions and known category name text to obtain visual and text features. The localization prediction module is used to perform localization prediction using visual features, and obtain the predicted bounding box and localization confidence. The adaptive localization optimization module is used to update the localization threshold of the current training round using the localization reliability, and to select the region of interest from the candidate regions based on the localization reliability and the updated localization threshold; The multimodal fusion module is used to find the visual features of the region of interest from the visual features. It performs bidirectional cross-modal multi-head attention fusion based on text features and the visual features of the region of interest to obtain fused features. The similarity calculation module is used to calculate the similarity between the fused features and the text features.

[0025] Furthermore, the adaptive localization optimization module is specifically used to update the localization threshold of the current training round using the localization confidence method through the exponential moving average method, and to select candidate regions whose localization confidence is greater than the updated localization threshold as regions of interest.

[0026] Furthermore, the multimodal fusion module is specifically used to fuse text features and visual features of regions of interest using a bidirectional attention mechanism and a multi-head attention machine to obtain fused features; wherein, the bidirectional attention mechanism is a visual-to-text attention mechanism and a text-to-visual attention mechanism, and the multi-head attention includes visual-to-text multi-head attention and text-to-visual multi-head attention. The similarity calculation module is used to calculate the cosine similarity between the fused features and the text feature vector.

[0027] The training method for the above baggage open set detection model is as follows: The initial baggage open set detection model is trained using the training dataset. During training, the loss value calculated using the total loss function is used as the optimization objective. The model parameters are iteratively updated through backpropagation until the model converges, resulting in a trained baggage open set detection model. The following is a comparison... Figure 1 The specific training method of this invention will be described in detail below: Step S1: Acquire surveillance video frames to obtain raw luggage images. Manually annotate the raw luggage images to obtain ground truth bounding box coordinates and category labels, and organize them to obtain an annotated training image dataset. Use a zero-shot instance segmentation model, such as the Segment Anything Model (SAM), to perform zero-shot instance segmentation on the obtained raw luggage images, generate initial candidate bounding boxes, and perform geometric filtering: remove bounding boxes that match the annotated ground truth bounding boxes. >For regions with a set value (e.g., 0.5), non-standard shapes with excessively large (or small) aspect ratios are excluded. An area threshold is set based on prior knowledge, ultimately outputting a high-quality set of candidate regions that conform to the physical characteristics of luggage. Figure 2 The green box indicates the selected area.

[0028] Candidate region set The formula is as follows:

[0029] in, This represents the initial set of candidate bounding boxes generated by SAM, i.e. , of which each It is the segmentation mask generated by SAM. Convert it to bounding box format. This indicates the total number of initial candidate bounding boxes output; Candidate boxes With the actual annotation box The intersection-union ratio (CUNR) of the Ground Truth is used to measure the degree of overlap between two things; This is the intersection-union (IUU) threshold, used to exclude regions that overlap with known labels; The aspect ratio of the candidate box is represented by w, where w represents the width and h represents the length. It is used to evaluate whether the shape of the bounding box is reasonable. , The aspect ratio range; , The area threshold is used to adapt to the actual size distribution of luggage items.

[0030] Step S2: Combine the generated high-quality candidate regions to extract and predict visual and textual features.

[0031] Specifically, utilizing dynamics The encoder extracts from the candidate region set Visual features are extracted using the following formula:

[0032] The BERT encoder is used to extract text features from a set number (e.g., 3) of known category names. The extraction formula is as follows: =BERT ("Luggage, Parcel, Foam Box") Simultaneously, the candidate region set is utilized based on the following method. Location prediction obtained: For each candidate region, its visual features are... The input is fed into a lightweight localization subnetwork (LocHead), which uses visual features of candidate regions. The input consists of a fully connected layer and a regression head, which output the predicted bounding box coordinates, respectively. Location reliability The process can be formally represented as:

[0033] in, Represents the location subnetwork. Indicates the predicted bounding box coordinates. Represented as the first The location confidence of the nth candidate region in the image. Among them, text features... At this stage, it does not directly participate in coordinate regression, but instead provides semantic priors for subsequent cross-modal classification tasks.

[0034] Step S3: Construct an adaptive localization optimization strategy.

[0035] S301: A dynamic positioning threshold mechanism is used to adjust the selection criteria for positioning samples, and the positioning threshold is dynamically updated using the exponential moving average (EMA) method. The purpose is to assess the model's location reliability during the current training cycle. From adaptive adjustment of positioning threshold The specific formula is:

[0036] in, In training rounds Dynamic positioning threshold at time; This is the initial threshold; This is a momentum parameter, and its value can be determined based on experience. This is the batch size, which is the number of samples in each training batch; For the first Locational confidence of the nth candidate region in an image.

[0037] S302. Filtering regions of interest based on dynamic positioning thresholds: The location confidence of each candidate region obtained in step S2 Dynamic localization threshold in the current training round By comparing and selecting candidate regions with high location confidence as regions of interest (RoIs), their corresponding predicted bounding boxes are recorded. Formal representation:

[0038] in, These are the visual features corresponding to the candidate regions. Step S302 ensures that subsequent classification and fusion modules only process regions with high localization quality, improving training efficiency and model accuracy.

[0039] S303. Simultaneously, a weighted localization loss function for the entire batch is introduced, as shown in the following formula:

[0040] in, For batch size, The weighted localization loss function for a single image is given by the following formula:

[0041] in, For indicator functions, when Returns 1 if the condition is met; otherwise, returns 0. For the first Zhang Image ( The number of candidate regions in ) For the first The first image The localization confidence of each candidate region is used to measure the model's confidence in the localization quality of that candidate region. The shape weight is the nth candidate region in the kth image, which can be set according to the shape of the object: irregular objects (such as soft bags, foam boxes). Regular objects (such as standard suitcases) To improve the positioning accuracy of objects with complex shapes; To smooth the L1 loss function, input the bounding box coordinates predicted by the model. and the actual bounding box coordinates The specific formula is as follows:

[0042] in .

[0043] Step S4: Perform multimodal fusion of visual features and text features of the region of interest.

[0044] S401: Identify the visual features of the region of interest from the visual features.

[0045] After selecting high-quality regions of interest (RoIs) in step S302, step S401 selects the visual features of the corresponding regions directly from the visual feature set extracted and stored in step S2, based on the indices of these regions. These features represent the visual information of the selected image regions with high localization quality and will be directly used as input to the subsequent cross-modal fusion module.

[0046] S402: Perform cross-modal fusion of visual features and text features of the region of interest and calculate similarity to obtain fused features and similarity.

[0047] By leveraging a bidirectional cross-modal multi-head attention mechanism to achieve deep interaction between visual and textual features, and fully utilizing the complementary information of visual and textual data, it is easier to achieve high-precision baggage detection and classification in open-world scenarios.

[0048] Establish a bidirectional attention mechanism: attention from region of interest to text. Using visual features as queries and textual features as keys and values, the region of interest (ROI) features actively retrieve relevant information from the text semantics. The formula is as follows:

[0049] in, , is the visual query matrix, composed of the visual feature matrix Visual query projection matrix Obtained through conversion; , is the text key matrix, which is composed of the text feature matrix. Text key projection matrix Obtained through conversion; , is the text value matrix, which is composed of the text feature matrix. and text value projection matrix Obtained through conversion; In attention mechanisms, this represents the dimension of the key vector.

[0050] Text to Visual Attention Using text features as queries and region of interest features as keys and values, this approach allows textual concepts to actively seek supporting evidence from the region of interest. The formula is as follows:

[0051] in, , is the text query matrix, composed of the text feature matrix. and text query projection matrix Obtained through conversion; , is the visual key matrix, which is composed of the visual feature matrix. and visual key projection matrix Obtained through conversion; , is the visual value matrix, derived from the visual feature matrix. and visual value projection matrix Obtained through conversion; In attention mechanisms, this represents the dimension of the key vector.

[0052] Finally, a multi-head attention mechanism is designed, and its formula is as follows: Multi-head attention from vision to text:

[0053] Multi-head attention from text to vision:

[0054] in, It's about the number of heads; It is the first Visual-to-text attention parameters for size; It is the first Text size to visual attention parameters; It is the output projection matrix.

[0055] Multi-head attention from vision to text allows each visual feature to focus on the relevant parts of the text features, thereby enhancing the semantic representation of the visual features; multi-head attention from text to vision allows each text feature to focus on the relevant parts of the visual features, thereby enhancing the visual grounding of the text features.

[0056] Ultimately, the outputs of these two multi-head attention can be further fused to form cross-modal fusion features.

[0057] Next, we will utilize cross-modal fusion features to perform the first... The fusion feature vector of the candidate region and the first Text feature vector similarity for each category The calculation is performed using the following formula:

[0058] in, This is a global index for the candidate regions in the batch. Indexed by category; For the first The fused feature vector of each candidate region; For the first Text feature vectors for each category; , These are the moduli of the fused feature vector and the text feature vector, respectively; The range of values ​​is ,near This indicates a high degree of match between visual features and textual concepts.

[0059] S5. Calculate the classification loss.

[0060] The classification loss consists of four parts: weighted positioning loss, alignment loss, triplet loss, and fairness loss.

[0061] By using cross-entropy loss to encourage the alignment of the predicted distribution with the normalized distribution, the model identification bias caused by the imbalance in the number of luggage samples of different categories in the scenario is addressed. (Fairness loss function) The formula is as follows:

[0062] in, Indicates the first In each training batch, the model's predicted probabilities for all classes are calculated using an exponential moving average (EMA). Indicates the first The exponential moving average (EMA) of the class distribution histogram of pseudo-labels in each training batch. The EMA represents the probability distribution of model predictions up to the previous batch in the historical global timeframe. The EMA represents the histogram of pseudo-label category distribution over the historical period (i.e., up to the previous batch); This indicates that the input vector is summed and normalized, transforming it into a probability distribution; This represents the cross-entropy loss function, used to measure the difference between two probability distributions.

[0063] After each training batch, the above statistics are updated using the exponential moving average method, as shown in the following formula:

[0064]

[0065] in, The momentum coefficient controls the degree to which historical information is retained. This represents the number of samples in the current batch. For the first The predicted probability vector of each candidate region; For the first False labels for candidate regions; This is a function used to calculate the category histogram of the pseudo-labels in the current batch and normalize it to a probability distribution.

[0066] Alignment loss calculation function By comparing the similarity between visual and textual features with the true matching relationship, the open-world classification problem is transformed into a feature matching optimization task. First, an alignment matrix is ​​generated based on the labeled training image dataset. Then, the alignment loss is calculated using the alignment matrix. The alignment loss calculation function is... The calculation formula is as follows:

[0067] in, No. The fusion feature vector of the candidate region and the first Text feature vector similarity for each category , express Candidate regions and The true alignment relationship of each category (1 indicates a match, 0 indicates no match). This represents the total number of candidate regions in this batch. This represents the total number of categories in this batch.

[0068] Triple loss function By bringing similar samples closer together and distancing dissimilar samples further apart, the model's ability to distinguish between different categories of objects in the feature space is enhanced. The formula is as follows:

[0069] in, This represents the feature extraction function, which takes the input sample (anchor sample) as input. Positive samples negative samples The corresponding image region is mapped to a vector representation in the feature space; Represents the distance function; These are boundary parameters.

[0070] Step S6: Calculate the total loss of the multi-objective optimization loss function and update the model parameters through backpropagation.

[0071] Total loss The calculation formula is as follows:

[0072] in, The weighted localization loss function introduced in step S2; The alignment loss calculation function is as described above; The fairness loss function introduced for step S4; The loss function for the triplet is as described above; Weighting for fairness loss; The weights for the triplet loss.

[0073] Step S7: Model Convergence and Saving After one round of training, the updated model parameters (including all trainable parameters of the DyHead encoder, BERT encoder, localization subnetwork and cross-modal attention module) are used to evaluate the performance on the validation set, and this process is repeated iteratively from the S2 stage.

[0074] When the overall performance metric on the validation set stabilizes and no longer significantly improves, the model is considered converged, and the training process is terminated. To achieve open set recognition during the inference phase, a classification decision threshold is calculated during training based on the similarity distribution of known categories on the validation set. .

[0075] After the model converges, based on all known classes on the validation set (denoted as the set) The classification threshold for samples is... The calibration process is as follows: For each known category in the validation set Calculate the mean maximum similarity between all positive samples of this class and the corresponding text features of this class, denoted as . Then calculate the mean of this statistic for all known categories. with standard deviation :

[0076]

[0077] Then calculate the classification threshold. The model is calibrated during the training phase based on the similarity distribution of known categories on the validation set. The calculation formula is as follows:

[0078] in, Represents all known classes on the validation set. The maximum mean similarity; This represents the standard deviation of the corresponding similarity distribution; For balance parameters; This is a dynamically adjusted factor.

[0079] The final converged model weights are then compared with the calculated classification threshold. These are stored together to form a deployable baggage open set detection model. In subsequent inference, for the first... The fusion features of each candidate region are used to calculate its maximum similarity with all known categories. By maximum similarity With classification threshold Making decisions:

[0080] in, Indicates the first Masking of candidate regions For indicator functions, when If the condition is met, return 1, classifying it as "unknown" (other); otherwise, return 0, classifying it as the known category with the highest similarity.

[0081] Here is the classification threshold. It is "adaptive," mainly in two aspects: First, it is calculated based on the actual performance (similarity distribution) of the currently trained model on the validation set, rather than a preset fixed value; second, its calculation formula incorporates the overall trend of the distribution (mean). ) and dispersion (standard deviation) ), and through adjustable parameters and It adapts to the needs of different scenarios. This allows the threshold to automatically adapt to the capabilities of specific models and the characteristics of the data.

[0082] Classification threshold It is not dynamically updated in each training round. Instead, it is calculated and fixed once as a post-processing step after the model has fully converged. Therefore, this value does not change during training. This design ensures that the threshold can be calibrated based on stable model performance, avoiding the impact of early training fluctuations. The final converged model weights are then compared with the calculated classification threshold. These are stored together to form a deployable baggage open set detection model.

[0083] The trained lightweight detection model is deployed in a real-world scenario for detection.

[0084] The method for detecting baggage using a trained open set detection model is as follows: Input the baggage image to be detected into the trained open set detection model; The trained baggage open set detection model is used for baggage detection to obtain baggage bounding boxes and category labels; the trained baggage open set detection model is as follows: Figure 3 As shown, it contains two modules: Candidate region generation module: With the same structure as the training phase, it is used to perform zero-shot segmentation using the SAM model and to perform filtering operations based on geometric rules and prior knowledge to obtain high-quality candidate regions.

[0085] The localization and classification recognition network module is the core recognition unit of the system. Specifically, it includes: The feature extraction module includes the DyHead encoder, the BERT encoder, and a bidirectional cross-modal multi-head attention mechanism. Its structure is consistent with that of the training phase. It is used to extract features from candidate regions and known category name text to obtain visual and text features. The localization prediction module (LocHead) is... Figure 3 The localization subnetwork in this example has the same structure as the localization subnetwork used in the training phase. It is used to perform localization prediction using visual features and output the predicted bounding box. This localization subnetwork is optimized during the training phase and used as a fixed parameter during inference.

[0086] Multimodal fusion module, i.e. Figure 3 The bidirectional cross-modal multi-head attention mechanism is used to perform bidirectional cross-modal multi-head attention fusion based on text features and visual features, and the fused features are sent to the classification decision module. Classification decision module, i.e. Figure 3 The classification decision part calculates the similarity between the fused features and the text features of known categories, and uses the classification threshold calculated during the training phase. Make a decision and output a category label.

[0087] The aforementioned localization and classification network module is the core recognition unit. It performs refined processing on each candidate region, extracting features from visual appearance and category semantics in parallel, and using a cross-modal attention mechanism for deep fusion. Based on this, a localization head refines / optimizes the bounding box coordinates of the candidate regions, and a classification head matches the fused features with the semantic features of all known categories to obtain the maximum similarity. An adaptive threshold mechanism is employed to automatically classify objects with low matching scores as "unknown categories," thereby achieving accurate classification of known luggage in an open world and robust recognition of novel luggage. Finally, it synchronously outputs the precise location and category label of each target.

[0088] In actual deployment, the system receives real-time video streams, performs rapid inference on the input image frames, calculates the similarity between each detection region and known categories, and dynamically determines the category of unknown objects based on adaptive classification thresholds. Finally, it outputs the localization bounding box and recognition category result of the luggage object in real time. The luggage detection and recognition results are as follows: Figure 4 As shown, the red box represents a suitcase, the yellow box represents a parcel, and the purple box represents other items.

[0089] This achieves high-precision open-set baggage detection driven by a large model. Zero-shot candidate region generation is achieved by introducing SAM (System-Agent Learning), adaptive localization optimization strategies improve detection accuracy for irregular objects, and a bidirectional cross-modal multi-head attention mechanism combines visual and textual features. A fairness learning mechanism is integrated to address class imbalance, and a multi-objective loss function is used to collaboratively optimize localization, classification, and feature discrimination capabilities. Finally, a lightweight detection model is deployed to output baggage object localization and recognition results, comprehensively improving the robustness and accuracy of detecting known and unknown baggage in open-world environments.

[0090] In summary, this invention can improve the robustness of baggage detection in open-world environments through large model-guided candidate region generation, adaptive localization optimization, multimodal fusion classification, fairness learning, and multi-objective collaborative optimization. This strategy is suitable for airport baggage sorting scenarios.

[0091] The present invention also provides an electronic device, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform any of the methods described above.

[0092] The hardware architecture of electronic devices / devices can be implemented using a bus architecture. A bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. A bus connects various circuits, including one or more processors, memories, and / or hardware modules. A bus can also connect various other circuits such as peripherals, voltage regulators, power management circuits, external antennas, etc. Buses can be Industry Standard Architecture (ISA) buses, Peripheral Component Interconnect (PCI) buses, or Extended Industry Standard Component (EISA) buses, etc. Buses can be categorized as address buses, data buses, control buses, etc.

[0093] For ease of explanation, certain steps of the above method are described in relation to modules. It should be understood that the corresponding module performing one or more steps of the above method may be one or more hardware modules specifically configured to perform the corresponding step, or implemented by a processor configured to perform the corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination thereof.

[0094] The specific implementation of each module in the above-mentioned device can be referred to the implementation process of the corresponding steps in the above-mentioned method embodiments of the present invention, and will not be repeated here.

[0095] The present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection (electronic device) having one or more wires, a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0096] This invention also provides a computer program product. The method of this invention can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, the processes or functions of this invention are performed wholly or partially.

[0097] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a baggage open collection detection model, characterized in that, The application relates to a luggage open set detection method and device. An initial luggage open set detection model and a training data set are constructed, and a total loss function is designed; The initial luggage open set detection model is trained by using the training data set, and in the training process, a loss value calculated by the total loss function is used as an optimization target, model parameters are iteratively updated through back propagation until the model converges, and a trained luggage open set detection model is obtained. The initial luggage open set detection model comprises a candidate region generation module, a feature extraction module, a positioning prediction module, an adaptive positioning optimization module and a multi-modal fusion module. The candidate region generation module is used for performing zero-shot instance segmentation on an original luggage image by using a zero-shot instance segmentation model and performing filtering operation based on geometric rules and priori knowledge in sequence, and a candidate region is obtained. The feature extraction module is used for performing feature extraction on the candidate region and a known category name text, and visual features and text features are obtained. The positioning prediction module is used for performing positioning prediction on the visual features, and a predicted bounding box and a positioning confidence are obtained. The adaptive positioning optimization module is used for updating a positioning threshold of a current training round by using the positioning confidence, and a region of interest is screened from the candidate region based on the positioning confidence and the updated positioning threshold. The multi-modal fusion module is used for finding visual features of the region of interest from the visual features, and performing bidirectional cross-modal multi-head attention fusion based on the text features and the visual features of the region of interest, and obtaining fusion features.

2. The method of claim 1, wherein, The similarity calculation module is used for calculating a similarity between the fusion features and the text features.

3. The method of claim 1, wherein, The adaptive positioning optimization module is specifically used for updating the positioning threshold of the current training round by using the positioning confidence through an exponential moving average method, and screening the candidate region corresponding to the positioning confidence greater than the updated positioning threshold as the region of interest. The multi-modal fusion module is specifically used for fusing the text features and the visual features of the region of interest by using a bidirectional attention mechanism and a multi-head attention mechanism, and obtaining the fusion features.

4. The method of claim 1, wherein, The similarity calculation module is used for calculating a cosine similarity between the fusion features and a text feature vector.

5. The method of claim 4, wherein, The total loss function is constructed based on a positioning loss function, a fairness loss function, an alignment loss function and a triplet loss function. wherein, is the number of images in each training batch, is the weighted localization loss function for a single image, which is formulated as follows: where subscript represents the th image th candidate region in the th image; is the position confidence of the candidate region; is the weight of the th candidate region in the th image, and the specific formula is as follows: wherein .

6. A large model driven high-precision open set detection method for luggage, characterized in that, The positioning loss function is used for optimizing the positioning accuracy of a bounding box of a luggage object by the model. The fairness loss function is used for relieving class imbalance and ensuring fairness of a recognition result of the model. The alignment loss is used for comparing the similarity and a real matching relationship. The triplet loss function is used for narrowing down same-class samples and pushing away different-class samples in a feature space. The positioning loss function is as follows: The application further relates to a luggage detection method and device. The luggage image to be detected is input into the trained open set detection model. The trained open set detection model is used for luggage detection, and a luggage bounding box and a category label are obtained. The trained open set detection model comprises a candidate region generation module. The candidate region generation module is used for performing zero-shot instance segmentation on an original luggage image by using a zero-shot instance segmentation model and performing filtering operation based on geometric rules and priori knowledge in sequence, and a candidate region is obtained. The feature extraction module is configured to perform feature extraction on the candidate region and the known category name text to obtain visual features and text features. The localization prediction module is configured to perform localization prediction on the visual features to output a predicted bounding box. The multi-modal fusion module is configured to perform bidirectional cross-modal multi-head attention fusion based on the text features and the visual features to obtain fusion features, which are sent to the classification decision module. a classification decision module, configured to calculate similarity between the fused feature and a known class text feature, and utilize a classification threshold make a decision, and output a class label based on a result of the decision.

7. The method of claim 6, wherein, The classification threshold is calculated based on a similarity distribution of known categories on a validation set in a training phase.

8. An electronic device, comprising: comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the method in any one of claims 1 to 7.

9. A readable storage medium, characterized by, The readable storage medium stores execution instructions, which are executed by the processor to implement the method in any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method in any one of claims 1 to 7.

Images