Open world object detection method and device based on causal rectification learning

Abstract

The application discloses an open world target detection method and device based on causal rectification learning. The method comprises the following steps: according to the semantics of each known class, the prediction result of unknown class features is adjusted by a backdoor to obtain decontaminated unknown class features, and a semantic causal intervention loss function is constructed according to the unknown class features and the decontaminated unknown class features; an unknown causal enhancement loss function is established based on similarity; the overall training function obtained by the semantic causal intervention loss function and the unknown causal enhancement loss function is used to train the detection device, and open world target detection is performed through the trained detection device. The application can improve the classification accuracy of the open world target detection model facing unknown classes.

Description

Technical Field

[0001] This invention relates to an object detection method, and more particularly to an open-world object detection method based on causal correction learning, as well as a corresponding open-world object detection device, belonging to the field of computer vision technology. Background Technology

[0002] Object detection, or object detection, is a core problem in computer vision. It involves identifying all objects of interest in an image and determining their category and location. Currently, most mainstream object detection methods are based on the assumption of a closed set, meaning all categories to be detected are available during training. Under this static, closed-set premise, the object detection model only needs to detect known (labeled during training) categories. This model typically ignores instances of unknown (not labeled in the training set) categories as background or incorrectly classifies them as known categories, focusing only on detection results on known categories and evaluating performance accordingly. However, this premise does not apply in the real world. The real world is open, not closed. There are an infinite number of categories in the real world, far exceeding the number of categories in existing object detection datasets.

[0003] To address the aforementioned issues, object detection models encounter two challenging problems in practical applications: 1. Test images may contain objects from unknown categories, which should be classified as unknown by the object detection model; 2. When annotation information for these already identified unknown objects becomes available, the object detection model should be able to incrementally learn the categories of these newly labeled objects. Therefore, the technical approach of Open World Object Detection (OWOD) has been proposed.

[0004] Compared to closed-world object detection, open-world object detection aims not only to make accurate predictions within fully labeled known classes, but also to identify unknown instances as "unknown" during the testing phase and learn unknown targets as labels are incrementally added. Specifically, open-world object detection utilizes an unknown perception region generation network with an automatic labeling strategy, coupled with an energy-based binary classifier to distinguish between unknown and known classes. However, despite the performance improvements in open-world object detection, existing methods still heavily rely on information from known classes to extract unknown features, leading to inaccurate feature extraction bias towards known features. This biased feature extraction further affects the accuracy of known and unknown classes; that is, unknown classes are often misclassified as known classes or even suppressed, resulting in reduced accuracy for known classes. Summary of the Invention

[0005] The primary technical problem to be solved by this invention is to provide an open-world object detection method based on causal correction learning.

[0006] Another technical problem to be solved by the present invention is to provide an open-world object detection device based on causal correction learning.

[0007] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0008] According to a first aspect of the present invention, an open-world object detection method based on causal correction learning is provided, comprising the following steps:

[0009] Based on the semantics of each known class, the prediction results of the unknown class features are adjusted by a backdoor to obtain the deconfounded unknown class features. A semantic causal intervention loss function is constructed based on the unknown class features and the deconfounded unknown class features.

[0010] Establish an unknown causal enhancement loss function based on similarity;

[0011] An open-world object detection model is trained using an overall training function obtained from a semantic causal intervention loss function and an unknown causal enhancement loss function. Object detection is then performed using the trained open-world object detection model.

[0012] Specifically, the step of adjusting the prediction results of unknown class features based on the high-level semantics of each known class via a backdoor includes:

[0013] The semantics of each known class are hierarchically divided into semantic features of each known class;

[0014] The backdoor adjustment enables the interaction between the unknown class features and the semantic features of each known class, and calculates the probability between the unknown class features and the semantic features of each known class.

[0015] The feature expression for removing the unknown class of confounding is as follows:

[0016]

[0017] In formula (1), x u Representing features of unknown class, Indicates the predicted category, s k Representing known semantic features, P(s) represents the score of the k-th class obtained by the unknown class feature on the known class classifier. k ) represents the probability distribution of known semantic features.

[0018] The step of constructing a semantic causal intervention loss function based on unknown class features and deconfounded unknown class features specifically includes the following sub-steps:

[0019] The visual similarity matrix of the unknown class features and the semantic similarity matrix of the unconfounded unknown class features are calculated separately to construct the semantic causal intervention loss function.

[0020] The semantic similarity matrix is ​​as follows:

[0021]

[0022] In formula (2), To remove features from the unknown class;

[0023] The visual similarity matrix is ​​as follows:

[0024]

[0025] In formula (3), x u Indicates features of unknown class;

[0026] The semantic causal intervention loss function is:

[0027]

[0028] In formula (4), A s A v Let D represent the semantic similarity matrix and visual feature similarity matrix between the same batch of training data, respectively. σ is the softmax function, and D... KL This is the relative entropy.

[0029] The step of establishing an unknown causal enhancement loss function based on similarity specifically includes the following sub-steps:

[0030] An unknown class feature memory is established based on similarity, and a clustering loss function is constructed based on the unknown class features and the unknown class feature memory.

[0031] A loss function is constructed by calculating the pairwise similarity between features of the unknown class.

[0032] Based on the similarity of unknown class features in the same batch in the feature space and probability space, construct a dot product loss function;

[0033] The unknown causal enhancement loss function is composed of the clustering loss function, the pair loss function, and the dot product loss function.

[0034] The step of establishing an unknown class feature memory based on similarity and constructing a clustering loss function based on the unknown class features and the unknown class feature memory specifically includes the following sub-steps:

[0035] Unknown features in the preset batch are retained without creating an unknown feature memory.

[0036] Calculate the similarity between each vector in the memory and the currently unknown class feature;

[0037] Vectors with a similarity greater than a preset threshold are considered positive samples, and vectors with a similarity less than or equal to the preset threshold are considered negative samples.

[0038] Construct a clustering loss function based on the current unknown class features, positive samples, and negative samples.

[0039] The step of constructing a loss function by calculating the pairwise similarity between unknown class features specifically includes the following sub-steps:

[0040] Calculate the similarity between pairs of features of unknown class. Pairs with similarity higher than a predetermined threshold are positive sample pairs, and the rest are negative sample pairs.

[0041] Establish a loss function based on positive and negative sample pairs.

[0042] Wherein, the clustering loss function for:

[0043]

[0044] In formula (5), B is the unknown candidate box in the current batch, x i The unknown class feature represents the unknown class candidate box, m j For memory bank with x i Features whose similarity exceeds a threshold Let M be the set of positive samples in the memory bank, and M′ be the set of negative samples in the memory bank. t represents the features of other samples in the current batch of training data, and t is a parameter that controls the distribution size.

[0045] Wherein, the loss function for:

[0046]

[0047] In formula (6), Describes the set of positive sample pairs. Let represent the set of negative sample pairs, max(·) represents the maximum value, ⊙ represents the dot product operation, and BCE(·) represents the binary cross-entropy loss.

[0048] The dot product loss function for:

[0049]

[0050] In formula (7), F i,j P represents the dot product similarity of unknown class features in the same batch in the feature space. i,j It represents the dot product similarity of unknown class features in the same batch in the probability space.

[0051] According to a second aspect of the present invention, an open-world object detection device based on causal correction learning is provided, comprising a processor and a memory; the memory is used to store a computer program, which, when executed by the processor, implements the above-described open-world object detection method based on causal correction learning.

[0052] Compared with existing technologies, the open-world object detection method and apparatus provided by this invention, on the one hand, can eliminate the dependence of unknown class features on known classes by adjusting the constructed semantic causal intervention loss function through a backdoor, thus severing the link between unknown class features and known class features. On the other hand, the unknown causal enhancement loss function based on similarity can strengthen the causal relationship between unknown class features and input images, significantly improving the classification accuracy when facing unknown classes. Attached Figure Description

[0053] Figure 1 A flowchart of an open-world target detection method provided in an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of the open-world target detection method provided in an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of an open-world target detection device provided in an embodiment of the present invention. Detailed Implementation

[0056] The technical content of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0057] In open-world object detection, each stage has a corresponding set of known classes. Where N + The set represents positive integers. To realistically simulate the real world, all unknown classes encountered during the reasoning phase are categorized as U = {C+1, ...}. All known object categories are labeled as D in the dataset. t ={X t ,Y t}, where X represents all input images and Y represents the image labels.

[0058] In one embodiment of the present invention, an input image set contains m training images, i.e., X t ={I1,…,I m Correspondingly, each image is associated with its corresponding label, thus generating a label set Y. t ={Y1,…,Y M Each image contains P object instances, represented as Y. i ={y1,y2,…,yP Each instance's label contains category and location information, represented as y. k =[l k ,x k ,y k ,w k ,h k ], where l k ∈K t And x k ,y k ,w k ,h k These represent the center position and the length and width of the rectangle, respectively.

[0059] Suppose that open-world object detection has a set of incremental tasks T = {1, 2, ..., t, ...}. In task t, the dataset D... t ={I t ,Y t} Contains input image set I t and object-level tag set Y t ={Y1,…,Y M}, where Y i Includes class tag L i and the true bounding box b i Specifically, in the training set, only known objects are assigned labels, i.e., L. i ∈K t K t Let {1, 2, ..., C} be the set of known classes, and C be the number of known classes. However, during the testing phase, the set of unknown classes U... t Unknown instances of {C+1, C+2, ...} should also be identified as "unknown". Furthermore, in task t+1, incremental annotations are applied to the unknown classes of interest, thus updating the known class set to K. (t+1) =K t ∪{C+1, ..., C+U}, where U is the total number of new annotation classes. This process continues until... Therefore, open-world object detection models need to progressively identify previously known classes and currently known classes, while identifying the remaining classes as "unknown" classes.

[0060] like Figure 1 As shown in the figure, an open-world object detection method based on causal correction learning provided by an embodiment of the present invention includes the following steps:

[0061] 101. Based on the semantics of each known class, the prediction results of the unknown class features are adjusted by a backdoor to obtain the deconfounded unknown class features. Based on the unknown class features and the deconfounded unknown class features, a semantic causal intervention loss function is constructed.

[0062] 102. Establish an unknown causal enhancement loss function based on similarity;

[0063] 103. Train an open-world object detection model using the overall training function obtained by semantic causal intervention loss function and unknown causal enhancement loss function, and then perform object detection using the trained open-world object detection model.

[0064] like Figure 2 As shown, based on the high-level semantics of each known class, the prediction results of the unknown class features are adjusted using a backdoor, specifically including the following sub-steps:

[0065] 1011. Divide the semantics of each known class into semantic features of each known class;

[0066] 1012. Based on backdoor adjustment, the unknown class features interact with the semantic features of each known class, and the probability between the unknown class features and the semantic features of each known class is calculated.

[0067] In one embodiment of the present invention, a structural causal model (SCM) is proposed to represent causal relationships in open-world object detection tasks and identify factors leading to bias problems. This SCM is used to perform backdoor adjustment on the prediction results of unknown class features based on the semantics of each known class, obtaining deconfounded unknown class features; constructing a semantic causal intervention loss function based on the unknown class features and the deconfounded unknown class features; constructing an unknown causal enhancement loss function based on similarity; and training an open-world object detection model using the overall training function obtained from the semantic causal intervention loss function and the unknown causal enhancement loss function, and performing object detection using the trained open-world object detection model.

[0068] In one embodiment of the present invention, the structural causal model includes a semantic causal intervention module (SCI) and an unknown causal enhancement module (UCE). The semantic causal intervention module constructs a semantic causal intervention loss function, which partitions known class features at the class level to eliminate the influence of confounding factors, and uses the semantic embeddings of known classes as stable representations for each class through semantic alignment with visual features. The unknown causal enhancement module constructs an unknown causal enhancement loss function, which enhances the causal relationship between images and unknown features through contrastive learning across different batches, explicitly learning to separate different categories among unlabeled data. This unknown causal enhancement module also uses pairwise similarity and self-supervised alignment of unknown instances in the feature space to learn discriminative features, thereby strengthening the causal relationship between unknown class features and predictions.

[0069] In one embodiment of the invention, the semantic causal intervention module hierarchically assigns features of the known class to the confounding factor with the aid of visual and semantic alignment, thereby forcing the unknown class features to be treated fairly with all possible cases of the confounding factor. This approach eliminates spurious correlations, thereby improving the performance of both the known and unknown classes. The unknown causal enhancement module strengthens the causal relationship between images and unknown class features by imposing constraints on the unknown feature space through a memory bank. By imposing constraints on pairwise similarities between unknown instances, the prediction space is aligned with the feature space.

[0070] like Figure 2 As shown, this structural causal model is a directed acyclic graph, where directed links represent causal relationships between two nodes. Formally, X k and X u X represents the known and unknown class features extracted from image I, while Y is their multi-class prediction vector. Where X... u ←X k →Y is a backdoor path, where confounding factor X k Simultaneously affects X u The label Y leads to a false correlation between known and unknown information, causing unknown class features to point to known class features. Since open-world object detection uses known supervision to generate pseudo-unknown labels, it inevitably relies on known unknown features. And X... k →Y represents the obvious causal relationship between known features and their predictions. →X u →Y. Extract unknown features X from image I. u This information is then used to obtain predictions. However, relying solely on uncertain pseudo-unknown labels provides too weak a causal relationship. Therefore, to obtain reasonable unknown features, it is necessary to cut off X. k →X u and strengthen I→X u →Y link.

[0071] Due to the learned prediction P(Y|X) u I) by X k Confusion, embodiments of the present invention interrupt X through causal intervention k →X u Remove confounding factor X k This encourages enhanced causal detection of unknowns. Since physical intervention to collect objects in any environment is impossible, embodiments of this invention apply backdoor adjustments to achieve P(Y|do(X)). u The methods employed include: 1. Cutting off link X k →X u 2. X k Layered into blocks X k ={x k}, so as to force the input to interact fairly with all possible scenarios of confounding factors. In order to introduce all possible x k , will x k It is divided at the class level. High-level semantic word embeddings are used for each known class. k ∈S to achieve backdoor adjustment:

[0072]

[0073] In formula (8), I represents the image, X k and X u This represents the known and unknown class features extracted from image I. Y is their multi-class prediction vector, P(Y|do(X)). u ,I)) indicates that the do operation (causal intervention) is used on X k The prediction results for the unknown class after intervention. k ∈S represents high-level semantic word embeddings, which here use the embedding vectors of the category names of known classes.

[0074] Using a backdoor to adjust the do operation on the unknown class features means that the features of the unknown class interact with the features of every possible known class. After using semantic embedding to replace the known classes, the formula is expressed as the probability calculation between the features of the unknown class and the semantics of all known classes.

[0075]

[0076] In formula (9), NWGM represents the normalized weighted geometric mean. This represents the expectation in terms of the known class semantics. Equation (9) incorporates external variables and condition variables, reducing the expensive cost of computing each instance individually. For an unknown class feature, it is first input into a classifier of the known class, producing a softmax output. Corresponding to class k, it is possible to force the input to interact with confounding factors in all allowed cases:

[0077]

[0078] In formula (10), P(s) k Let be a uniform 1 / C, where C is the number of known classes. W is the parameter of the classifier.

[0079] In one embodiment of the present invention, the unknown class is decontaminated. The characteristic expression is:

[0080]

[0081] In the above formula, x u Representing features of unknown class, Indicates the predicted category, s k Representing known semantic features, P(s) represents the score of the k-th class obtained by the unknown class feature on the known class classifier. k ) represents the probability distribution of known semantic features.

[0082] The semantic causal intervention loss function is constructed based on the unknown class features and the deconfounded unknown class features, specifically including the following sub-steps:

[0083] 1013. Calculate the visual similarity matrix of unknown class features and the semantic similarity matrix of unconfounded unknown class features respectively, and then construct the semantic causal intervention loss function.

[0084] The semantic similarity matrix is ​​as follows:

[0085]

[0086] In the above formula, To remove features from the unknown class;

[0087] The visual similarity matrix is ​​as follows:

[0088]

[0089] In the above formula, x u Indicates features of unknown class;

[0090] The semantic causal intervention loss function is:

[0091]

[0092] In the above formula, A s A v Let D represent the semantic similarity matrix and visual feature similarity matrix between the same batch of training data, respectively. σ is the softmax function, and D... KL This is the relative entropy.

[0093] In one embodiment of the invention, through a hierarchical confounding factor X k By aligning visual and semantic features, to force X u Interacting with all possible confounding factors, using the semantic embedding of known classes as a stable representation of each class can eliminate spurious correlations and improve performance for both known and unknown classes.

[0094] like Figure 2 As shown, step 102: Establishing an unknown causal enhancement loss function based on similarity, specifically including the following sub-steps:

[0095] 1021. Establish a feature memory for unknown classes based on similarity, and construct a clustering loss function based on the features of unknown classes and the feature memory for unknown classes;

[0096] 1022. Construct a loss function by calculating the pairwise similarity between features of the unknown class;

[0097] 1023. Construct a dot product loss function based on the dot product similarity of unknown class features in the same batch in the feature space and probability space;

[0098] 1024. The unknown causal enhancement loss function is composed of the clustering loss function, the pair loss function, and the dot product loss function.

[0099] Sub-step 1021 specifically includes the following sub-steps:

[0100] 10211. Unknown features in the preset batch are retained, but no unknown feature memory is created;

[0101] 10212. Calculate the similarity between each vector in the memory and the current unknown class feature;

[0102] 10213. Vectors with a similarity greater than a preset threshold are considered positive samples, and vectors with a similarity less than or equal to the preset threshold are considered negative samples;

[0103] 10214. Construct a clustering loss function based on the current unknown class features, positive samples, and negative samples.

[0104] In one embodiment of the present invention, in order to obtain unknown class features, it is necessary to enhance I→X. u → The causal relationship of the Y link P(Y|X) u I) and P(X) u |I), and strengthen the causal relationship of I→Xu.

[0105] P(X u The causal relationship of |I) can be strengthened by constraining the feature space of the unknown class. Since there is no supervision for the unknown class, this embodiment of the invention employs memory-based contrastive learning to explicitly learn to separate different classes among unlabeled data, which relies on finding positive samples in different batches corresponding to the desired class. Specifically, the unknown features of the previous batch (preset batch) are first retained as a regional memory M, and each unknown class feature x... i Measure the similarity between the vector and all memory vectors. Choose the similarity ω of the memory vectors ij Samples greater than a preset threshold θ are considered positive samples. It is a similarity function. The clustering loss function... for:

[0106]

[0107] Where B is the unknown candidate box in the current batch, x i The unknown class feature represents the unknown class candidate box, m j For memory bank with x i Features whose similarity exceeds a threshold Let M be the set of positive samples in the memory bank, and M′ be the set of negative samples in the memory bank. t represents the features of other samples in the current batch of training data, and t is a parameter that controls the distribution size.

[0108] In one embodiment of the invention, to align the prediction space with the feature space, pairwise similarity is first applied between known instances. Specifically, given a pair of selected unknown proposals (r... i r j ), extract features (x) i x j And calculate their similarity as pseudo-labels y. i,j =I[φ(x i x j )>=λ], where I[φ(x i x j )>=λ] is the Kronecker delta function, when φ(x) i x j When λ >= λ, it equals 1.

[0109] Therefore, sub-step 1022 specifically includes the following sub-steps:

[0110] 10221. Calculate the similarity between pairs of features of unknown class. Pairs with similarity higher than a predetermined threshold are positive sample pairs, and the rest are negative sample pairs.

[0111] 10222. Establish a pair loss function based on positive and negative sample pairs.

[0112] The loss function for:

[0113]

[0114] in, Describes the set of positive sample pairs. Let represent the set of negative sample pairs, max(·) represents the maximum value, ⊙ represents the dot product operation, and BCE(·) represents the binary cross-entropy loss.

[0115] In one embodiment of the invention, a soft consistency constraint is added, that is, aligning the distributions between the feature space and the probability space, thereby reducing the causal effects of the same class. Thus, the dot product loss function... for:

[0116]

[0117] Among them, F i,j P represents the dot product similarity of unknown class features in the same batch in the feature space. i,j It represents the dot product similarity of unknown class features in the same batch in the probability space.

[0118] In summary, the unknown causality enhancement loss function for:

[0119]

[0120] In formula (11), β1 and β2 are hyperparameters for adjusting the weights.

[0121] like Figure 2 As shown, the backbone network extracts feature representation f from the input image, and the region generation network takes f as input to generate region candidate boxes P = R(f; θ), where each candidate box is represented by the positional offset of a related predefined anchor point. These candidate boxes are passed to the region of interest module in the open-world object detection model to obtain the features of positive and negative candidate boxes, denoted as P. (b)+ and P (b)- .

[0122] In one embodiment of the invention, in each training task t, the open-world object detection model is first trained with a new class. The extracted features X... k and X u The obfuscation is removed, and then the data is passed to an open-world object detection model to obtain the final prediction. Furthermore, features of unknown classes are constrained by an unknown causal enhancement loss function. Therefore, the overall training function in step 103 is:

[0123]

[0124] In formula (12), For the Region Generation Network (RPN) loss, For cross-entropy loss, For the smoothed L1 loss of the regression, α1 and α2 represent the weights, P (b)+ For positive candidate boxes, P (b)+ These are negative sample candidate boxes.

[0125] Based on the aforementioned open-world object detection method based on causal correction learning, this invention also provides an open-world object detection device based on causal correction learning. For example... Figure 3As shown, this open-world target detection device includes a processor and a memory, and may further include communication components, sensor components, power supply components, multimedia components, and input / output interfaces as needed. The memory, communication components, sensor components, power supply components, multimedia components, and input / output interfaces are all connected to the processor.

[0126] In different embodiments of the present invention, the memory may be static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, etc. The processor may be a central processing unit (CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), digital signal processing (DSP) chip, etc. Other communication components, sensor components, power supply components, multimedia components, etc., can all be implemented using common components found in existing smartphones, and will not be specifically described here.

[0127] exist Figure 3 In the open-world target detection device shown, the processor reads a computer program from memory to perform the following operations:

[0128] Based on the semantics of each known class, the prediction results of the unknown class features are adjusted by a backdoor to obtain the deconfounded unknown class features. A semantic causal intervention loss function is constructed based on the unknown class features and the deconfounded unknown class features.

[0129] Establish an unknown causal enhancement loss function based on similarity;

[0130] An open-world object detection model is trained using an overall training function obtained from a semantic causal intervention loss function and an unknown causal enhancement loss function. Object detection is then performed using the trained open-world object detection model.

[0131] The following section compares the technical performance of the open-world object detection method (denoted as Ours-RCNN / DETR) provided in this invention embodiment with existing technologies, including: 1. Faster-RCNN; 2. Faster-RCNN+ fine-tuning in each incremental task (which cannot identify unknown instances but can serve as a reference for known performance); 3. ORE model; 4. SA model. Furthermore, this invention embodiment also compares its technical performance with DETR-based object detection methods, including DETR and OW-DETR. Specific comparison results are shown in Tables 1 and 2:

[0132] Table 1

[0133]

[0134] Table 2

[0135]

[0136] In Tables 1 and 2, mAP represents the average precision; WI (Wilderness Impact) refers to the impact of unknown classes on the classification precision of known classes in an open environment setting; A-OSE (Absolute Open-Set Error) refers to the number of unknown classes that are misclassified as known classes. U-mAP and U-Rec represent the mAP and recall of the unknown class, respectively; UD-Rate represents the localization accuracy of the unknown class; and UD-Pre refers to the correct classification rate of the located unknown class.

[0137] The comparison results in Tables 1 and 2 show that the open-world object detection method and apparatus provided by this invention, on the one hand, can eliminate the dependence of unknown class features on known classes by adjusting the constructed semantic causal intervention loss function through a backdoor, thus severing the link between unknown class features and known class features. On the other hand, the unknown causal enhancement loss function based on similarity can strengthen the causal relationship between unknown class features and input images, significantly improving the classification accuracy of the open-world object detection model when facing unknown classes.

[0138] The above provides a detailed description of the open-world object detection method and apparatus based on causal bias correction learning provided by this invention. Any obvious modifications made by those skilled in the art without departing from the essential spirit of this invention will constitute an infringement of the patent rights of this invention and will incur corresponding legal liability.

Claims

1. An open-world object detection method based on causal rectified learning, characterized in that The method comprises the following steps: According to the semantic of each known class, the prediction result of the unknown class feature is adjusted by backdoor, and the decontaminated unknown class feature is obtained, and a semantic causal intervention loss function is constructed according to the unknown class feature and the decontaminated unknown class feature; An unknown class feature memory bank is established based on similarity, a clustering loss function is constructed according to the unknown class feature and the unknown class feature memory bank, a pair loss function is constructed by calculating the similarity between each pair of unknown class features, and a point multiplication loss function is constructed according to the point multiplication similarity of the unknown class features in the same batch in the feature space and the probability space; The unknown causal enhancement loss function is composed of the clustering loss function, the pair loss function and the point multiplication loss function; The open world target detection model is trained by using the overall training function obtained by the semantic causal intervention loss function and the unknown causal enhancement loss function, and target detection is performed by using the trained open world target detection model.

2. The open-world object detection method based on causal counterfactual learning of claim 1, wherein According to the high-level semantics of each known class, the prediction result of the unknown class feature is adjusted by backdoor, which comprises the following sub-steps: Each known class semantic is layered into known class semantic features; Based on the backdoor adjustment, the unknown class feature is interacted with each known class semantic feature, and the probability between the unknown class feature and each known class semantic is calculated.

3. The causal rectified learning based open-world object detection method of claim 1, wherein The deconfounded unknown class feature is represented by the following equation: wherein, represents an unknown class feature, represents a predicted class, represents a known semantic feature, represents a score of the unknown class feature on the known class classifier for the k th class, represents a probability distribution of the known semantic feature.

4. The causal rectified learning based open-world object detection method of claim 1, wherein The unknown class feature memory bank is established based on similarity, and the clustering loss function is constructed according to the unknown class feature and the unknown class feature memory bank, which comprises the following sub-steps: The unknown class features in the preset batch are not stored in the unknown class feature memory bank; The similarity between each vector in the memory bank and the current unknown class feature is calculated; Vectors with similarity greater than a preset threshold are used as positive samples, and vectors with similarity less than or equal to a preset threshold are used as negative samples; The clustering loss function is constructed according to the current unknown class feature, the positive samples and the negative samples.

5. The causal rectified learning based open-world object detection method of claim 1, wherein The pair loss function is constructed by calculating the similarity between each pair of unknown class features, which comprises the following sub-steps: The similarity between each pair of unknown class features is calculated, and the similarity higher than a predetermined threshold is a positive sample pair, and the rest is a negative sample pair; The pair loss function is established according to the positive sample pair and the negative sample pair.

6. An open-world object detection device based on causal correction learning, characterized in that... The method comprises a processor and a memory; The memory is used to store a computer program, when the computer program is executed by the processor, the method for open world target detection based on causal rectification learning in claim 1-5 is realized.

Images