Open vocabulary object detection method and system for power construction scene images

Abstract

The present invention relates to the field of target detection technology, solves the technical problem that traditional open target detection cannot be applied to electric power construction scene image detection, and particularly relates to an open vocabulary target detection method for electric power construction scene images, including pre-training of image-text embedding representation based on cross-modal contrastive learning, training of target detection framework based on known concept supervision, fine-tuning of weakly supervised detection model based on image-level multi-label data set, and forward reasoning to obtain detection results. The present invention can be oriented to the vertical field of electric power construction scene image management, support the input of any vocabulary in the electric power construction scene and output the target rectangular bounding box coordinates of the corresponding object, and at the same time, there is no need to accurately mark the target rectangular bounding box label information for all data, thereby achieving full interaction of multi-modal information and improving the semantic understanding and generalization ability of the open vocabulary target detection model.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to an open vocabulary target detection method for electric power construction scene pictures. Background Art

[0002] Open vocabulary object detection has become a new trend in modern object detection, aiming to detect objects beyond the scope of predefined categories. Early studies performed this by training detectors on basic categories and evaluating novel (unknown) categories. However, while this open vocabulary setting can evaluate the detector's ability to detect and recognize novel objects, it is still limited to open scenarios and lacks generalization to other domains due to training on limited datasets and vocabulary. Inspired by vision-language pre-training, recent studies have cast open vocabulary object detection as an image-text matching task and leveraged large-scale image-text data to expand the size of the training vocabulary.

[0003] Existing open object detection methods can achieve a certain degree of open vocabulary object detection, but there are still huge problems when directly used for power construction scene images. After a large number of experimental comparisons and theoretical analysis, it is known that the problems mainly exist in three aspects:

[0004] First, the data sets of all current public methods are big data collected and annotated on the Internet, which have domain differences and semantic gaps with power construction scenarios;

[0005] Second, all the data used in the existing training methods have manually annotated precise target rectangular bounding box label information, that is, each concept or phrase knows the coordinate value of the corresponding object, which is very time-consuming and labor-intensive.

[0006] Third, the existing open vocabulary object detection methods all adopt the design of object detectors in terms of framework, and inject language information into the final classification layer of the detection network, failing to achieve full interaction of multimodal information, which seriously affects the semantic understanding and generalization ability of the model. Summary of the invention

[0007] The technical problem to be solved by the present invention is that traditional open target detection cannot be applied to power construction scene image detection.

[0008] To solve the above technical problems, the present invention provides the following technical solutions: an open vocabulary target detection method for electric power construction scene pictures, the method comprising the following steps:

[0009] S1. Obtain a database composed of known data sets, including the "Wukong" data set, Object365 and Visual Genome data sets, GoldG+ data set, Open Images data set, and image-level multi-label data set of power construction scenes;

[0010] S2. Use the "Wukong" dataset to perform pre-training of image-text embedding representation based on cross-modal contrastive learning to obtain an image embedding representation function for extracting arbitrary image feature representation and text feature representation in the target detection network model. And the text embedding representation function ;

[0011] S3, based on Object365 and Visual Genome datasets, GoldG+ datasets, and image embedding representation functions And text embedding representation function Train and optimize the target detection network model to obtain a target detection network model supervised by known concepts;

[0012] S4, using the Open Images dataset and the image-level multi-label dataset of power construction scenes to generate pseudo labels for the coordinates of the rectangular boxes of weakly supervised open vocabulary objects based on the weakly supervised detection model;

[0013] S5. Using pseudo labels to train the target detection network model to obtain an open vocabulary target detection model for power construction scene images;

[0014] S6. Adjust the input concept to the open vocabulary that needs to be detected as the input of the open vocabulary target detection model, and perform score screening on the predicted target rectangular box to output the open vocabulary target detection result.

[0015] Furthermore, in step S2, the specific process includes the following steps:

[0016] S21. Image data in the "Wukong" dataset and text data Process the image data separately The feature representation And text data The feature representation ;

[0017] S22. Introducing temperature coefficient into Sigmoid contrast loss function and a learnable bias ;

[0018] S23, using Sigmoid contrast loss function to represent all features in a training batch and feature representation Conduct cross-modal contrastive learning;

[0019] S24. Obtain image data when the Sigmoid contrast loss function reaches the minimum value and text data Image embedding representation function And the text embedding representation function .

[0020] Furthermore, in step S21, the specific process includes the following steps:

[0021] S211, image data The images in are uniformly scaled to a resolution of 384×384 and evenly divided into 12×12 small grids;

[0022] S212, shielding at most 40% of the small grids by random masking, and arranging the remaining small grids that are not shielded in order into an image block sequence;

[0023] S213, input the image block sequence into the image encoder to obtain image data The feature representation ;

[0024] S214: Text data The text semantics in is rewritten and saved with a probability of not less than 50%;

[0025] S215: Input the rewritten text into a text encoder to obtain text data The feature representation .

[0026] Further, in step S23, specifically: the image text pair training data in a training batch is recorded as , the goal is to achieve the image text belonging to the same pair The distance between different pairs of images and texts is smaller than The distance between them, the Sigmoid contrast loss function is formulated as follows:

[0027]

[0028] in, Indicates the number of image-text pairs in a batch; Representing images and text The label of the image-text pair composed of , when the two represent the same semantics, the label value is 1, otherwise the label value is -1; is the temperature coefficient; is the bias term; For images The feature representation of For Text The feature representation of .

[0029] Furthermore, in step S3, the specific process includes the following steps:

[0030] S31. Merge the Object365, Visual Genome, and GoldG+ datasets into a training set, and each image in the training set contains one or more concepts. , each concept The coordinates of the corresponding target rectangular bounding box in the image are known;

[0031] S32. Each concept The vector representation after mapping by the text encoder BERT is recorded as ;

[0032] S33, the input image is mapped by the image encoder Swin-B to obtain a The three-dimensional feature map is Line The regional features corresponding to the columns are expressed as , and each position corresponds to The target query vector is denoted as ;

[0033] S34. According to vector representation and regional feature representation Guide the generation of the target query vector and The generated target query vectors are screened to obtain a target query vector combination;

[0034] S35, combining the target query vectors as the input of a decoder composed of 4 layers of Transformers, and outputting the predicted target rectangular bounding box coordinates corresponding to each target query vector to form a predicted target rectangular bounding box coordinate set;

[0035] S36, performing Hungarian matching on the predicted target rectangular bounding box coordinate set and the real target rectangular bounding box coordinate set to achieve one-to-one correspondence;

[0036] S37, according to the highest matching score The total intersection-over-union loss is calculated for each frame, and the gradient descent method is used to guide the parameter optimization of the target detection network model.

[0037] Further, in step S34, the specific process includes the following steps:

[0038] S341、Calculate different concepts The corresponding vector representation The regional feature representation of each position in the three-dimensional feature map Similarity ;

[0039] S342, select similarity The largest value The target query vector corresponding to each point is taken as the effective query vector;

[0040] S343. Screen all valid query vectors to obtain a target query vector combination as an input of the decoder.

[0041] Furthermore, in step S341, the similarity The calculation formula is:

[0042]

[0043] In the above formula, represents the inner product operation, Represents the magnitude of the eigenvector.

[0044] Further, in step S37, the specific process includes the following steps:

[0045] S371, set The predicted box and the true box are and , total crossover ratio Defined as:

[0046]

[0047] in, Represents the intersection area of ​​two bounding boxes; represents the area of ​​their union;

[0048] S372, according to the total intersection ratio Calculate the total intersection loss , total transaction loss The calculation formula is:

[0049]

[0050] S373, using the gradient descent algorithm to update the network parameters according to the gradient obtained by back propagation, the updated expression is:

[0051]

[0052] in, Represents the target detection network model parameters; is the learning rate; is the gradient of the intersection-over-union loss function with respect to the training parameters

[0053] Furthermore, in step S4, the specific process includes the following steps:

[0054] S41. Concept The text description is fixed as "an object", and after being mapped by the text encoder BERT, a fixed concept feature representation is obtained;

[0055] S42, the input image is mapped by the image encoder Swin-B to obtain the regional feature representation of the object, and each position corresponds to target query vector;

[0056] S43, according to the fixed concept characteristics to express The target query vectors are screened, and finally the decoder generates the predicted target rectangular frame coordinates;

[0057] S44, after inputting the image-level concept weak label into the text encoder BERT, a feature representation of each concept is obtained, and then an inner product operation is performed between the feature representation of each concept and the feature representation corresponding to the predicted target rectangular box coordinates to obtain the category information and confidence of each target rectangular box;

[0058] S45. Select a target rectangular box whose confidence is greater than a predetermined threshold as a pseudo label of the image-level concept.

[0059] The present invention also provides an open vocabulary target detection system for electric power construction scene images corresponding to any of the above-mentioned solutions and methods, comprising:

[0060] The database composition module is used to obtain the known dataset composition database, including the "Wukong" dataset, Object365 and Visual Genome datasets, GoldG+ dataset, Open Images dataset, and image-level multi-label dataset of power construction scenes;

[0061] Pre-training module, used to use the "Wukong" dataset to perform image-text embedding representation pre-training based on cross-modal contrastive learning, and obtain image embedding representation functions for extracting arbitrary image feature representations and text feature representations in the target detection network model And the text embedding representation function ;

[0062] Training module for Object365 and Visual Genome datasets, GoldG+ datasets, and image embedding representation functions And text embedding representation function Train and optimize the target detection network model to obtain a target detection network model supervised by known concepts;

[0063] A pseudo-label generation module is used to generate pseudo-labels for the coordinates of rectangular boxes of weakly supervised open vocabulary objects based on the weakly supervised detection model using the Open Images dataset and the image-level multi-label dataset of power construction scenes;

[0064] A detection model acquisition module is used to train the target detection network model using pseudo labels to obtain an open vocabulary target detection model for power construction scene images;

[0065] The detection result output module is used to adjust the input concept to the open vocabulary that needs to be detected as the input of the open vocabulary target detection model, and to score and screen the predicted target rectangular box to output the open vocabulary target detection result.

[0066] By means of the above technical solution, the present invention provides an open vocabulary target detection method for electric power construction scene pictures, which has at least the following beneficial effects:

[0067] The open vocabulary target detection method proposed in the present invention supports the input of any vocabulary in the electric power construction scene and outputs the target rectangular bounding box coordinates of the corresponding object. It not only avoids the domain differences and semantic gaps between the existing public method data sets and the electric power construction scene, but also does not need to accurately mark the target rectangular bounding box label information for all data. It can thus be oriented to the vertical field of electric power construction scene image management, thereby realizing full interaction of multimodal information and improving the semantic understanding and generalization capabilities of the open vocabulary target detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0069] Figure 1 A network framework diagram of a cross-modal contrastive learning model for image-text embedding representation of the present invention;

[0070] Figure 2 A network framework diagram of a target detection network model of the present invention;

[0071] Figure 3 It is a network framework diagram of the weak supervision detection model of the present invention;

[0072] Figure 4 It is a forward reasoning process diagram of the open vocabulary object detection model of the present invention;

[0073] Figure 5The picture to be detected in the electric power construction scene of the present invention;

[0074] Figure 6 The test result diagram of the input word "iron tower" of the present invention;

[0075] Figure 7 The detection result diagram of the present invention when the word "person in white shirt" is input. DETAILED DESCRIPTION

[0076] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods, so that the implementation process of how the present application uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.

[0077] The present invention designs an efficient open vocabulary object detection method for power construction scene images that combines known concept supervision and weak label training. Figure 1-Figure 7 In this embodiment, a known data set is used for pre-training of image-text embedding representation of cross-modal contrastive learning, and then a target detection framework is trained based on known concept supervision, and then an open vocabulary target detection model is obtained by fine-tuning a weakly supervised detection model based on an image-level multi-label data set, and then the input concept is adjusted to the open vocabulary to be detected as the input of the open vocabulary target detection model for forward reasoning to obtain the detection result, which can accurately identify the open vocabulary targets in the electric power construction scene pictures. The method includes the following steps:

[0078] S1. Obtain a known data set to form a database, which includes the "Wukong" data set, Object365 and Visual Genome data sets, GoldG+ data set, Open Images data set, and a multi-label data set of electric power construction scene images; in this embodiment, in order to improve the adaptability of the open vocabulary target detection model designed by the present invention to any input vocabulary, and to achieve better results in the vertical field of electric power construction scene images, this embodiment uses the following five types of data sets during model training, as follows:

[0079] 1. General domain image-level Chinese multimodal picture-text pair open source dataset "Wukong";

[0080] The "Wukong" dataset is a large-scale Chinese cross-modal dataset open sourced by Huawei's Noah's Ark Laboratory. It consists of more than 100 million image-text pairs. Each image-text pair instance contains an image and its corresponding Chinese text description sentence.

[0081] 2. General domain object detection datasets Object365 and Visual Genome;

[0082] Object365 is a large-scale, high-quality object detection dataset released by Beijing Zhiyuan, containing a total of 630,000 images, covering 365 categories, and up to 10 million object bounding box coordinates.

[0083] Visual Genome is a large-scale image semantic understanding dataset released by Fei-Fei Li's group at Stanford University in 2016. It contains about 100,000 images and a total of 76,340 object types. On average, each image has 16 objects, and the corresponding rectangular bounding box positions are marked.

[0084] 3. GoldG+, an open source dataset of regional-level graph-text pairs in general fields;

[0085] The GoldG+ dataset is a large-scale, general-domain region-level image-text pair open source dataset released by Microsoft. It contains approximately 1.3 million images, each of which is annotated with multiple object regions and their corresponding descriptive phrases instead of categories, such as "standing person".

[0086] 4. Open Images, a general-purpose image-level multi-label dataset;

[0087] Open Images is a large-scale image dataset released by Google, with a total of about 9 million images. Its labels are at the image level, that is, it is marked what kind of objects each image contains, but their locations are not marked. The total number of categories is about 6,000.

[0088] 5. Image-level multi-label dataset of power construction scenes;

[0089] In actual practice, 10,000 images of power construction scenes were collected. In order to reduce the annotation cost, only image-level annotation was performed on each image. The annotation content can be categories or descriptive phrases, such as "truck", "iron tower", "person in yellow vest", etc., as long as the content exists in the image.

[0090] Since the above labels 2, 3, and 4 are in English, this embodiment uses translation software to translate them into Chinese and use them as final training labels.

[0091] S2. Pre-training of image-text embedding representation based on cross-modal contrastive learning based on the "Wukong" dataset to obtain an image embedding representation function for extracting arbitrary image feature representation and text feature representation in the target detection network model. And the text embedding representation function In this embodiment, the image-text embedding representation pre-training is performed based on the general domain image-level Chinese multimodal image-text pair open source dataset "Wukong". The network framework of the image-text embedding representation cross-modal contrastive learning model designed by the present invention is as follows Figure 1As shown, the text encoder uses BERT, which is commonly used in the field, and the image encoder uses Swin-B, which is commonly used in the field. Represents the image-text pair data input to the model. The specific process includes the following steps:

[0092] S21. Image data in the "Wukong" dataset and text data Process the image data separately The feature representation And text data The feature representation ; In step S21, the specific process includes the following steps:

[0093] S211, image data The images in are uniformly scaled to a resolution of 384×384 and evenly divided into 12×12 small grids;

[0094] S212, shielding at most 40% of the small grids by random masking, and arranging the remaining small grids that are not shielded in order into an image block sequence;

[0095] S213, input the image block sequence into the image encoder to obtain image data The feature representation ;

[0096] S214: Text data The text semantics in is rewritten and saved with a probability of not less than 50%;

[0097] S215: Input the rewritten text into a text encoder to obtain text data The feature representation .

[0098] In this embodiment, for image data First, the image is uniformly scaled to a resolution of 384×384, and then the image is evenly divided into 12×12 small grids. Then, a random mask is used to block out up to 40% of the small grids and the remaining unblocked small grid image areas are arranged in sequence as an image block sequence and sent to the image encoder to obtain the image The feature representation .

[0099] For text data Although the Chinese multimodal image-text open source dataset "Wukong" contains more than 100 million image-text pairs, each image corresponds to only one sentence of text, and the diversity of the text is still insufficient. To solve this problem, this embodiment uses a 50% probability of calling the Chat GPT3.5 API to rewrite the text with the same semantics each time the original text is feature modeled. The prompt word is "Please rewrite the following sentence and keep the original semantics unchanged: ", and then use the text encoder BERT, which is commonly used in the field of text features, to extract the feature representation of text data .

[0100] S22. Introducing a learnable temperature coefficient in the Sigmoid contrast loss function and the bias term ; During the training initialization, serious data imbalance (the number of mismatched image-text pairs is much larger than the number of matched image-text pairs) will dominate the optimization process of the loss function. In order to alleviate this situation, the present invention introduces the temperature coefficient and a learnable bias A larger temperature coefficient is set during the initial training, so that no value in the probability distribution is close to zero, and the gradient is easier to propagate. As the training progresses, the value of the temperature coefficient t is gradually reduced, and the bias term b is set as a learnable parameter. The image-text embedding representation cross-modal contrastive learning model can automatically adjust its preference for different categories instead of relying on manual adjustment.

[0101] S23, using Sigmoid contrast loss function to represent all features in a training batch and feature representation Cross-modal contrastive learning is performed; the feature representation of all images and texts in a training batch is subjected to cross-modal contrastive learning using the Sigmoid contrast loss function. The image-text pair training data in a training batch is denoted as , the goal is to achieve the image text belonging to the same pair The distance between different pairs of images and texts is smaller than The Sigmoid contrast loss function designed by the present invention is formulated as follows:

[0102]

[0103] in, Indicates the number of image-text pairs in a batch; Representing images and text The label of the image-text pair composed of , when the two represent the same semantics, the label value is 1, otherwise the label value is -1; is the temperature coefficient; is the bias term; For images The feature representation of For Text It can be seen from the formula that the temperature coefficient affects the shape of the Sigmoid function. Specifically, the temperature coefficient t is used to adjust the scale of the entire loss function, that is, in the Sigmoid function, the input will be adjusted to When the temperature coefficient is large, the Sigmoid function becomes smoother, that is, the decision boundary is softer; on the contrary, when the temperature coefficient is small, the Sigmoid function becomes steeper, making the output closer to 0 or 1, and the decision is more "hard". The quantitative analysis results are shown in Table 1. Obviously, by adjusting the temperature coefficient t, the learning behavior of the model and the certainty of the output can be effectively controlled. The present invention uses a higher temperature coefficient t in the early stage of training to increase the exploratory nature of the model, and then gradually reduces it to make the model more deterministic. In the specific implementation, the temperature coefficient is first set to 1.0 for 30 rounds of training, then the temperature coefficient is set to 0.5 for 80 rounds of training, and finally the temperature coefficient is set to 0.05 for 100 rounds of training until convergence.

[0104] Table 1 Quantitative analysis results

[0105]

[0106] S24. Obtain image data when the Sigmoid contrast loss function reaches the minimum value and text data Image embedding representation function And the text embedding representation function When the training is completed, that is, when the Sigmoid contrast loss function reaches the minimum value, the image embedding representation function can be obtained. And text embedding representation function . and In the subsequent target detection network model, it is used to extract feature representations of any image and text.

[0107] S3, based on Object365 and Visual Genome datasets, GoldG+ datasets, and image embedding representation functions And text embedding representation function The target detection network model is trained and optimized to obtain a target detection network model supervised by known concepts. This embodiment trains a target detection network model supervised by known concepts based on the general domain target detection datasets Object365 and Visual Genome and the general domain regional level graph-text pair open source dataset GoldG+. The framework diagram of the target detection network model is shown in Figure 2 As shown. The above three data sets are first merged to form a complete training set. Each image in the training set contains one or more concepts (a concept can be a single noun or a descriptive phrase composed of words), and the coordinates of the target rectangular bounding box corresponding to each concept in the image are also known (it should be a set of coordinates of the real target rectangular bounding box). Based on these data, the training steps of the target detection network model based on known concept supervision designed in this embodiment are as follows:

[0108] S31. Merge the Object365, Visual Genome, and GoldG+ datasets into a training set, and each image in the training set contains one or more concepts. , each concept The coordinates of the corresponding target rectangular bounding box in the image are known;

[0109] S32. Each concept The vector representation after mapping by the text encoder BERT is recorded as ;

[0110] S33, the input image is mapped by the image encoder Swin-B to obtain a The three-dimensional feature map of Indicates the number of channels, and Represent the height and width of the three-dimensional feature map respectively, so the three-dimensional feature map Line The regional feature representation corresponding to the column is a dimension The vector of , and each position corresponds to The target query vector is a four-dimensional vector denoted by , respectively represent the center point coordinates and width and height of the target, and the coordinates of the corresponding object can be directly predicted by the decoder.

[0111] S34, according to the vector representation and regional feature representation Guide the generation of the target query vector and The generated target query vector is filtered to obtain the target query vector combination; since each position corresponds to The target query vector is related to the concept The total number of target query vectors corresponding to the entire three-dimensional feature map is too large. Failure to screen the generated target query vectors will result in too high redundancy, which will seriously affect the training speed. In view of this, the present invention proposes to use conceptual feature representation to guide the generation and implementation of target query vectors. The specific steps are as follows:

[0112] S341、Calculate different concepts The corresponding vector representation The regional feature representation of each position in the three-dimensional feature map Similarity , where the similarity The calculation formula is:

[0113]

[0114] In the above formula, represents the inner product operation, represents the magnitude of the eigenvector;

[0115] S342, select similarity The largest value The target query vector corresponding to each point is taken as the effective query vector;

[0116] S343, all valid query vectors are screened to obtain a target query vector combination as an input of the decoder, such as Figure 2 The target query vector combination consists of the concept 1 target query vector, the concept 2 target query vector and the concept 3 target query vector.

[0117] S35. The target query vector combination is used as the input of a decoder composed of 4 layers of Transformers, and the predicted target rectangular bounding box coordinates corresponding to each target query vector are output to form a predicted target rectangular bounding box coordinate set; in the present invention, a decoder composed of 4 layers of Transformers is designed, the input is the target query vector combination of the previous step, and the output is the predicted target rectangular bounding box coordinates and its score corresponding to each target query vector (the score value is limited to the range of [0,1]).

[0118] S36, perform Hungarian matching on the predicted target rectangular bounding box coordinate set and the real target rectangular bounding box coordinate set to achieve one-to-one correspondence; it should be noted that in step S31, each concept The corresponding target rectangular bounding box coordinates in the image are all known, so the real target rectangular bounding box coordinate set here is the several target rectangular bounding box coordinates known in step S31.

[0119] S37, according to the highest matching score The total intersection-over-union loss is calculated for each frame, and the gradient descent method is used to guide the parameter optimization of the target detection network model. The specific process is:

[0120] S371, set The predicted box and the true box are and , the total intersection-union ratio is defined as:

[0121]

[0122] in, Represents the intersection area of ​​two bounding boxes; represents the area of ​​their union.

[0123] S372. Calculate total cross-over ratio loss based on total cross-over ratio , total transaction loss The calculation formula is:

[0124]

[0125] According to the above formula, this means that when the overlap between the predicted box and the true box is higher, the total intersection-over-union value is larger and the total intersection-over-union loss is smaller.

[0126] S373, using the gradient descent algorithm to update the network parameters according to the gradient obtained by back propagation, the updated expression is:

[0127]

[0128] in represents the model parameters; is the learning rate; is the gradient of the intersection-over-union loss function with respect to the training parameters.

[0129] S4. Generate pseudo labels for weakly supervised open vocabulary object rectangular frame coordinates based on the weakly supervised detection model using the Open Images dataset and the electric power construction scene image-level multi-label dataset. This embodiment designs a weakly supervised detection model for weakly supervised open vocabulary object rectangular frame coordinate pseudo labels based on the general domain image-level multi-label dataset Open Images and the electric power construction scene image-level multi-label dataset. Figure 3 As shown in Figure 1, it mainly includes a text encoder, an image encoder, a concept semantics-guided target query vector generation and screening, a decoder, and a pseudo-label generation module. Figure 2 The structure is the same, and the training parameters are directly loaded with the parameters trained in S3 as the initial parameters. Only a text encoder is added to generate pseudo labels for each predicted box. In step S4, the specific process includes the following steps:

[0130] S41. Concept The text description is fixed as "an object", and after being mapped by the text encoder BERT, a fixed concept feature representation is obtained;

[0131] S42, the input image is mapped by the image encoder Swin-B to obtain the regional feature representation of the object, and each position corresponds to target query vector;

[0132] S43, according to the fixed concept characteristics to express The target query vectors are screened, and finally the decoder generates the predicted target rectangular box coordinates; it should be noted that, consistent with the training steps of the above-mentioned target detection network model, the target query vectors are screened according to the fixed concept feature representation in the previous step on the basis of extracting the regional feature representation, and finally the decoder generates the predicted target rectangular box coordinates.

[0133] S44. In the usage scenario of the present invention, the target rectangular frame coordinates obtained in the previous step have no real labels corresponding to them. To solve this problem, this embodiment obtains the feature representation of each concept by inputting the image-level concept weak labels into the text encoder BERT, and then performs an inner product operation on the feature representation of each concept and the feature representation corresponding to the predicted target rectangular frame coordinates to obtain the category information and confidence of each target rectangular frame. The specific method is as follows:

[0134] Remember the current image The concept text is , the output concept text feature representation of the text encoder BERT is recorded as , No. The coordinates of the target box are , Represent the center point coordinates and width and height of the object frame respectively, and then obtain the corresponding feature representation vector within the target frame range in the image feature representation layer according to the coordinates, and average these vectors to obtain the feature representation corresponding to the target rectangular frame , then The target box belongs to The confidence of the concept text category for:

[0135]

[0136] in Represents the inner product operation.

[0137] S45, select the target rectangular box with a confidence greater than a predetermined threshold as the pseudo label of the image-level concept. In this embodiment, the predetermined threshold determines which predicted bounding boxes are considered to be sufficiently credible as positive samples for training the target detection network model. Correctly setting this prediction threshold is an important factor in improving the performance of the target detection network model. In specific implementation, the present invention sets the predetermined threshold to 0.45.

[0138] S5. Using pseudo labels to train the target detection network model to obtain an open vocabulary target detection model for power construction scene images. Based on the pseudo labels obtained in the previous step, training is performed again according to the aforementioned training steps of the target detection network model, i.e., steps S31-S37, so as to obtain the final open vocabulary target detection model for power construction scene images.

[0139] S6. Adjust the input concept to the open vocabulary to be detected as the input of the open vocabulary target detection model, and score the predicted target rectangle to output the open vocabulary target detection result. After the training is completed according to the above steps, the forward reasoning process of the open vocabulary target detection model for power construction scene images is as follows: Figure 4 As shown in the figure, the parameters of all modules are obtained by training in the above steps and remain unchanged during the inference process. For example, the specific steps for detecting the target type of "people on the tower" in an image of a power construction scene are:

[0140] S61, input the text “the person on the tower” into the text encoder BERT to obtain the text feature representation g;

[0141] S62, input the image to be detected into the image encoder Swin-B to obtain the image feature representation f, each position of the feature representation corresponds to N query vectors, and the feature representation corresponding to each query vector ;

[0142] S63, the text feature representation g is combined with the feature representation corresponding to each query vector Perform inner product operation, and the query vectors corresponding to the first K largest values ​​are used as valid query vectors;

[0143] S64. The K valid query vectors are output by the decoder with the coordinates and scores of K target boxes, and the box with the largest score is selected as the target minimum rectangular bounding box corresponding to the text "Man on the Tower".

[0144] Specifically, Figure 5 The following is a picture to be tested. The test results of inputting the word "iron tower" into the open vocabulary object detection model are as follows: Figure 6 As shown in the figure, the detection result of the input word "person in white shirt" is as follows Figure 7 shown.

[0145] The open vocabulary target detection method proposed in the present invention is aimed at the vertical field of power construction scene image management. It is not limited to limited predefined categories and supports the input of arbitrary words (such as "truck", "worker wearing seat belt") and outputs the target rectangular bounding box coordinates of the corresponding object.

[0146] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, so the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0147] The above implementation methods have been described in detail. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An open vocabulary object detection method for power construction scene images, characterized in that: The method comprises the following steps: S1. Obtain a database composed of known data sets, including the "Wukong" data set, Object365 and VisualGenome data sets, GoldG+ data set, Open Images data set, and image-level multi-label data set of power construction scenes; S2. Use the "Wukong" dataset to perform pre-training of image-text embedding representation based on cross-modal contrastive learning to obtain an image embedding representation function for extracting arbitrary image feature representation and text feature representation in the target detection network model. And the text embedding representation function ; S3, based on Object365 and Visual Genome datasets, GoldG+ datasets, and image embedding representation functions And text embedding representation function Train and optimize the target detection network model to obtain a target detection network model supervised by known concepts; S4, using the Open Images dataset and the image-level multi-label dataset of power construction scenes to generate pseudo labels for the coordinates of the rectangular boxes of weakly supervised open vocabulary objects based on the weakly supervised detection model; S5. Using pseudo labels to train the target detection network model to obtain an open vocabulary target detection model for power construction scene images; S6, adjusting the input concept to the open vocabulary to be detected as the input of the open vocabulary target detection model, and performing score screening on the predicted target rectangular box to output the open vocabulary target detection result; In step S2, the specific process includes the following steps: S21. Image data in the "Wukong" dataset and text data Process the image data separately The feature representation And text data The feature representation ; S22. Introducing temperature coefficient into Sigmoid contrast loss function and a learnable bias ; S23, using Sigmoid contrast loss function to represent all features in a training batch and feature representation Conduct cross-modal contrastive learning; S24. Obtain image data when the Sigmoid contrast loss function reaches the minimum value and text data Image embedding representation function And the text embedding representation function ; In step S21, the specific process includes the following steps: S211, image data The images in are uniformly scaled to a resolution of 384×384 and evenly divided into 12×12 small grids; S212, shielding at most 40% of the small grids by random masking, and arranging the remaining small grids that are not shielded in order into an image block sequence; S213, input the image block sequence into the image encoder to obtain image data The feature representation ; S214: Text data The text semantics in is rewritten and saved with a probability of not less than 50%; S215: Input the rewritten text into a text encoder to obtain text data The feature representation ; In step S23, specifically: the image text pair training data in a training batch is recorded as , the goal is to achieve the image text belonging to the same pair The distance between different pairs of images and texts is smaller than The distance between them, the Sigmoid contrast loss function is formulated as follows: in, Indicates the number of image-text pairs in a batch; Representing images and text The label of the image-text pair composed of , when the two represent the same semantics, the label value is 1, otherwise the label value is -1; is the temperature coefficient; is the bias term; For images The feature representation of For Text The feature representation of In step S3, the specific process includes the following steps: S31. Merge the Object365, Visual Genome, and GoldG+ datasets into a training set, and each image in the training set contains one or more concepts. , each concept The coordinates of the corresponding target rectangular bounding box in the image are known; S32. Each concept The vector representation after mapping by the text encoder BERT is recorded as ; S33, the input image is mapped by the image encoder Swin-B to obtain a The three-dimensional feature map is Line The regional features corresponding to the columns are expressed as , and each position corresponds to The target query vector is denoted as ; S34. According to vector representation and regional feature representation Guide the generation of the target query vector and The generated target query vectors are screened to obtain a target query vector combination; S35, combining the target query vectors as the input of a decoder composed of 4 layers of Transformers, and outputting the predicted target rectangular bounding box coordinates corresponding to each target query vector to form a predicted target rectangular bounding box coordinate set; S36, performing Hungarian matching on the predicted target rectangular bounding box coordinate set and the real target rectangular bounding box coordinate set to achieve one-to-one correspondence; S37, according to the highest matching score The total intersection-over-union loss is calculated for each frame, and the gradient descent method is used to guide the parameter optimization of the target detection network model; In step S34, the specific process includes the following steps: S341、Calculate different concepts The corresponding vector representation The regional feature representation of each position in the three-dimensional feature map Similarity ; S342, select similarity The largest value The target query vector corresponding to each point is taken as the effective query vector; S343. Screen all valid query vectors to obtain a target query vector combination as an input of the decoder.

2. The open vocabulary target detection method for electric power construction scene pictures according to claim 1 is characterized in that: In step S341, the similarity The calculation formula is: In the above formula, represents the inner product operation, Represents the magnitude of the eigenvector.

3. The open vocabulary target detection method for electric power construction scene pictures according to claim 1 is characterized in that: In step S37, the specific process includes the following steps: S371, set The predicted box and the true box are and , total crossover ratio Defined as: in, Represents the intersection area of ​​two bounding boxes; represents the area of ​​their union; S372, according to the total intersection ratio Calculate the total intersection loss , total transaction loss The calculation formula is: S373, using the gradient descent algorithm to update the network parameters according to the gradient obtained by back propagation, the updated expression is: in, Represents the target detection network model parameters; is the learning rate; is the gradient of the intersection-over-union loss function with respect to the training parameters.

4. The open vocabulary target detection method for electric power construction scene pictures according to claim 1 is characterized in that: In step S4, the specific process includes the following steps: S41. Concept The text description is fixed as "an object", and after being mapped by the text encoder BERT, a fixed concept feature representation is obtained; S42, the input image is mapped by the image encoder Swin-B to obtain the regional feature representation of the object, and each position corresponds to target query vector; S43, according to the fixed concept characteristics to express The target query vectors are screened, and finally the decoder generates the predicted target rectangular frame coordinates; S44, after inputting the image-level concept weak label into the text encoder BERT, a feature representation of each concept is obtained, and then an inner product operation is performed between the feature representation of each concept and the feature representation corresponding to the predicted target rectangular box coordinates to obtain the category information and confidence of each target rectangular box; S45. Select a target rectangular box whose confidence is greater than a predetermined threshold as a pseudo label of the image-level concept.

5. A system using the open vocabulary target detection method for power construction scene images according to any one of claims 1 to 4, characterized in that: include: The database composition module is used to obtain the known dataset composition database, including the "Wukong" dataset, Object365 and Visual Genome datasets, GoldG+ dataset, Open Images dataset, and image-level multi-label dataset of power construction scenes; Pre-training module, used to use the "Wukong" dataset to perform pre-training of image-text embedding representation based on cross-modal contrastive learning, and obtain image embedding representation functions for extracting arbitrary image feature representation and text feature representation in the target detection network model And the text embedding representation function ; Training module for Object365 and Visual Genome datasets, GoldG+ datasets, and image embedding representation functions And text embedding representation function Train and optimize the target detection network model to obtain a target detection network model supervised by known concepts; A pseudo-label generation module is used to generate pseudo-labels for the coordinates of rectangular boxes of weakly supervised open vocabulary objects based on the weakly supervised detection model using the Open Images dataset and the image-level multi-label dataset of power construction scenes; A detection model acquisition module is used to train the target detection network model using pseudo labels to obtain an open vocabulary target detection model for power construction scene images; The detection result output module is used to adjust the input concept to the open vocabulary that needs to be detected as the input of the open vocabulary target detection model, and to score and screen the predicted target rectangular box to output the open vocabulary target detection result.

Images