Target detection method, device and server

Abstract

This specification provides a target detection method, device and server. Based on this method, before specific implementation, a large number of point-annotated sample images, a small number of fully-annotated sample images, and a sample transformation model including at least a point amplification module can be used according to weak semi-supervised learning rules based on point annotations to train a target detection model with high accuracy and good effect at a relatively low cost through weak semi-supervised learning; during specific implementation, the target image collected can be processed by using the above target detection model to accurately determine whether there is a target object in the target image; and when it is determined that there is a target object in the target image, the location information and object type of the target object in the target image can be further determined. Thereby, target detection for image data can be achieved efficiently and accurately at a relatively low processing cost.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more particularly to target detection methods, devices and servers. Background Art

[0002] In many application scenarios (for example, traffic road supervision scenarios, community security scenarios, oil field construction scenarios, etc.), it is often necessary to use corresponding detection models to perform target detection on the images collected by monitoring to determine whether there are corresponding security risks.

[0003] However, based on the existing methods, most of them need to first spend a lot of annotation costs to obtain a large number of fully annotated samples, and then use the fully annotated samples to train the corresponding detection model, and then use the detection model to perform specific target detection on the image. When the above methods are implemented, there are often technical problems such as long model training time and high training cost, which in turn affects the overall processing cost and processing efficiency.

[0004] Currently, no effective solution has been proposed to the above problems. Summary of the invention

[0005] This specification provides a target detection method, device and server, which can efficiently and accurately implement target detection for image data at a relatively low processing cost.

[0006] This specification provides a target detection method, including:

[0007] Get the target image;

[0008] The target image is processed by a target detection model to obtain a corresponding target processing result; wherein the target detection model is obtained by weak semi-supervised learning training in advance using a point-annotated sample image, a fully-annotated sample image, and a sample transformation model according to a weak semi-supervised learning rule based on point annotation; the sample transformation model at least includes: a point amplification module;

[0009] According to the target processing result, it is determined whether there is a target object in the target image; and when it is determined that there is a target object in the target image, the position information and the object type of the target object in the target image are determined.

[0010] In one embodiment, before acquiring the target image, the method further includes:

[0011] According to the weak semi-supervised learning rule based on point annotation, a point-annotated sample image and a fully-annotated sample image are obtained; wherein the number of the point-annotated sample images is greater than the number of the fully-annotated sample images; the fully-annotated sample image at least includes a labeling box for selecting a sample object, and a type label for the sample object carried by the labeling box; the point-annotated sample image at least includes a labeling point for indicating the sample object, and a type label for the sample object carried by the labeling point;

[0012] Processing the point-annotated sample image using the sample transformation model to obtain a processed point-annotated sample image carrying a pseudo-annotation frame;

[0013] Combining the fully annotated sample images and the processed point-annotated sample images to obtain a training sample set;

[0014] The training sample set is used to train the initial detection model to obtain a target detection model that meets the requirements.

[0015] In one embodiment, the sample transformation model further includes: a feature extraction network, a Transformer module, a point encoder, and a prediction head module;

[0016] The feature extraction network is connected to the Transformer module; the point amplification module is also connected to the point encoder; the point encoder is also connected to the Transformer module; the Transformer module is also connected to the prediction head module;

[0017] The prediction head module is used to determine and mark a pseudo-annotation box for selecting a sample object in the point-annotated sample image of the input sample transformation model based on the intermediate result data output by the Transformer module, so as to output the corresponding processed point-annotated sample image carrying the pseudo-annotation box.

[0018] In one embodiment, the feature extraction network is used to extract features from the point-annotated sample image of the input sample transformation model to obtain a corresponding sample feature map; and send the sample feature map to the Transformer module;

[0019] The point amplification module is used to receive the high-dimensional features of the point-annotated sample image output by the feature extraction network; and obtain the high-dimensional feature information at the annotation point through bilinear interpolation sampling in the high-dimensional features of the point-annotated sample image; according to the high-dimensional feature information at the annotation point, four auxiliary points associated with the annotation point and located in four different directions are determined through position offset prediction; and the type labels carried by the annotation point are respectively set on the four auxiliary points to obtain the amplified pseudo-annotation points.

[0020] In one embodiment, the point encoder comprises: an improved point encoder; wherein the improved point encoder comprises at least: an absolute position branch processing structure, a relative position branch processing structure, a category index branch processing structure, and a semantic alignment branch processing structure;

[0021] The point encoder is used to obtain and output corresponding joint codes by processing the annotation points in the point annotation sample image and the amplified pseudo annotation points.

[0022] In one embodiment, the point encoder processes the annotation points in the point annotation sample image and the amplified pseudo annotation points in the following manner to obtain and output the corresponding joint code:

[0023] Through the absolute position branch processing structure, according to the position coordinates of the annotation point and the position coordinates of the amplified pseudo annotation point, preset encoding processing is performed respectively to obtain the corresponding absolute position code;

[0024] Through the relative position branch processing structure, according to the position coordinates of the marking point and the position coordinates of the pseudo-marking point after expansion, determine and generate a corresponding relative position vector according to the offset distance of the marking point and the pseudo-marking point after expansion relative to the upper boundary, lower boundary, left boundary and right boundary of the sample image marked by the marking point; convert the relative position vector into corresponding relative ratio data; perform preset encoding processing on the relative ratio data to obtain corresponding relative position encoding;

[0025] The type label of the annotation point and the type label of the amplified pseudo annotation point are processed by the category index branch processing structure to obtain a corresponding category index code;

[0026] Through the semantic alignment branch processing structure, linear mapping is performed on the acquired high-dimensional features of the annotation points and the high-dimensional features of the amplified pseudo-annotation points to obtain corresponding image semantic feature codes; wherein the image semantic feature codes, absolute position codes, relative position codes, and category index codes have the same dimensions;

[0027] The absolute position coding, relative position coding, category index coding, and image semantic feature coding are used in combination to obtain a corresponding joint coding.

[0028] In one embodiment, the absolute position coding, relative position coding, category index coding, and image semantic feature coding are used in combination to obtain corresponding joint coding, including:

[0029] Performing an addition operation using the absolute position code, the relative position code and the category index code to obtain an intermediate code;

[0030] The intermediate code and the characteristic code are used to perform a multiplication operation to obtain the joint code.

[0031] In one embodiment, the Transformer module includes: a Transformer encoder and a Transformer decoder; wherein the Transformer encoder is connected to the feature extraction network and the Transformer decoder respectively; the Transformer decoder is also connected to the point encoder and the prediction head module;

[0032] The Transformer module is used to generate and output corresponding intermediate result data according to the input sample feature map, position coding, and joint coding.

[0033] In one embodiment, the Transformer module generates and outputs corresponding intermediate result data according to the input sample feature map, position coding, and joint coding in the following manner:

[0034] Processing the combination of the sample feature map and the position code through a Transformer encoder to generate corresponding embedded data; and sending the embedded data to a Transformer decoder;

[0035] Get the matching attention mask through the Transformer decoder;

[0036] The joint encoding is used to generate corresponding object query data through a Transformer decoder; and the attention mask, object query data and embedded data are used to generate and output corresponding intermediate result data.

[0037] In one embodiment, the Transformer decoder comprises: a decoding layer; wherein the decoding layer comprises: a self-attention layer, a cross-attention layer, and a feedforward neural network layer;

[0038] The self-attention layer is used to process the object query data using an attention mask to strengthen the interaction between the object query data indicating the same sample object;

[0039] The cross attention layer is used to refine the processed object query data output from the self-attention layer by using the embedded data to obtain refined object query data;

[0040] The feedforward neural network layer is used to predict the bounding box containing the sample object in the point-annotated sample image according to the refined object query data, and obtain corresponding intermediate result data.

[0041] In one embodiment, the sample transformation model is further connected to a matching loss module; wherein the matching loss module is configured with a matching loss function and a denoising loss function.

[0042] In one embodiment, generating corresponding object query data using the joint encoding through a Transformer decoder includes:

[0043] Get noisy data based on type labels;

[0044] Generate corresponding matching data according to the object query data;

[0045] combining the noise data and the matching data to obtain the object query data;

[0046] Correspondingly, the matched attention mask includes: a denoising mask area corresponding to the noise data, and a matching mask area corresponding to the matching data.

[0047] In one embodiment, the sample transformation model is trained in the following manner:

[0048] According to a weak semi-supervised learning rule based on point annotation, a fully annotated sample image is acquired and used to generate first sample data;

[0049] Constructing an initial transformation model; wherein the initial transformation model at least includes an initial point amplification module;

[0050] The initial transformation model is trained using the first sample data to obtain a sample transformation model that meets the requirements.

[0051] This specification also provides a target detection device, including:

[0052] An acquisition module, used for acquiring a target image;

[0053] A processing module, used to process the target image using a target detection model to obtain a corresponding target processing result; wherein the target detection model is obtained by weak semi-supervised learning training in advance using a point-annotated sample image, a fully-annotated sample image, and a sample transformation model according to a weak semi-supervised learning rule based on point annotation; the sample transformation model at least includes: a point amplification module;

[0054] The determination module is used to determine whether a target object exists in the target image according to the target processing result; and when it is determined that the target object exists in the target image, determine the position information and object type of the target object in the target image.

[0055] The present specification also provides a server, comprising a processor and a memory for storing processor executable instructions, wherein the processor implements the relevant steps of the target detection method when executing the instructions.

[0056] Based on the target detection method, device and server provided in this specification, before specific implementation, a large number of point-annotated sample images, a small number of fully-annotated sample images, and a sample transformation model including at least a point amplification module can be used according to weak semi-supervised learning rules based on point annotation to train a target detection model with high accuracy and good effect at a relatively low cost through weak semi-supervised learning; during specific implementation, the target image can be processed by using the above target detection model to accurately determine whether there is a target object in the target image; and when it is determined that there is a target object in the target image, the location information and object type of the target object in the target image can be further determined. By using a sample transformation model including at least a point amplification module for weak semi-supervised learning according to weak semi-supervised learning rules based on point annotation, only a small number of fully-annotated sample images need to be obtained and used, and mainly relying on and using point-annotated sample images with low annotation costs, a target detection model with high accuracy and good effect can be quickly trained, thereby effectively reducing the overall processing cost; and the target detection model trained based on the above method can be used to efficiently and accurately realize target detection for image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of this specification, the drawings required for use in the embodiments will be briefly introduced below. The drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 is a flow chart of a target detection method provided by an embodiment of this specification;

[0059] Figure 2 is a schematic diagram of an embodiment of an object detection method provided by an embodiment of this specification, in a scenario example;

[0060] Figure 3 is a schematic diagram of an embodiment of an object detection method provided by an embodiment of this specification, in a scenario example;

[0061] Figure 4 It is a schematic diagram of a composition structure of a sample transformation model provided by an embodiment of this specification;

[0062] Figure 5is a schematic diagram of an embodiment of an object detection method provided by an embodiment of this specification, in a scenario example;

[0063] Figure 6 is a schematic diagram of an embodiment of an object detection method provided by an embodiment of this specification, in a scenario example;

[0064] Figure 7 is a schematic diagram of an embodiment of an object detection method provided by an embodiment of this specification, in a scenario example;

[0065] Figure 8 It is another schematic diagram of the composition structure of a sample transformation model provided by an embodiment of this specification;

[0066] Fig. 9 is a schematic diagram of an embodiment of an object detection method provided by an embodiment of this specification, in a scenario example;

[0067] Fig.10 is a schematic diagram of an embodiment of an object detection method provided by an embodiment of this specification, in a scenario example;

[0068] Fig.11 is a schematic diagram of an embodiment of an object detection method provided by an embodiment of this specification, in a scenario example;

[0069] Fig.12 It is another structural diagram of a sample transformation model provided by an embodiment of this specification;

[0070] Fig.13 is a schematic diagram of an embodiment of an object detection method provided by an embodiment of this specification, in a scenario example;

[0071] Fig.14 is a schematic diagram of an embodiment of an object detection method provided by an embodiment of this specification, in a scenario example;

[0072] Fig.15 It is a schematic diagram of the structure of a server provided by an embodiment of this specification;

[0073] Fig.16 It is a schematic diagram of the structural composition of a target detection device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0074] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0075] See also Figure 1 As shown, the embodiment of this specification provides a target detection method, wherein the method is specifically applied to the server side. When implemented specifically, the method may include the following contents:

[0076] S101: Acquire a target image;

[0077] S102: Processing the target image using a target detection model to obtain a corresponding target processing result; wherein the target detection model is obtained by weak semi-supervised learning training in advance using a point-annotated sample image, a fully-annotated sample image, and a sample transformation model according to a weak semi-supervised learning rule based on point annotation; the sample transformation model at least includes: a point amplification module;

[0078] S103: determining whether a target object exists in the target image according to the target processing result; and determining the position information and object type of the target object in the target image when it is determined that the target object exists in the target image.

[0079] In some embodiments, the target image can be specifically understood as image data to be detected to determine whether a target object exists. Specifically, the target image can be a photo captured by a camera, or an image frame captured from a video. For different application scenarios, the target object can be different types of target objects.

[0080] For example, see Figure 2 As shown, in the oilfield construction scenario, the above-mentioned target image can specifically be a field image collected at the oilfield construction site; correspondingly, the target object to be detected can specifically be a risk object that may have safety hazards in the oilfield construction site, such as workers who are performing illegal construction activities that pose safety hazards in the oilfield construction site, or abnormal fire points, etc.

[0081] In specific implementation, the monitoring camera deployed at the oilfield construction site can collect the on-site image of the oilfield construction site in real time or at a fixed time as the target image to be detected; and then send the above target image to the server by wired or wireless means. The server can use the target detection model pre-trained based on weak semi-supervised learning to perform target detection on the target image to determine whether there is a risk object in the target image. In the case of determining that there is a risk object in the target image, the server can further determine the location information and risk type of the risk object; and generate corresponding risk warning information based on the location information and risk type of the risk object; and then send the above risk warning information to the user terminal held by the safety inspector at the oilfield construction site for prompting. Correspondingly, the safety inspector can go to the corresponding location in time to confirm the risk according to the above risk warning information, and make targeted treatment according to the risk type, so as to eliminate safety hazards in time and ensure the construction safety of the oilfield construction site.

[0082] The server may specifically include a background server applied to one side of the oilfield construction safety management center, capable of realizing functions such as data transmission and data processing. Specifically, the server may be, for example, an electronic device with data calculation, storage and network interaction functions. Alternatively, the server may also be a software program running in the electronic device to provide support for data processing, storage and network interaction. In this embodiment, the number of the servers is not specifically limited. The server may specifically be one server, or several servers, or a server cluster formed by several servers.

[0083] The user terminal may specifically include a front end applied to the security inspector side, capable of realizing functions such as data collection and data transmission. Specifically, the user terminal may be, for example, an electronic device such as a tablet computer, a laptop computer, a smart phone, etc. Alternatively, the user terminal may also be a software application that can be run in the above electronic devices. For example, it may be a security management center client APP running on a smart phone, etc.

[0084] For another example, in a traffic road supervision scenario, the target image may specifically be an image of a vehicle traveling on the road collected by a traffic road monitoring system; correspondingly, the target object to be detected may specifically be a vehicle object traveling on the road with a risk of illegal driving behavior.

[0085] Of course, it should be noted that the application scenarios, target images and target objects listed above are only schematic illustrations. In specific implementation, the target detection method can also be applied to other application scenarios according to specific circumstances and requirements. Accordingly, the target images and target objects involved can also include target images and target objects in other application scenarios. This specification does not limit this.

[0086] In some embodiments, the above-mentioned target detection model may specifically include a neural network model that is pre-trained according to weak semi-supervised learning rules based on point annotation, by introducing and using a sample transformation model that includes at least a point amplification module, using a large number of point-annotated sample images, and a small number of fully-annotated sample images, through weak semi-supervised learning, and is able to automatically detect target objects in input image data.

[0087] The above-mentioned point-annotated sample images can be specifically understood as sample images annotated with annotation points for indicating sample objects and type labels of sample objects. The above-mentioned fully-annotated sample images can be specifically understood as sample images annotated with annotation boxes for selecting sample objects and type labels of sample objects. Usually, the annotation cost of fully-annotated sample images is higher than that of point-annotated sample images.

[0088] Specifically, the point-annotated sample image may include one or more sample objects; and each sample object is annotated with a corresponding annotation point and type label.

[0089] The above-mentioned sample transformation model can be specifically understood as an algorithm model constructed and trained according to weak semi-supervised learning rules based on point annotation, which can predict and automatically annotate a bounding box (pseudo-annotation box) that can select the sample object based on the input point annotation sample image.

[0090] Furthermore, the sample transformation model at least includes a point: point amplification module. The point amplification module can be used to amplify a label point originally corresponding to a sample object in a point-annotated sample image into multiple label points. Furthermore, the sample transformation model can comprehensively utilize the information of the original label point and the information of multiple amplified pseudo-label points to mine relatively more comprehensive and rich image features, so as to more accurately predict the pseudo-label box used to select the sample object.

[0091] In this embodiment, according to the weak semi-supervised learning rules based on point annotation, by introducing and using the above-mentioned sample transformation model that at least includes a point amplification module, the point-annotated sample image with relatively low annotation cost can be automatically transformed into a processed point-annotated sample image with a pseudo-annotation box with relatively good effect; then, the above-mentioned processed point-annotated sample image and a smaller number of fully-annotated sample images can be used to efficiently train a target detection model with relatively high accuracy and relatively good effect through weak semi-supervised learning.

[0092] The above-mentioned weak semi-supervised learning rules based on point annotation can be specifically understood as a set of rules based on the algorithm conception of weak semi-supervised learning and the characteristics of point-annotated sample images, on how to construct a training sample transformation model, and how to use the above-mentioned sample transformation model to train the target detection model using point-annotated sample images and fully-annotated sample images.

[0093] In some embodiments, see Figure 3 As shown, before acquiring the target image, the method may further include the following contents when implemented:

[0094] S1: According to a weak semi-supervised learning rule based on point annotation, a point-annotated sample image and a fully-annotated sample image are obtained; wherein the number of the point-annotated sample images is greater than the number of the fully-annotated sample images; the fully-annotated sample image at least includes a labeling box for selecting a sample object, and a type label for the sample object carried by the labeling box; the point-annotated sample image at least includes a labeling point for indicating a sample object, and a type label for the sample object carried by the labeling point;

[0095] S2: Processing the point-annotated sample image using the sample transformation model to obtain a processed point-annotated sample image carrying a pseudo-annotation frame;

[0096] S3: combining the fully annotated sample image and the processed point annotated sample image to obtain a training sample set;

[0097] S4: Using the training sample set to train an initial detection model to obtain a target detection model that meets the requirements.

[0098] The above-mentioned type label may specifically include a marking label for indicating the object type of the sample object.

[0099] In specific implementation, the sample transformation model can be used to process the point-annotated sample images to automatically transform the original point-annotated sample images containing only the annotation points into processed point-annotated sample images containing pseudo-annotation frames; and then the processed point-annotated sample images can be mixed with a small amount of fully annotated sample images to obtain a training sample set for training the target detection model. Then, the training sample set can be used to perform weak semi-supervised learning on the initial detection model to obtain a target detection model that meets the requirements.

[0100] Among them, the above-mentioned weak semi-supervised learning can be specifically understood as a model training method that uses a larger amount of weakly labeled data (for example, point-labeled sample images) and a smaller amount of instance-level labeled data (for example, fully labeled sample images) to train and learn the model.

[0101] The above initial detection model may specifically include an initial model based on the FCOS structure.

[0102] The above-mentioned FCOS (Fully Convolutional One-Stage Object Detection) can specifically refer to an anchor-free detection model, based on which the category of the point on the feature map downsampled S times can be predicted first; then the size and position of the bounding box are determined by the four values ​​of l, r, t, and b of the predicted point, which can help NMS suppress low-quality boxes and further improve the performance of the network. Among them, l, r, t, and b represent the distance between the point and the left boundary, right boundary, upper boundary, and lower boundary of the bounding box, respectively.

[0103] The sample transformation model may specifically be an algorithm model that is pre-constructed according to weak semi-supervised learning rules based on point annotation and trained using fully annotated sample images.

[0104] In some embodiments, see Figure 4 As shown, the sample transformation model may further include: a feature extraction network, a Transformer module, a point encoder, a prediction head module and other structures;

[0105] The feature extraction network is connected to the Transformer module; the point amplification module is also connected to the point encoder; the point encoder is also connected to the Transformer module; the Transformer module is also connected to the prediction head module;

[0106] In specific implementation, the feature extraction network can be used to extract features from the point-annotated sample image of the input sample transformation model to obtain the corresponding sample feature map; and send the sample feature map to the Transformer module. In addition, the feature extraction network will also send the high-dimensional features of the point-annotated sample image extracted during the acquisition of the sample feature map to the point amplification module.

[0107] The point amplification module can be specifically used to receive the high-dimensional features of the point-annotated sample image output by the feature extraction network; and obtain the high-dimensional feature information at the annotation point through bilinear interpolation sampling in the high-dimensional features of the point-annotated sample image; obtain the position offset prediction value through position offset prediction based on the high-dimensional feature information at the annotation point; determine four auxiliary points associated with the annotation point and located in four different directions based on the coordinates of the original annotation point and the position offset prediction value; and set the type labels carried by the annotation point on the four auxiliary points respectively to obtain the amplified pseudo-annotation point.

[0108] For details, see Figure 5 As shown, the point expansion module can be used to take the existing annotation point as the center, and through position offset prediction, four different auxiliary points located in different directions of the upper left, upper right, lower left, and lower right of the annotation point, and indicating the same sample object as the annotation point, can be found as the pseudo-annotation points after expansion. Thus, the original one-to-one correspondence between one annotation point and one sample object can be expanded to a many-to-one correspondence between five annotation points (existing annotation points and pseudo-annotation points after expansion) and one sample object. Then, the many-to-one correspondence can be used to replace the original one-to-one correspondence, so that richer and more diverse annotation point information can be mined and used, and the prediction of the bounding box of the sample object can be completed more accurately.

[0109] The point encoder can be specifically used to obtain corresponding joint coding through relevant coding processing according to the labeling point information in the point labeling sample image and the amplified pseudo labeling point information.

[0110] The Transformer module can be specifically used to label the joint encoding of the sample image and the sample feature map according to the input points, and obtain the output corresponding intermediate result data through corresponding processing.

[0111] The prediction head module is used to determine and mark a pseudo-annotation box for selecting a sample object in the point-annotated sample image of the input sample transformation model based on the intermediate result data output by the Transformer module, so as to output the corresponding processed point-annotated sample image carrying the pseudo-annotation box.

[0112] Among them, the above-mentioned Transformer module, specifically by introducing and using the Attention mechanism, can reduce the distance between any two positions in the input data sequence to a constant, so that when analyzing and predicting longer data, the effect of capturing semantic associations with longer intervals is relatively better; in addition, the Transformer module is not a sequential structure similar to RNN, has better parallelism, conforms to the existing GPU framework, and can use distributed GPUs for parallel training to improve model training efficiency.

[0113] The feature extraction network may specifically include: a residual network, for example, a backbone network of ResNet50. The backbone network of ResNet50 has 50 layers in total. Based on the network, it can be divided into five parts according to the feature scale. Each part uses a different size and number of convolution kernels, and outputs feature maps of different scales and numbers of channels. Finally, the last part can output a feature map C5 of 2048 dimensions as the corresponding sample feature map.

[0114] The prediction head module may specifically include: a multilayer perceptron (MLP). Based on the multilayer perceptron, the prediction head module can use the intermediate result data output by the Transformer module to predict the bounding box containing the sample object, thereby determining and marking the corresponding pseudo-annotation box.

[0115] In some embodiments, the point encoder may specifically include: an improved point encoder; wherein the improved point encoder includes at least: an absolute position branch processing structure, a relative position branch processing structure, a category index branch processing structure, and a semantic alignment branch processing structure;

[0116] The point encoder is used to obtain and output corresponding joint codes by processing the annotation points in the point annotation sample image and the amplified pseudo annotation points.

[0117] For details, please refer to Figure 6 As shown, based on the above-mentioned improved point encoder, by additionally introducing and utilizing the relative position branch processing structure and the semantic alignment branch processing structure, it is possible to fully and comprehensively utilize the encoding based on different dimensions, and obtain a joint encoding with higher integration and richer information, and then subsequently generate relatively better object query data based on the above-mentioned joint encoding. The object query data will be described in detail later. Among them, x and y respectively represent the horizontal coordinate and vertical coordinate of the annotation point (including the annotation point and the pseudo-annotation point after expansion); c represents the type label of the annotation point; w and h respectively represent the width and height of the point annotation sample image.

[0118] In some embodiments, based on the above improved point encoder, when specifically implemented, refer to Figure 7 As shown, the point encoder obtains and outputs the corresponding joint code by processing the annotation points in the point annotation sample image and the amplified pseudo annotation points in the following manner:

[0119] S1: Through the absolute position branch processing structure, according to the position coordinates of the annotation point and the position coordinates of the amplified pseudo annotation point, a preset encoding process is performed respectively to obtain the corresponding absolute position code;

[0120] S2: by using the relative position branch processing structure, according to the position coordinates of the marking point and the position coordinates of the amplified pseudo-marking point, determine and generate a corresponding relative position vector according to the offset distance of the marking point and the amplified pseudo-marking point relative to the upper boundary, lower boundary, left boundary and right boundary of the marking sample image; convert the relative position vector into corresponding relative ratio data; perform preset encoding processing on the relative ratio data to obtain a corresponding relative position code;

[0121] S3: Processing the type label of the annotation point and the type label of the amplified pseudo annotation point through the category index branch processing structure to obtain a corresponding category index code;

[0122] S4: linearly mapping the acquired high-dimensional features of the annotation points and the high-dimensional features of the amplified pseudo-annotation points through the semantic alignment branch processing structure to obtain corresponding image semantic feature codes; wherein the image semantic feature codes, absolute position codes, relative position codes, and category index codes have the same dimension;

[0123] S5: Combining the absolute position coding, relative position coding, category index coding, and image semantic feature coding to obtain a corresponding joint coding.

[0124] Among them, the preset encoding process may specifically refer to sin encoding.

[0125] In specific implementation, when generating absolute position codes, the absolute position branch processing structure can be used to perform sin encoding on the position coordinates of the annotation points of the same sample object and the amplified pseudo-annotation points, respectively, to obtain a 256-dimensional absolute position code corresponding to each annotation point.

[0126] When generating relative position coding, the relative position branch processing structure can be used to calculate the offset distance (top, abbreviated as t), the offset distance (down, abbreviated as d), the offset distance (left, abbreviated as l), and the offset distance (right, abbreviated as r) of each annotation point relative to the upper boundary of the annotation sample image at the point where it is located, based on the position coordinates of the annotation points for the same sample object and the pseudo-annotation points after expansion; and based on the above offset distances, the relative position vectors corresponding to each annotation point are generated, which are recorded as: (top, down, left, right). Based on the above relative position vectors, the ratio of the offset distance relative to the upper boundary to the offset distance relative to the lower boundary, and the ratio of the offset distance relative to the left boundary to the offset distance relative to the right boundary are calculated; and the relative ratio data corresponding to the annotation point is constructed using the above two ratios, which are recorded as: (top / down, left / right). Finally, the above relative ratio data is sin-encoded to obtain the corresponding 256-dimensional relative position coding.

[0127] When generating the category index code, the category index branch processing structure is used to process the annotation points for the same sample object and the type labels of the amplified pseudo annotation points to obtain a 256-dimensional category index code corresponding to each annotation point.

[0128] When generating image semantic feature codes, the semantic alignment branch processing structure is used to annotate sample images through processing points, and the annotation points for the same sample object and the high-dimensional features of the amplified pseudo-annotation points are extracted. After interpolation sampling, linear mapping is performed to obtain 256-dimensional image semantic feature codes corresponding to each annotation point.

[0129] Finally, by combining the five codes based on different levels, namely, the absolute position code, the relative position code, the category index code, and the image semantic feature code, the information contained in the different codes is fully integrated to obtain the required joint code.

[0130] In some embodiments, see Figure 6 As shown, the above combination uses the absolute position coding, relative position coding, category index coding, and image semantic feature coding to obtain the corresponding joint coding. When implemented specifically, the following contents may be included:

[0131] S1: performing an addition operation using the absolute position code, the relative position code and the category index code to obtain an intermediate code;

[0132] S2: Perform multiplication operation using the intermediate code and the feature code to obtain the joint code.

[0133] Based on the above embodiments, the above five encodings based on different levels can be effectively and comprehensively utilized to obtain a relatively good joint encoding effect.

[0134] In some embodiments, see Figure 8 As shown, the Transformer module may specifically include: a Transformer encoder (Encoder) and a Transformer decoder (Decoder); wherein the Transformer encoder is connected to the feature extraction network and the Transformer decoder respectively; the Transformer decoder is also connected to the point encoder and the prediction head module;

[0135] The Transformer module is used to generate and output corresponding intermediate result data according to the input sample feature map, position coding, and joint coding.

[0136] The above positional encoding may specifically be a positional encoding obtained based on a point-annotated sample image.

[0137] In some embodiments, in combination Figure 8 , see Fig. 9 As shown, in a specific implementation, the Transformer module can generate and output corresponding intermediate result data according to the input sample feature map, position coding, and joint coding in the following manner:

[0138] S1: Processing the combination of the sample feature map and the position code through a Transformer encoder to generate corresponding embedded data (e.g., embeddings); and sending the embedded data to a Transformer decoder;

[0139] S2: Get the matching attention mask (e.g., Attention mask) through the Transformer decoder;

[0140] S3: Generate corresponding object query data (e.g., object query) using the joint encoding through a Transformer decoder, and generate and output corresponding intermediate result data using the attention mask, object query data, and embedded data.

[0141] Among them, the object query data (object query) can be specifically understood as a query object, which is a way to describe the target to be detected in the detection task. It is specifically an important structure in Transformer. Based on the object query data, the detection task can be converted into a similarity calculation between the prediction result and the feature map. In Detr, each object query can be regarded as a representation of a target, usually generated by the Transformer decoder, and its vector representation can be regarded as an encoding of the category and position of the target. During prediction, each object query is matched with the embedded data output by the Transformer encoder (for example, encoder embeddings) to determine which position in the embedded data the object query should be associated with, so as to obtain an accurate description of the target position and category and realize target detection.

[0142] The above attention mask can be found in detail Fig.10 As shown, the attention mask using the above structure can strengthen the interaction between object query data of different annotation points of the same sample object; at the same time, it can avoid the interaction between object query data of different annotation points of different sample objects, so that the model can focus relatively on jointly utilizing the query data of multiple annotation points of the same sample object, so as to make full use of the one-to-many correspondence mined by the point amplification module, and then more accurately and quickly find the bounding box used to select the corresponding sample object.

[0143] In some embodiments, see Fig.11 As shown, the Transformer decoder may specifically include: a decoding layer; wherein the decoding layer includes: a self-attention layer, a cross-attention layer, and a feedforward neural network layer and other structures; in addition, the above decoding layer may also include structures such as an intermediate layer (for example, add&norm). Among them, box indicates the intermediate result data used to predict the bounding box, and class indicates the intermediate result data used to predict the type label. It should be noted that in actual use, only the intermediate result data containing the box is output and used. The input decoder embeddings (tgt) can be understood as the copy data of the object query data (object query). Positional embeddings can specifically be positional features extracted based on the embedded data output by the Transformer encoder.

[0144] The self-attention layer (e.g., self-attention) may be specifically used to process the object query data using an attention mask to strengthen the interaction between the object query data indicating the same sample object;

[0145] The cross-attention layer (e.g., cross-attention) may be specifically used to refine the processed object query data output from the self-attention layer using the embedded data to obtain refined object query data;

[0146] The feedforward neural network layer (eg, FFN) may be specifically used to predict a bounding box containing a sample object in a point-annotated sample image according to the refined object query data, and obtain corresponding intermediate result data.

[0147] The Transformer decoder may include multiple decoding layers. The intermediate result data output by the previous decoding layer may be input to the next decoding layer as the object query data received by the next decoding layer. Then, the next decoding layer may further process the intermediate result data obtained by the previous decoding layer to obtain relatively more accurate updated intermediate result data, which is then input to the next decoding layer. Finally, the intermediate result data with higher accuracy may be output by the Transformer decoder.

[0148] The Feed Forward Neural Network (FFN) is the simplest neural network, where neurons are arranged in layers and each neuron is only connected to the neurons in the previous layer. Based on the network layer, it can receive the output of the previous layer and output it to the next layer, and there is no feedback between the layers.

[0149] In this embodiment, by constructing a self-attention layer and a cross-attention layer, two different attention mechanisms can be introduced in the Transformer decoder so as to mix and use two different embedding sequences (for example, embedded data and object query data). Among them, the two embedding sequences usually have the same dimension, and the two embedding sequences can be in different mode forms (for example, text, image, audio). Specifically, one sequence can be used as the input Q to define the length of the output sequence; another sequence can be used to provide the input K and V. Among them, the input of the self-attention mechanism usually comes from the same sequence for a single processing embedding sequence; and the cross-attention mechanism input usually comes from different sequences, which is used to asymmetrically combine two different embedding sequences for processing, for example, one sequence is used as the query Q input, and the other sequence is used as the key K and value V input.

[0150] For specific implementation, see Fig.11 It can be seen that the transformer decoder can first extract the position encoding information p of the relevant annotation points based on the object query data s; and input the object query data into the FFN layer for prediction to obtain the content feature information T; at the same time, based on the object query data, combined with the position encoding information p s The initial V component, initial K component and initial Q component are obtained, and the initial V component, initial K component and initial Q component are input into the self-attention layer for corresponding processing, and the first object query data is output. The embedded data, the first object query data, and the previously obtained position encoding information p are then combined and used. s The content feature information T is transported accordingly to obtain the corresponding first V component, first K component and first Q component; the first V component, first K component and first Q component are then input into the cross attention layer for processing to achieve the refinement of the object query data using the embedded data, and the second object query data with richer information is output. Finally, the second object query data is respectively input into the FFN layer for bounding box prediction and the FFN layer for object type prediction for corresponding prediction processing to obtain the corresponding bounding box prediction data and object type prediction data; the bounding box prediction data and object type prediction data are combined to obtain and output the intermediate result data.

[0151] In some embodiments, see Fig.12 As shown, the sample transformation model can also be connected to a matching loss module; wherein the matching loss module is configured with a matching loss function and a denoising loss function. Specifically, the matching loss model can be connected after the prediction head module. Based on the matching loss module, by additionally introducing a label denoising task and combining the label denoising task with the matching optimization of the model, it is possible to use feature information related to label denoising for assistance, so as to more quickly train and obtain a sample transformation model that can automatically output a bounding box and meets the requirements.

[0152] In some embodiments, the joint encoding is used to generate corresponding object query data through a Transformer decoder. When implemented specifically, the following contents may be included:

[0153] S1: Obtaining noise data based on type labels (or denoising part);

[0154] S2: Generate corresponding matching data (or matching part) according to the object query data;

[0155] S3: combining the noise data and the matching data to obtain the object query data;

[0156] Correspondingly, the matching attention mask may specifically include: a denoising mask area corresponding to the noise data, and a matching mask area corresponding to the matching data.

[0157] The noise data may specifically be a noise label (for example, a noise label) obtained by performing methods such as label flipping and artificial noise addition on the type label of the annotation point.

[0158] Specifically, based on the above concept, the original attention mask can be transformed by combining the noise data in the object query data and the data structure of the matching data, and the transformed attention mask can be used as the matching attention mask. Fig.13 As shown in the figure. The modified attention mask contains not only the matching mask area, but also the denoising mask area. For the attention mask based on the above structure, see Fig.12 The cross mark in the Transformer decoder can not only strengthen the interaction between object query data indicating the same sample object, but also effectively prevent the noise data in the object query data from initiating interactive queries to the matching data, so as to avoid the influence of the leakage of matching data to noise data during model training on the learning of type label prediction based on noise data; in addition, it can also effectively prevent the interaction between noise data in object query data for different sample objects, as well as the interaction between matching data. Thus, the effectiveness of the model training process can be effectively guaranteed.

[0159] In some embodiments, see Fig.14 As shown, the sample transformation model can be trained in the following way:

[0160] S1: According to the weak semi-supervised learning rule based on point annotation, obtain and use the fully annotated sample image to generate the first sample data;

[0161] S2: constructing an initial transformation model; wherein the initial transformation model at least includes an initial point amplification module;

[0162] S3: Using the first sample data, the initial transformation model is trained to obtain a sample transformation model that meets the requirements.

[0163] When specifically generating the first sample data, any point in the annotation box in the fully annotated sample image can be selected as the annotation point, and an object type label indicating the sample object can be marked on the annotation point; at the same time, the original annotation box in the fully annotated sample image is recorded as the true result for evaluating the model prediction result, and the corresponding first sample data is obtained.

[0164] When constructing the initial transformation model, the initial transformation model may include, in addition to the initial point amplification module, an initial feature extraction network, an initial Transformer module, an initial point encoder, an initial prediction head module, and a matching loss module.

[0165] Specifically, when the first sample data is used to train the initial transformation model, in each round of training, the initial transformation model can be used to process the first sample data to obtain the corresponding prediction result; then the matching loss module is used to match the prediction result and the real result of the first sample data, and the matching result is used to calculate the loss cost (including: reconstruction loss for type label prediction and matching loss for bounding box prediction) based on the relevant loss function; then, according to the loss cost, the model parameters are optimized and adjusted in a targeted manner in the direction of minimizing the loss cost. Through multiple rounds of training and adjustment, the accuracy of the model is continuously improved to finally obtain a sample transformation model that meets the requirements.

[0166] In some embodiments, the matching loss module includes: a Hungarian matching loss module based on the Hungarian algorithm.

[0167] Among them, the Hungarian algorithm is a matching algorithm, that is, an algorithm for finding the maximum matching in graph theory, and is mostly used to solve problems related to bipartite graph matching.

[0168] In some embodiments, after determining the location information and object type of the target object in the target image, the method may further include the following when implemented: automatically using a bounding box in the target image to select the target object in the target image according to the location information and object type of the target object in the target image; and marking the object type of the target object at a critical position of the bounding box to obtain a processed target image. Displaying the processed target image to the staff.

[0169] In addition, during specific implementation, it is also possible to detect whether there is a security risk based on the location information and object type of the target object in the target image; if it is determined that there is a security risk, a corresponding alarm prompt can be triggered to prompt the staff to eliminate the relevant security risk in a timely manner.

[0170] As can be seen from the above, based on the target detection method provided by the embodiment of this specification, before specific implementation, the target detection model with high accuracy and good effect can be obtained by training at a relatively low cost according to the weak semi-supervised learning rule based on point annotation, using the point-annotated sample image, the fully annotated sample image, and the sample transformation model including at least the point amplification module; during specific implementation, the target image can be processed by using the above target detection model to accurately determine whether there is a target object in the target image; and when it is determined that there is a target object in the target image, the location information and object type in the target image can be further determined. By using the sample transformation model including at least the point amplification module according to the weak semi-supervised learning rule based on point annotation, only a small number of fully annotated sample images need to be obtained and used, and mainly relying on and using the point-annotated sample images with low annotation cost, a target detection model with high accuracy and good effect can be quickly trained, thereby effectively reducing the overall processing cost, and using the target detection model trained based on the above method to efficiently and accurately realize target detection for image data.

[0171] This specification also provides a method for training a target detection model, which may include the following steps during implementation:

[0172] S1: According to a weak semi-supervised learning rule based on point annotation, a point-annotated sample image and a fully-annotated sample image are obtained; wherein the number of the point-annotated sample images is greater than the number of the fully-annotated sample images; the fully-annotated sample image at least includes a labeling box for selecting a sample object, and a type label for the sample object carried by the labeling box; the point-annotated sample image at least includes a labeling point for indicating a sample object, and a type label for the sample object carried by the labeling point;

[0173] S2: Processing the point-annotated sample image using the sample transformation model to obtain a processed point-annotated sample image carrying a pseudo-annotation frame;

[0174] S3: combining the fully annotated sample image and the processed point annotated sample image to obtain a training sample set;

[0175] S4: Using the training sample set to train an initial detection model to obtain a target detection model that meets the requirements.

[0176] Based on the above embodiments, a target detection model with higher accuracy can be obtained by efficiently training at a lower cost through weak semi-supervised learning rules based on point annotation, using a large number of point-annotated sample images, a small number of fully-annotated sample images, and a sample transformation model that includes at least a point amplification module.

[0177] This specification also provides a training method for a sample transformation model, which may include the following steps during implementation:

[0178] S1: According to the weak semi-supervised learning rule based on point annotation, obtain and use the fully annotated sample image to generate the first sample data;

[0179] S2: constructing an initial transformation model; wherein the initial transformation model at least includes an initial point amplification module;

[0180] S3: Using the first sample data, the initial transformation model is trained to obtain a sample transformation model that meets the requirements.

[0181] Based on the above embodiments, a sample transformation model with better effect can be trained based on weak semi-supervised learning of point annotation.

[0182] This specification also provides a server, see Fig.15 As shown, the server includes a network communication port 1501, a processor 1502 and a memory 1503, and the above structures are connected through internal cables so that each structure can perform specific data interaction.

[0183] The network communication port 1501 can be specifically used to obtain a target image.

[0184] The processor 1502 can be specifically used to process the target image using a target detection model to obtain a corresponding target processing result; wherein the target detection model is pre-trained through weak semi-supervised learning according to a weak semi-supervised learning rule based on point annotation using a point-annotated sample image, a fully-annotated sample image, and a sample transformation model; the sample transformation model at least includes: a point amplification module; determining whether there is a target object in the target image based on the target processing result; and when it is determined that there is a target object in the target image, determining the location information and object type of the target object in the target image.

[0185] The memory 1503 may be specifically used to store corresponding instruction programs.

[0186] In this embodiment, the network communication port 1501 can be a virtual port that is bound to different communication protocols so that different data can be sent or received. For example, the network communication port can be a port responsible for web data communication, a port responsible for FTP data communication, or a port responsible for email data communication. In addition, the network communication port can also be a physical communication interface or communication chip. For example, it can be a wireless mobile network communication chip, such as GSM, CDMA, etc.; it can also be a Wifi chip; it can also be a Bluetooth chip.

[0187] In this embodiment, the processor 1502 may be implemented in any appropriate manner. For example, the processor may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) executable by the (micro)processor, a logic gate, a switch, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, etc. This specification does not limit this.

[0188] In this embodiment, the memory 1503 may include multiple levels. In a digital system, anything that can store binary data can be a memory; in an integrated circuit, a circuit with a storage function that has no physical form is also called a memory, such as RAM, FIFO, etc.; in a system, a storage device with a physical form is also called a memory, such as a memory stick, TF card, etc.

[0189] The embodiments of the present specification also provide a computer-readable storage medium based on the above-mentioned target detection method, wherein the computer-readable storage medium stores computer program instructions, which, when executed, implement: acquiring a target image; processing the target image using a target detection model to obtain a corresponding target processing result; wherein the target detection model is pre-trained through weak semi-supervised learning according to a weak semi-supervised learning rule based on point annotation using a point-annotated sample image, a fully-annotated sample image, and a sample transformation model; the sample transformation model at least includes: a point amplification module; determining whether a target object exists in the target image based on the target processing result; and when it is determined that a target object exists in the target image, determining the location information and object type of the target object in the target image.

[0190] In this embodiment, the storage medium includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a cache, a hard disk (HDD), or a memory card. The memory may be used to store computer program instructions. The network communication unit may be an interface for network connection communication set in accordance with the standard specified by the communication protocol.

[0191] In this embodiment, the functions and effects specifically implemented by the program instructions stored in the computer-readable storage medium can be explained in comparison with other implementations and will not be described in detail here.

[0192] See also Fig.16As shown, at the software level, the embodiment of this specification also provides a target detection device, which may specifically include the following structural modules:

[0193] An acquisition module 1601 may be specifically used to acquire a target image;

[0194] The processing module 1602 may be specifically used to process the target image using a target detection model to obtain a corresponding target processing result; wherein the target detection model is obtained by using a point-annotated sample image, a fully-annotated sample image, and a sample transformation model in advance according to a weak semi-supervised learning rule based on point annotation, through weak semi-supervised learning training; the sample transformation model at least includes: a point amplification module;

[0195] The determination module 1603 may be specifically configured to determine whether a target object exists in the target image according to the target processing result; and when it is determined that a target object exists in the target image, determine the position information and object type of the target object in the target image.

[0196] In some embodiments, before acquiring the target image, the device can also be specifically used to: acquire a point-annotated sample image and a fully-annotated sample image according to a weak semi-supervised learning rule based on point annotation; wherein the number of point-annotated sample images is greater than the number of fully-annotated sample images; the fully-annotated sample image at least includes a labeling box for selecting a sample object, and a type label for the sample object carried by the labeling box; the point-annotated sample image at least includes a labeling point for indicating the sample object, and a type label for the sample object carried by the labeling point; the point-annotated sample image is processed using a sample transformation model to obtain a processed point-annotated sample image carrying a pseudo-annotation box; the fully-annotated sample image and the processed point-annotated sample image are combined to obtain a training sample set; and the initial detection model is trained using the training sample set to obtain a target detection model that meets the requirements.

[0197] In some embodiments, the sample transformation model may further include: a feature extraction network, a Transformer module, a point encoder, a prediction head module, etc.;

[0198] The feature extraction network is connected to the Transformer module; the point amplification module is also connected to the point encoder; the point encoder is also connected to the Transformer module; the Transformer module is also connected to the prediction head module;

[0199] The prediction head module is used to determine and mark a pseudo-annotation box for selecting a sample object in the point-annotated sample image of the input sample transformation model based on the intermediate result data output by the Transformer module, so as to output the corresponding processed point-annotated sample image carrying the pseudo-annotation box.

[0200] In some embodiments, the feature extraction network is used to extract features from the point-annotated sample image of the input sample transformation model to obtain a corresponding sample feature map; and send the sample feature map to the Transformer module;

[0201] The point amplification module is used to receive the high-dimensional features of the point-annotated sample image output by the feature extraction network; and obtain the high-dimensional feature information at the annotation point through bilinear interpolation sampling in the high-dimensional features of the point-annotated sample image; according to the high-dimensional feature information at the annotation point, four auxiliary points associated with the annotation point and located in four different directions are determined through position offset prediction; and the type labels carried by the annotation point are respectively set on the four auxiliary points to obtain the amplified pseudo-annotation points.

[0202] In some embodiments, the point encoder may include: an improved point encoder; wherein the improved point encoder includes at least: an absolute position branch processing structure, a relative position branch processing structure, a category index branch processing structure, and a semantic alignment branch processing structure;

[0203] The point encoder is used to obtain and output corresponding joint codes by processing the annotation points in the point annotation sample image and the amplified pseudo annotation points.

[0204] In some embodiments, the point encoder can specifically obtain and output the corresponding joint code by processing the annotation points in the point-annotated sample image and the amplified pseudo-annotated points in the following manner: through the absolute position branch processing structure, according to the position coordinates of the annotation points and the position coordinates of the amplified pseudo-annotated points, respectively perform preset encoding processing to obtain the corresponding absolute position code; through the relative position branch processing structure, according to the position coordinates of the annotation points and the position coordinates of the amplified pseudo-annotated points, determine and generate the corresponding relative position vector according to the offset distance of the annotation points and the amplified pseudo-annotated points relative to the upper boundary, lower boundary, left boundary, and right boundary of the point-annotated sample image; convert the relative position vector into the corresponding relative ratio data; performing preset encoding processing on the relative ratio data to obtain corresponding relative position encoding; processing the type label of the annotation point and the type label of the amplified pseudo annotation point through a category index branch processing structure to obtain corresponding category index encoding; performing linear mapping on the acquired high-dimensional features of the annotation point and the amplified high-dimensional features of the pseudo annotation point through a semantic alignment branch processing structure to obtain corresponding image semantic feature encoding; wherein the image semantic feature encoding, absolute position encoding, relative position encoding, and category index encoding have the same dimension; combining the absolute position encoding, relative position encoding, category index encoding, and image semantic feature encoding to obtain corresponding joint encoding.

[0205] In some embodiments, when the device is implemented, the absolute position code, relative position code, category index code, and image semantic feature code can be combined and used in the following manner to obtain a corresponding joint code: use the absolute position code, relative position code and category index code to perform an addition operation to obtain an intermediate code; use the intermediate code and feature code to perform a multiplication operation to obtain the joint code.

[0206] In some embodiments, the Transformer module may specifically include: a Transformer encoder and a Transformer decoder, etc.; wherein the Transformer encoder is connected to the feature extraction network and the Transformer decoder respectively; the Transformer decoder is also connected to the point encoder and the prediction head module;

[0207] The Transformer module is used to generate and output corresponding intermediate result data according to the input sample feature map, position coding, and joint coding.

[0208] In some embodiments, the Transformer module can specifically generate and output corresponding intermediate result data according to the input sample feature map, position code, and joint code in the following manner: process the combination of the sample feature map and the position code through the Transformer encoder to generate corresponding embedded data; and send the embedded data to the Transformer decoder; obtain the matching attention mask through the Transformer decoder; generate corresponding object query data through the Transformer decoder using the joint code; and generate and output corresponding intermediate result data using the attention mask, object query data and embedded data.

[0209] In some embodiments, the Transformer decoder may specifically include: a decoding layer; wherein the decoding layer includes: a self-attention layer, a cross-attention layer, and a feedforward neural network layer;

[0210] The self-attention layer is used to process the object query data using an attention mask to strengthen the interaction between the object query data indicating the same sample object;

[0211] The cross attention layer is used to refine the processed object query data output from the self-attention layer by using the embedded data to obtain refined object query data;

[0212] The feedforward neural network layer is used to predict the bounding box containing the sample object in the point-annotated sample image according to the refined object query data, and obtain corresponding intermediate result data.

[0213] In some embodiments, the sample transformation model may be further connected to a matching loss module; wherein the matching loss module is configured with a matching loss function and a denoising loss function.

[0214] In some embodiments, when the device is specifically implemented, the corresponding object query data can be generated by using the joint encoding through a Transformer decoder in the following manner: obtaining noise data based on a type label; generating corresponding matching data according to the object query data; combining the noise data and the matching data to obtain the object query data;

[0215] Correspondingly, the matching attention mask may specifically include: a denoising mask area corresponding to the noise data, and a matching mask area corresponding to the matching data.

[0216] In some embodiments, the sample transformation model can be trained specifically in the following manner: according to a weak semi-supervised learning rule based on point annotation, a fully annotated sample image is acquired and utilized to generate first sample data; an initial transformation model is constructed; wherein the initial transformation model includes at least an initial point amplification module; and the initial transformation model is trained using the first sample data to obtain a sample transformation model that meets the requirements.

[0217] It should be noted that the units, devices or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by functions divided into various modules. Of course, when implementing this specification, the functions of each module can be implemented in the same or more software and / or hardware, or the modules that implement the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0218] It can be seen from the above that the target detection device provided by the embodiment of the present specification, through the weak semi-supervised learning rules based on point annotation, adopts a sample transformation model that at least includes a point amplification module, and only needs to obtain and use a small number of fully annotated sample images, and mainly relies on and uses point-annotated sample images with low annotation costs. It can quickly train to obtain a target detection model with higher accuracy and better effect, thereby effectively reducing the overall processing cost, and using the target detection model trained based on the above method to efficiently and accurately realize target detection for image data.

[0219] In a specific scenario example, the relevant ideas of the target detection method provided in this specification can be applied to implement specific weak semi-supervised target detection.

[0220] In this scenario example, Superior Point-DETR, namely SP-DETR, is proposed based on Point-DETR. A point amplification module is introduced in the model (for example, the sample transformation model) to predict multiple annotation points for the same object, realize the one-to-many relationship between objects and points, and reduce the difficulty of detection. In addition, the relative position and semantic alignment branches (for example, the relative position branch processing structure and the semantic alignment branch processing structure in the improved point encoder) are combined to further mine the annotation information and enhance the encoding ability of the point encoder. Point encoding is used to optimize the object query generation of the Decoder and the matching process of the cross-attention, accelerating the convergence of the model. Finally, in order to maintain the stability of Hungarian matching, an additional label denoising task is introduced to accelerate the convergence of the model and improve the efficiency of model training.

[0221] Specifically, a weak semi-supervised object detection method based on point annotation is proposed. A teacher model (e.g., a sample transformation model) that maps from points to bounding boxes is trained. In addition to the basic components of DETR, the model also includes a point amplification module, a point encoder, a decoder optimization module, and an additional label denoising task. The point amplification module amplifies the point annotation information to reduce the detection difficulty; the stronger point encoder encodes the point annotation information to generate a more guiding object query; the idea of ​​optimizing the decoder by combining the reference point encoding (using the Transformer decoder in the Transformer module) is used to optimize the generation of object queries and the matching of cross-attention, accelerating the convergence of the Transformer; the denoising task is introduced to ensure the stability of the Hungarian matching algorithm and accelerate the convergence of the model. Furthermore, in this scenario example, considering the introduction of the point amplification module and the label denoising task, a special attention mask is designed to control the interaction of the object query.

[0222] For specific implementation, see Fig.12 As shown in the figure, a weak semi-supervised target detection method based on point annotation is proposed, which is a weak semi-supervised target detection method based on DETR. The whole process involves a teacher model and a student model (e.g., a target detection model).

[0223] Among them, the teacher model includes: a feature extraction module (e.g., a feature extraction network), a point amplification module, a point encoder, a Transformer module, a prediction head module and a Hungarian matching module (e.g., a matching loss module).

[0224] Among them, the teacher model is specifically used to train the point-to-bounding box encoder based on the fully labeled data to generate complete annotations for most of the point labeled data.

[0225] The student model is trained using fully labeled data and generated pseudo-labeled data (to obtain the final target detection model). The student model selects the FCOS model by default.

[0226] Among them, the feature extraction module of the teacher model uses the ResNet50 backbone network to extract the features of the input image for subsequent calculations; the point amplification module of the teacher model uses the high-dimensional features of the point annotations to predict the coordinates of the surrounding points, realize the amplification of the point annotations, reduce the difficulty of detection, and introduce the attention mask to control the interaction between multiple object queries of the same object; the point encoder of the teacher model uses four calculation branches to encode the point annotations into 256-dimensional point codes as the object query input of the subsequent Transformer Decoder; the Transformer calculation module of the teacher model includes an encoder and a decoder. The encoding stage encodes the input image features, and the decoding stage uses the object query (a set of learnable position codes in the Transformer) to learn from the encoded features, adopts the idea of ​​optimizing the Decoder with reference point encoding, and uses point encoding to optimize the Decoder's object query generation and its cross-attention matching process; the prediction head module of the teacher model uses a multi-layer perceptron MLP to calculate the object The query predicts the relative position of the bounding box; the Hungarian matching module of the teacher model matches the predicted set with the given real set and distributes them according to the principle of minimum cost, and the cost rule is determined by a set of loss functions. Introducing an additional label denoising task To maintain the stability of the Hungarian algorithm, the noisy label is also used as the object query, and the object query is divided into the matching part and the denoising part. For this purpose, new constraints are added based on the attention mask introduced in the point amplification module.

[0227] When the above teacher model is implemented, it can be constructed and trained according to the following steps.

[0228] Step 1: Select coordinate points from the complete annotation and generate point annotations (obtain the first sample data).

[0229] In this scenario example, in order to use a small amount of fully annotated data to train the encoding from point to bounding box, we first randomly select a coordinate point from the fully annotated bounding box, and calculate the up, down, left, and right position offset of the point relative to the bounding box as the target result to be predicted, and generate point annotation information in combination with category information.

[0230] Step 2: Use the resnet50 backbone network to extract features from the input image.

[0231] In this scenario example, the ResNet50 used has a total of 50 layers, which can be divided into five parts according to the feature scale. Each part uses a different size and number of convolution kernels, and outputs feature maps of different scales and numbers of channels, respectively, to obtain C2, C3, C4, and C5. SP-DETR is based on the DETR model and uses the output of the last layer C5, i.e., 2048-dimensional features, for encoding and decoding.

[0232] Step 3: Use the point amplification module to realize the amplification of point annotations and the corresponding attention mask design.

[0233] Usually, Point-DETR mostly adopts a one-to-one strategy, that is, one object is labeled with one point, and one point label is obtained through a point encoder to obtain a point encoding as an object query. In Transformer, the object query is responsible for detecting the object that the point label is responsible for. Because of the one-to-one strategy, the size of the prediction set and the true set are equal, and each prediction result only calculates the loss with the true result of the corresponding object. There is no need to perform set matching, and the Hungarian matching module is removed in Point-DETR. However, during the training process, the result of each prediction of the point label represents the best result of the corresponding object in this prediction. There is a problem that it is difficult to locate the entire object using only one point, and a longer training cycle is required. Secondly, it may also cause the learning of the model to be limited to the point label position, and the generalization performance of different point label positions is poor. Based on this, SP-DETR proposes a point amplification module, whose structure is as follows: Figure 5 shown.

[0234] In this scenario example, the implementation process of the point expansion module used is as follows: For each given annotated point, a bilinear interpolation sampling method is used to sample from the high-dimensional features input to the Transformer, that is, the feature map with 2048 channels in the last layer extracted by the backbone network (SP-DETR is based on DETR, and its input Transformer features are single-scale features), and the high-dimensional features corresponding to the coordinates are obtained, and then the features are input to the designed position offset prediction branch. Through the prediction branch, the position offsets of the four points (for example, auxiliary points) around the original point coordinates are predicted, and then the actual coordinates of the predicted points are obtained based on the coordinates of the original point. These five points all represent points responsible for the same object, changing the original one-to-one strategy to a one-to-many strategy.

[0235] For the newly generated point coordinates, it is necessary to add corresponding annotation information for training, and its category is consistent with the category information of the original point. For the bounding box annotation information, because the bounding box prediction of SP-DETR uses the upper, lower, left, and right position offsets of the current point coordinates relative to its original true bounding box, the position of the original true bounding box can be calculated based on the bounding box information of the original point, and then the relative position of each newly generated point relative to the true bounding box is calculated as its bounding box annotation information. This embodiment only adopts this strategy in the training phase, because the verification phase lacks information for updating.

[0236] Because of the one-to-many strategy, the Hungarian matching component is still used in SP-DETR, and the bounding box loss is used as its cost loss calculation. It is responsible for the optimal matching between multiple point annotation prediction results of the same object, reducing the learning difficulty and improving the robustness of the model to the point annotation position learning. In addition, it is necessary to introduce the attention mask. The attention mask is a matrix used by the self-attention module of the Decoder in the Transformer to control the interaction of object queries, and its size is the number of object queries. For the object query input to the Transformer Decoder, because it is a one-to-many (5) strategy, every 5 object queries are responsible for detecting the same object. They need to interact with each other and do not need to interact with the object queries responsible for other objects. The corresponding row and column values ​​in each group in the Attentionmask are 0, indicating interaction, and the row and column values ​​between different groups are negative infinity, indicating no interaction. The attention mask designed in this paper: A 1 The graphical results are as follows Fig.10 As shown, for intuitive understanding, the 1 and 0 in the figure represent 0 and negative infinity in the actual encoding respectively.

[0237] Step 4: The point encoder encodes the point annotation information using four computation branches.

[0238] In this scenario example, the point encoder is the core structure of the entire model, which realizes the mapping from point annotation to bounding box. In addition to the absolute position branch (e.g., the absolute position branch processing structure) and the randomly initialized learnable category encoding (e.g., the category index branch processing structure), the point encoder in SP-DETR adds a relative position encoding branch (e.g., the relative position branch processing structure) and a feature semantic alignment branch (e.g., the semantic alignment branch processing structure). The overall structure of the point encoder is shown in Figure 6 shown.

[0239] Among them, based on the relative position encoding branch, how to make full use of the limited annotation information is the starting point for the design of semi-supervised and weakly supervised methods. In addition to its absolute coordinates and category information, a given point also has its relative position coordinates based on the image. Its relative position is represented by its offset relative to the four boundaries of the image. For the object to be detected, its relative position to the image is also an important factor to assist its detection.

[0240] To this end, a relative position calculation branch is designed to convert the (top, down, left, right) vector calculated for each point into (top / down, left / right), and then use sin encoding to encode the code representing the relative position of the point. This code is added to the previous code as the subsequent object query (for example, object query data).

[0241] Regarding the feature semantic alignment branch, matching is a common concept in visual tasks, especially in contrastive tasks such as face recognition and object tracking. The core idea is to predict the similarity between inputs. Empirical results show that the Siamese-based architecture, which projects the two sides to be matched into the same encoding space, performs well on tasks involving matching. Based on this, the cross-attention of DETR can be interpreted as a "matching and feature refinement" process. In order to achieve fast convergence, it is necessary to ensure the alignment semantics between the object query and the image feature encoding, that is, to project them both into the same encoding space. The object query encoding is randomly projected into the encoding space at initialization, and a very long training cycle is required to learn a meaningful match with the image features. To this end, a feature semantic alignment branch is added. The high-dimensional features of the corresponding positions of the point annotations are extracted by the bilinear interpolation sampling method. The features are linearly mapped to obtain an encoding consistent with the existing encoding dimension of the point encoder, and then multiplied with the point encoding to obtain a new point encoding to achieve semantic alignment between the features and the object query.

[0242] Step 5: Optimize object query and decoder matching based on point encoding.

[0243] In this scenario example, the Transformer (e.g., the Transformer module) includes an Encoder (Transformer encoder) and a Decoder (Transformer decoder). The output of the Encoder is a multi-scale feature with the same resolution as the input, and its key and query are both pixels from the multi-scale feature. The Decoder consists of a set of Decoder layers (e.g., encoding layers), and each Decoder layer consists of three main parts: (1) self-attention layer, which is used to remove duplicate predictions and for interaction between object queries. (2) cross-attention layer, which aggregates the embedding output from the Encoder to refine the object query and further improve the prediction of categories and bounding boxes. (3) FFN layer (e.g., feedforward neural network layer).

[0244] Specifically, the bounding box predicted based on the object query can be formulated as shown in Formula 1:

[0245] box=sigmoid(FFN(f)+[s T 0 0] T ) (1)

[0246] Where f represents the object query; the bounding box can be a four-dimensional vector represented as [b cx ,b cy ,b w ,b h ] T , respectively represent the center coordinates of the bounding box, the width and height of the bounding box; Sigmoid is used to normalize the bounding box within the range of [0,1]; FFN is used to predict the non-normalized bounding box; s is the coordinate of the non-normalized reference point, which is (0,0) in DETR and obtained through feature prediction in SP-DETR.

[0247] In specific implementation, the matching process of DETR's decoder can be described as follows: Decoder's cross-attention includes three inputs, queries, keys, and values. Among them, the key is a content key c k (output from Encoder) and spatial key p kAdd together, Value and content key are also embeddings output from Encoder. In DETR, each query is the content query c output by self-attention. q and spatialquery p q (object query o q ) are added together.

[0248] The relevant attention weight is calculated based on the dot product of query and key as shown in formula (2):

[0249] (c q +P q )T(c k +p k )

[0250] =c q T c k +c q T P k +p q T c k +p q T p k

[0251] =c q T c k +c q T p k +o q T c k +o q T p k (2)

[0252] SP-DETR is different from DETR. It does not use reference point encoding, but has point encoding obtained by point encoder encoding. The idea of ​​reference point is to accurately determine the position of reference point by continuously adjusting the prediction parameters. The point annotation in this paper gives accurate "reference" information at the beginning, which can better adapt to the optimization scenario. Fig.11 As shown, there is no need to go through the learnable encoding to two-dimensional coordinates and then to coordinate encoding, and directly use the point encoding as p s, T is also generated through Decoder embedding using FFN prediction, but different from Conditional-DETR, which initializes the Decoder embedding as a full-zero vector consistent with the shape of the object query, this paper initializes it as a point encoding, which can reduce its learning difficulty and introduce guidance information through point annotation.

[0253] Step 6: To ensure the stability of Hungarian algorithm matching, an additional label denoising task is introduced.

[0254] In this scenario, the slow convergence of DETR is also caused by the instability of Hungarian matching. Due to the nature of random optimization, this component is unstable in the early stages of training. For the same image, queries are often matched with different objects in different batches, making the optimization blurred and unstable. To this end, the denoising task is introduced to reduce the instability of Hungarian matching.

[0255] In DN-DETR, in addition to the Hungarian loss, the noisy ground-truth bounding box is usually input into the TransformerDecoder, and the model is trained to reconstruct the ground-truth bounding box, which effectively reduces the difficulty of bipartite graph matching and achieves faster convergence. Since the noisy bounding box does not need to undergo bipartite graph matching, the denoising task can be regarded as a simpler auxiliary task, which helps DETR alleviate unstable discrete bipartite matching and learn bounding box prediction faster. At the same time, the denoising task also helps to reduce the difficulty of optimization because the added random noise is small. To maximize the potential of the auxiliary task, the query of the Decoder is regarded as a bounding box plus class label plus noise encoding, and the bounding box and label denoising are performed at the same time. The Decoder query consists of two parts: the matching part, which is processed in the same way as DETR, using bipartite graph matching, and learning to predict the ground-truth bounding box label pair with the matching Decoder output, and the denoising part, whose input is the noisy ground-truth bounding box and label pair, and the output of the denoising part aims to reconstruct the real object. For label noise, label flipping is used to randomly flip some labels to other labels. For bounding box noise, center shift (adding noise to the bounding box center coordinates) and bounding box scaling (the width and height of the bounding box are randomly sampled from [(1-λ2)w, (1+λ2)w] and [(1-λ2)h, (1+λ2)h], respectively, λ2∈(0,1)) are used.

[0256] SP-DETR is based on the DETR model, which is different from DN-DETR which is based on the DAB-DETR model. DN-DETR uses a four-dimensional learnable code (x, y, w, h) to initialize the object query as a learnable bounding box code. The output of each decoder layer contains (Δx, Δy, Δw, Δh), which can be updated to (x+Δx, y+Δy, w+Δw, h+Δh) to participate in the calculation of the next layer. The decoder embedding is randomly initialized. When adding noise, the decoder embedding is concatenated with the noisy label, and the object query is concatenated with the noisy bounding box to achieve subsequent calculations.

[0257] SP-DETR uses the point encoding of the point encoder as the object query, without using the bounding box encoding. The encoding semantics of the point encoding cannot be used as the bounding box encoding. Therefore, when introducing the denoising task in SP-DETR, only the label denoising task is introduced. Since the point annotation information contains category information, the loss function of the SP-DETR matching part does not include the label loss. Introducing the label denoising task in SP-DETR helps to improve the accuracy of bounding box prediction. In implementation, since the decoder embedding is also initialized as the point encoding in the previous step, the new decoder embedding is obtained by concatenating the point encoding with the noisy label, and the noisy method is also achieved by flipping the label. For the object query also obtained by point encoding, the calculation of the Transformer requires that the vector shape of the decoder embedding and the object query are consistent. Without adding the noisy bounding box, the existing object query is expanded according to the shape and size of the decoder embedding, and the all-0 vector is expanded. The expansion of the all-0 vector will not affect the label noisy part. Correspondingly, the auxiliary loss is only the label loss.

[0258] Due to the addition of the denoising task, the decoder object query includes a matching part (e.g., matching data) and a denoising part (noise data), see Fig.12As shown, the attention mask input to the Decoder needs to be adjusted. The point amplification module has set the attention mask, but the previous mask only considered the matching part, which is responsible for controlling the interaction between each group of object queries for the same object through the mask. In order to avoid information leakage, it is necessary to add another dimension of control on its basis, that is, to divide the attention mask into the matching part and the denoising part. For the matching part, it is still divided according to the constraints of the point amplification module, and the object query of the denoising part is visible to the matching part, because the denoising part only contains label information, and the matching part also contains label information, which will not cause information leakage. The matching part is invisible to the denoising part, and the real label information contained in the matching part will interfere with the prediction of the denoising part. For the denoising part, because it contains several versions of denoising groups, the denoising groups do not interact with each other. Newly added mask control A 2 As shown in formula (3), the final mask A 3 Visualization Reference Fig.13 As shown, for intuitive understanding, the 1 and 0 in the figure represent 0 and negative infinity in formula (3) respectively.

[0259]

[0260] Among them, A 2 i,j Represents object query q i Is it consistent with object query q j For interaction, formula (3) only represents the newly added mask control and does not involve the design of the matching part; P represents the number of noise-added groups; M represents the number of group noise-added labels; for the entire mask, the first P×M rows and columns represent the denoising part, and the remaining rows and columns are the matching part; negative infinity means that interaction is not possible; 0 means that interaction is possible.

[0261] Step 7: Setting and calculating the loss function.

[0262] In this scenario, due to the addition of an additional denoising task during training, the object query of the decoder includes both the denoising part and the matching part, and the overall loss L SP-DETR The definition is shown in formula (4). The loss of the matching part L match (For example, the matching loss function) is defined as shown in formula (5). Since the object query contains category information, only the bounding box loss needs to be calculated. The bounding box loss L box Same as DETR definition. For bounding box prediction use Calculate, where is (x,y,x,y), (x,y) is the coordinate of the point, is the relative offset. The denoising part loss is defined as L denose (For example, denoising loss function), which only contains label calculation loss, and Focal Loss is used in specific implementation.

[0263] Overall loss L SP-DETR (including the matching part loss and the denoising part loss) is expressed as:

[0264] L SP-DETR =L match +L denose (4)

[0265] Among them, the matching part loss is expressed as:

[0266]

[0267] Among them, L box Based on L1 loss and GIoU loss, as shown in formula (6):

[0268]

[0269] Among them, λ iou Indicates that the value is 2 in the experiment; Indicates that the value is 5 in the experiment; L iou As shown in formula (7):

[0270]

[0271] Among them, |.| represents the area, and the intersection and union of the predicted box coordinates and the true box coordinates are used to represent the result box itself. The area of ​​the intersection or union is expressed by b σ(i) and The minimum / maximum value of the linear function is used to calculate, so that the loss function is smooth enough to facilitate optimization using the stochastic gradient descent algorithm; means containing b σ(i) and The largest box of (the area of ​​B is also calculated by the min / max values ​​of a linear function of the box coordinates).

[0272] The expression of Focal loss is shown in formula (8):

[0273]

[0274] Among them, α t represents the balance factor; γ represents the adjustment factor; α t It is a hyperparameter. Usually, the more categories there are, the more t The smaller it is, the smaller it is, and it is set to 0.25 in the experiment; γ is used to adjust the proportion of samples with different confidence levels in the total loss, and it is set to 2 in the experiment.

[0275] After the teacher model is trained in the above manner, the teacher model can be used to process the point annotation data (for example, the point annotation sample image) to obtain the corresponding pseudo-annotation data (for example, the processed point annotation sample image with the pseudo-annotation box), and then the fully annotated data and the pseudo-annotated data generated based on the teacher model can be used to jointly train the FCOS model to obtain the corresponding student model.

[0276] The final performance of the obtained student model can reflect the advantages of the teacher model established based on this method. If the performance of the student model meets the requirements, the student model can be used for specific target detection in the future.

[0277] It should be added that in this scenario example, SP-DETR mainly follows most of the training settings in DETR, but there are still many differences in the settings: in this scenario example, the model is trained for 108 epochs, and the batchsize set by the GPU is 2; the number of noise groups in the denoising task is set to P = 5, and the number of noise labels in each group M is the sum of the number of labels of all images in the batch. In addition, to ensure the stability of training, in this scenario example, a warm-up training scheme is used in the first training batch, and the learning rate is reduced by 10 times at the 72nd epoch and the 96th epoch respectively.

[0278] In addition, based on the method provided in this manual, Python 3.6.13 was used for coding in the specific implementation, and the model was trained and verified in the Pytorch deep learning framework. All experiments were run on the Ubuntu 20.04.4 operating system, NVIDIA GeForce GTX1080Ti GPU, and 11G video memory hardware device.

[0279] Through the above scenario examples, it is verified that the target detection method provided in this specification takes into account that the prediction difficulty is relatively low because the point annotation is encoded as an object query in Point-DETR to participate in the DETR training process. In addition, since the feature information is lost after the point annotation scheme is adapted for FCOS and Faster R-CNN, the prediction difficulty is relatively high. Therefore, the introduced SP-DETR adds a point amplification module, optimizes the point encoder, optimizes the matching process of the Decoder cross-attention, and adds a denoising task on its basis to improve the overall performance of the model. In addition, Group RCNN projects the labeled points to each layer of the feature map of FPN, and selects several points in the feature map near each projection point to generate proposals. Its function is similar to the point amplification module of SP-DETR, except that it is based on the multi-scale features of R-CNN; its Instance-aware Parameter Generation module is similar to the point encoder of SP-DETR, which operates on a group of point features representing the same object and then splices them with the category code. SP-DETR's point encoder is more complex and can fully mine point annotation information. It also optimizes the decoder matching process and designs denoising tasks based on point encoding, giving it a stronger point-to-bounding box mapping capability.

[0280] Although the present specification provides method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps, and does not represent a unique execution order. When the device or client product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, product or device. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or device including the elements. The first, second, etc. words are used to represent the name, and do not represent any particular order.

[0281] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0282] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer-readable storage media including storage devices.

[0283] Through the description of the above embodiments, it can be known that those skilled in the art can clearly understand that the present specification can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present specification can essentially be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in each embodiment of the present specification or some parts of the embodiments.

[0284] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. This specification can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0285] Although the present specification is described through embodiments, those skilled in the art will appreciate that there are many modifications and changes to the present specification without departing from the spirit of the present specification, and it is intended that the appended claims include these modifications and changes without departing from the spirit of the present specification.

Claims

1. A target detection method, characterized in that: include: Get the target image; The target image is processed by a target detection model to obtain a corresponding target processing result; wherein the target detection model is obtained by weak semi-supervised learning training in advance using a point-annotated sample image, a fully-annotated sample image, and a sample transformation model according to a weak semi-supervised learning rule based on point annotation; the sample transformation model at least includes: a point amplification module; Determine whether there is a target object in the target image according to the target processing result; and if it is determined that there is a target object in the target image, determine the position information and object type of the target object in the target image; The target detection model is trained in the following manner: using the sample transformation model to process the point-annotated sample image to transform the point-annotated sample image into a processed point-annotated sample image containing a pseudo-annotation frame; using the processed point-annotated sample image and the fully-annotated sample image to train the target detection model through weak semi-supervised learning; The sample transformation model also includes: a feature extraction network, a Transformer module, a point encoder, and a prediction head module; the feature extraction network is connected to the Transformer module; the point amplification module is also connected to the point encoder; the point encoder is also connected to the Transformer module; the Transformer module is also connected to the prediction head module; the point amplification module is used to receive the high-dimensional features of the point-annotated sample image output by the feature extraction network; and in the high-dimensional features of the point-annotated sample image, high-dimensional feature information at the annotated point is obtained through bilinear interpolation sampling; based on the high-dimensional feature information at the annotated point, four auxiliary points associated with the annotated point and located in four different directions are determined through position offset prediction; and the type labels carried by the annotated point are respectively set on the four auxiliary points to obtain the amplified pseudo-annotated points.

2. The method according to claim 1, characterized in that Before acquiring the target image, the method further includes: According to the weak semi-supervised learning rule based on point annotation, a point-annotated sample image and a fully-annotated sample image are obtained; wherein the number of the point-annotated sample images is greater than the number of the fully-annotated sample images; the fully-annotated sample image at least includes a labeling box for selecting a sample object, and a type label for the sample object carried by the labeling box; the point-annotated sample image at least includes a labeling point for indicating the sample object, and a type label for the sample object carried by the labeling point; Processing the point-annotated sample image using a sample transformation model to obtain a processed point-annotated sample image carrying a pseudo-annotation frame; Combining the fully annotated sample images and the processed point-annotated sample images to obtain a training sample set; The training sample set is used to train the initial detection model to obtain a target detection model that meets the requirements.

3. The method according to claim 2, characterized in that The prediction head module is used to determine and mark a pseudo-annotation box for selecting a sample object in the point-annotated sample image of the input sample transformation model based on the intermediate result data output by the Transformer module, so as to output the corresponding processed point-annotated sample image carrying the pseudo-annotation box.

4. The method according to claim 3, characterized in that: The feature extraction network is used to extract features from the point-annotated sample image of the input sample transformation model to obtain a corresponding sample feature map; and send the sample feature map to the Transformer module.

5. The method according to claim 3, characterized in that: The point encoder comprises: an improved point encoder; wherein the improved point encoder comprises at least: an absolute position branch processing structure, a relative position branch processing structure, a category index branch processing structure, and a semantic alignment branch processing structure; The point encoder is used to obtain and output corresponding joint codes by processing the annotation points in the point annotation sample image and the amplified pseudo annotation points.

6. The method according to claim 5, characterized in that The point encoder obtains and outputs the corresponding joint code by processing the annotation points in the point annotation sample image and the amplified pseudo annotation points in the following manner: Through the absolute position branch processing structure, according to the position coordinates of the annotation point and the position coordinates of the amplified pseudo annotation point, preset encoding processing is performed respectively to obtain the corresponding absolute position code; By using the relative position branch processing structure, according to the position coordinates of the marking point and the position coordinates of the pseudo-marking point after expansion, determine and generate a corresponding relative position vector according to the offset distance of the marking point and the pseudo-marking point after expansion relative to the upper boundary, the lower boundary, the left boundary and the right boundary of the sample image marked by the marking point; convert the relative position vector into corresponding relative ratio data; and perform a preset encoding process on the relative ratio data to obtain a corresponding relative position code; The type label of the annotation point and the type label of the amplified pseudo annotation point are processed by the category index branch processing structure to obtain a corresponding category index code; Through the semantic alignment branch processing structure, linear mapping is performed on the acquired high-dimensional features of the annotation points and the high-dimensional features of the amplified pseudo-annotation points to obtain corresponding image semantic feature codes; wherein the image semantic feature codes, absolute position codes, relative position codes, and category index codes have the same dimensions; The absolute position coding, relative position coding, category index coding, and image semantic feature coding are used in combination to obtain a corresponding joint coding.

7. The method according to claim 6, characterized in that The absolute position coding, relative position coding, category index coding, and image semantic feature coding are used in combination to obtain corresponding joint coding, including: Performing an addition operation using the absolute position code, the relative position code and the category index code to obtain an intermediate code; The intermediate code and the characteristic code are used to perform a multiplication operation to obtain the joint code.

8. The method according to claim 6, characterized in that The Transformer module includes: a Transformer encoder and a Transformer decoder; wherein the Transformer encoder is connected to the feature extraction network and the Transformer decoder respectively; the Transformer decoder is also connected to the point encoder and the prediction head module; The Transformer module is used to generate and output corresponding intermediate result data according to the input sample feature map, position coding, and joint coding.

9. The method according to claim 8, characterized in that The Transformer module generates and outputs the corresponding intermediate result data according to the input sample feature map, position encoding, and joint encoding in the following manner: Processing the combination of the sample feature map and the position code through a Transformer encoder to generate corresponding embedded data; and sending the embedded data to a Transformer decoder; Get the matching attention mask through the Transformer decoder; The joint encoding is used to generate corresponding object query data through a Transformer decoder; and the attention mask, object query data and embedded data are used to generate and output corresponding intermediate result data.

10. The method according to claim 9, characterized in that The Transformer decoder comprises: a decoding layer; wherein the decoding layer comprises: a self-attention layer, a cross-attention layer, and a feedforward neural network layer; The self-attention layer is used to process the object query data using an attention mask to strengthen the interaction between the object query data indicating the same sample object; The cross attention layer is used to refine the processed object query data output from the self-attention layer by using the embedded data to obtain refined object query data; The feedforward neural network layer is used to predict the bounding box containing the sample object in the point-annotated sample image according to the refined object query data, and obtain corresponding intermediate result data.

11. The method according to claim 9, characterized in that The sample transformation model is also connected to a matching loss module; wherein the matching loss module is configured with a matching loss function and a denoising loss function.

12. The method according to claim 11, characterized in that The joint encoding is used to generate corresponding object query data through a Transformer decoder, including: Get noisy data based on type labels; Generate corresponding matching data according to the object query data; combining the noise data and the matching data to obtain the object query data; Correspondingly, the matched attention mask includes: a denoising mask area corresponding to the noise data, and a matching mask area corresponding to the matching data.

13. The method according to claim 2, characterized in that The sample transformation model is trained in the following way: According to a weak semi-supervised learning rule based on point annotation, a fully annotated sample image is acquired and used to generate first sample data; Constructing an initial transformation model; wherein the initial transformation model at least includes an initial point amplification module; The initial transformation model is trained using the first sample data to obtain a sample transformation model that meets the requirements.

14. A target detection device, characterized in that: include: An acquisition module, used for acquiring a target image; A processing module, used to process the target image using a target detection model to obtain a corresponding target processing result; wherein the target detection model is obtained by weak semi-supervised learning training in advance using a point-annotated sample image, a fully-annotated sample image, and a sample transformation model according to a weak semi-supervised learning rule based on point annotation; the sample transformation model at least includes: a point amplification module; A determination module, used to determine whether there is a target object in the target image according to the target processing result; and when it is determined that there is a target object in the target image, determine the position information and object type of the target object in the target image; The target detection model is trained in the following manner: using the sample transformation model to process the point-annotated sample image to transform the point-annotated sample image into a processed point-annotated sample image containing a pseudo-annotation frame; using the processed point-annotated sample image and the fully-annotated sample image to train the target detection model through weak semi-supervised learning; The sample transformation model also includes: a feature extraction network, a Transformer module, a point encoder, and a prediction head module; the feature extraction network is connected to the Transformer module; the point amplification module is also connected to the point encoder; the point encoder is also connected to the Transformer module; the Transformer module is also connected to the prediction head module; the point amplification module is used to receive the high-dimensional features of the point-annotated sample image output by the feature extraction network; and in the high-dimensional features of the point-annotated sample image, high-dimensional feature information at the annotated point is obtained through bilinear interpolation sampling; based on the high-dimensional feature information at the annotated point, four auxiliary points associated with the annotated point and located in four different directions are determined through position offset prediction; and the type labels carried by the annotated point are respectively set on the four auxiliary points to obtain the amplified pseudo-annotated points.

15. A server, characterized in that: The method comprises a processor and a memory for storing processor-executable instructions, wherein the processor implements the steps of the method according to any one of claims 1 to 13 when executing the instructions.

Images