Training method of target detection model, target detection method and device

Abstract

The present disclosure provides a target detection model training method and target detection method and device, relates to the technical field of artificial intelligence, in particular to the technical field of deep learning, image processing and computer vision. The specific implementation scheme is: obtaining a sample image-text pair; wherein the sample image-text pair comprises a sample image and a sample text, and the sample text is a label of a target in the sample image; determining a sample image encoding feature of an unmasked image region in the sample image and a sequence feature of a masked image region in the sample image; determining a sample text encoding feature of an unmasked character in the sample text and a text vector of a masked character in the sample text; and pre-training a target detection model according to the sample image encoding feature, the sequence feature, the sample text encoding feature and the text vector. Through the above technical scheme, the accuracy of target detection can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of deep learning, image processing and computer vision. BACKGROUND

[0002] Currently, pre-training tasks are increasingly important in the field of artificial intelligence. Whether it is a natural language processing (NLP) task or a computer vision (CV) task, through the pre-training of massive data, a model with a large-scale parameter quantity can learn general knowledge and thus has good generalization ability. Since the target detection task is more complex than the picture classification task, it not only needs to detect and locate the target frame, but also needs to give the category of the target, resulting in that there is basically no pre-training task designed for the target detection task. Therefore, how to provide a pre-training method of a target detection model is crucial to effectively improve the accuracy of target detection. SUMMARY

[0003] The present disclosure provides a training method of a target detection model and a target detection method and device.

[0004] According to an aspect of the present disclosure, a pre-training method of a target detection model is provided, which comprises:

[0005] obtaining a sample image-text pair; the sample image-text pair comprises a sample image and a sample text, and the sample text is a label of a target in the sample image;

[0006] determining a sample image encoding feature of an unmasked image region in the sample image and a sequence feature of a masked image region in the sample image;

[0007] determining a sample text encoding feature of an unmasked character in the sample text and a text vector of a masked character in the sample text;

[0008] pre-training a target detection model according to the sample image encoding feature, the sequence feature, the sample text encoding feature and the text vector.

[0009] According to another aspect of the present disclosure, a training method of a target detection model is provided, which comprises:

[0010] serializing and converting a sample image to be detected to obtain an image sequence;

[0011] fine-tuning a target detection model according to the image sequence; wherein the target detection model is trained based on the pre-training method of the target detection model provided in any of the embodiments of the present disclosure.

[0012] According to another aspect of the present disclosure, a target detection method is provided, which comprises:

[0013] obtaining a target image to be detected;

[0014] detecting the target image to be detected by using a target detection model to obtain a target detection result of the target image to be detected; wherein the target detection model is trained according to the training method of the target detection model provided in any of the embodiments of the present disclosure.

[0015] According to another aspect of the present disclosure, a pre-training device of a target detection model is provided, which comprises:

[0016] a sample image-text pair acquisition module, configured to acquire a sample image-text pair; the sample image-text pair comprises a sample image and a sample text, and the sample text is a label of a target in the sample image;

[0017] a sample image feature determination module, configured to determine a sample image encoding feature of an unmasked image region in the sample image, and a sequence feature of a masked image region in the sample image;

[0018] a sample text feature determination module, configured to determine a sample text encoding feature of an unmasked character in the sample text, and a text vector of a masked character in the sample text;

[0019] a target detection model training module, configured to pre-train a target detection model according to the sample image encoding feature, the sequence feature, the sample text encoding feature and the text vector.

[0020] According to another aspect of the present disclosure, a training device of a target detection model is provided, which comprises:

[0021] an image sequence determination module, configured to perform sequence conversion on a sample image to be detected to obtain an image sequence;

[0022] a target detection model fine-tuning module, configured to fine-tune a target detection model according to the image sequence; wherein the target detection model is trained based on the pre-training method of the target detection model provided in any of the embodiments of the present disclosure.

[0023] According to another aspect of the present disclosure, a target detection device is provided, which comprises:

[0024] a target image to be detected acquisition module, configured to acquire a target image to be detected;

[0025] The target detection result determination module is configured to determine a target detection result of the target image to be detected by using a target detection model.

[0026] According to another aspect of the present disclosure, an electronic device is provided, which includes:

[0027] at least one processor; and

[0028] a memory connected to the at least one processor in communication; wherein

[0029] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the pre-training method, the training method, or the target detection method of the target detection model according to any one of the embodiments of the present disclosure.

[0030] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to perform the pre-training method, the training method, or the target detection method of the target detection model according to any one of the embodiments of the present disclosure.

[0031] According to another aspect of the present disclosure, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the pre-training method, the training method, or the target detection method of the target detection model according to any one of the embodiments of the present disclosure.

[0032] According to the technology of the present disclosure, the accuracy of target detection can be improved.

[0033] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0035] Figure 1 is a flowchart of a pre-training method of a target detection model according to an embodiment of the present disclosure;

[0036] Figure 2 is a flowchart of another pre-training method of a target detection model according to an embodiment of the present disclosure;

[0037] Figure 3is a flowchart of a target detection model training method according to an embodiment of the present disclosure;

[0038] Figure 4 is a flowchart of a target detection method according to an embodiment of the present disclosure;

[0039] Figure 5 is a structural schematic diagram of a target detection model pre-training device according to an embodiment of the present disclosure;

[0040] Figure 6 is a structural schematic diagram of a target detection model training device according to an embodiment of the present disclosure;

[0041] Figure 7 is a structural schematic diagram of a target detection device according to an embodiment of the present disclosure;

[0042] Figure 8 is a block diagram of an electronic device implementing a target detection model pre-training method, a target detection model training method, or a target detection method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are included to provide a thorough understanding of embodiments of the present disclosure by a person of ordinary skill in the art, and should not be construed as limiting the present disclosure to particular embodiments. Thus, it will be apparent to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in the following description, descriptions of well-known functions and constructions are omitted for clarity and conciseness.

[0044] It should be noted that the terms "target", "sample" and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product, or device.

[0045] In addition, it should also be noted that in the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of sample text pairs, sample images to be detected and target images to be detected and the like involved in the technical solutions comply with relevant legal regulations and do not violate public order and good customs.

[0046] In the field of NLP, the common pre-training methods include two basic paradigms: autoregressive (AR) pre-training and auto-encoding (AE) pre-training, among which the generative pre-training (GPT) model and the bidirectional encoder representation from transformers (BERT) model are representative models of the two paradigms. GPT autoregressive pre-training is suitable for text generation tasks, and the optimization goal is to model the joint probability of the sequence in one direction (from left to right or from right to left), and the words predicted later are conditioned on the words predicted earlier. While BERT is more suitable for text understanding tasks, because its modeling method is bidirectional, specifically by replacing some positions of tokens in the sequence with special mask characters, and then predicting these replaced characters (called MLM) in the target segment.

[0047] In the field of CV, the common pre-training methods include two categories: contrastive learning and masked auto-encoding, among which the moco series is a representative model of contrastive learning. The idea of this pre-training method is to enhance or weaken the input picture, so that the model learns that the enhanced picture is similar, while other pictures are not similar. While Masked Autoencoding is represented by Masked Autoencoders (MAE) and A Simple Framework for Masked Image Modeling (SimMIM). The idea of this model is to mask some areas of a picture, and then let the model predict the pixel values of the masked areas (called MIM).

[0048] The pre-training task in NLP has developed rapidly and maturely, which is due to the characteristics of language. Language itself is highly semantic and information-intensive. It is already a difficult task to remove a small number of words in a sentence and let the model predict the word itself. Therefore, the model can learn the common relationship between words through this simple masking method. This method is difficult to directly and simply copy to the field of CV.

[0049] In the field of CV, whether it is Contrastive Learning or Masked Autoencoding pre-training method, more information is used from the features of the picture itself, and the information from the text itself is ignored. In the target detection task, there are often multiple targets in a picture, and the targets are related to each other. For example, a shelf picture may contain potato chips, melon seeds, chocolates, spicy bars, and other goods. The picture of these goods itself has similarities.

[0050] Figure 1 A flowchart of a pre-training method of a target detection model is provided according to an embodiment of the present disclosure. The present example is applicable to the case of how to pre-train a target detection model in a target detection scenario. The method can be performed by a pre-training device of the target detection model, which can be implemented in software and / or hardware, and can be integrated into an electronic device that carries the pre-training function of the target detection model, such as a server. As shown in Figure 1 The pre-training method of the target detection model of the present embodiment can include:

[0051] S101, obtaining a sample image-text pair.

[0052] In the present embodiment, the sample image-text pair includes a sample image and a sample text, and the sample text is the label of the target in the sample image. The sample image refers to an image containing at least one target, wherein the target can be a stationary object or a moving object, etc. The sample text refers to a text composed of the label of the target in the sample image.

[0053] Specifically, a large amount of data with text labels can be collected, and then data with only images without text labels and data with only text labels without images can be filtered out, and images with text labels can be selected to construct a sample image-text pair.

[0054] An optional way is that for each image with a text label, the image and the text label corresponding to the image are taken as a sample image-text pair. Further, if the image contains multiple different text labels, the multiple text labels are spliced to form a text sequence, for example, the different text labels can be separated by a comma and spliced, and then the image and the text sequence corresponding to the image are taken as a sample image-text pair. For example, there are potato chips, melon seeds, chocolates, and spicy bars in a certain image, and the text labels of the image are "potato chips", "melon seeds", "chocolates", and "spicy bars". The text labels are separated by a comma and spliced to obtain a text sequence "potato chips, melon seeds, chocolates, spicy bars", and the image and the text sequence "potato chips, melon seeds, chocolates, spicy bars" of the image are taken as a sample image-text pair.

[0055] S102, determine a sample image coding feature of an unmasked image region in the sample image and a sequence feature of a masked image region in the sample image.

[0056] In this embodiment, the unmasked image region refers to an image region obtained by randomly masking image blocks in the sample image. The masked image region refers to an image region composed of image blocks remaining after removing the unmasked region in the sample image. The sample image coding feature refers to a feature obtained by feature extraction on the unmasked image region in the sample image, which can be represented in the form of a vector or a rectangle. The sequence feature refers to a feature obtained by sequencing the masked image region, which can be represented in the form of a vector or a matrix.

[0057] Specifically, a part of the region in the sample image can be randomly selected as the unmasked image region, and the remaining region can be selected as the masked region. Then, the unmasked image region is coded to obtain the sample image coding feature of the unmasked image region, and the image region in the sample image corresponding to the masked image region is sequenced to obtain the sequence feature of the masked image region.

[0058] S103, determine a sample text coding feature of an unmasked character in the sample text and a text vector of a masked character in the sample text.

[0059] In this embodiment, the unmasked character refers to a character obtained by randomly masking characters in the sample text. The masked character refers to a character remaining after removing the unmasked character in the sample text. The sample text coding feature refers to a feature obtained by feature extraction on the unmasked character in the sample text, which can be represented in the form of a vector or a rectangle. The text vector refers to a feature obtained by vectorizing the masked character, which can be represented in the form of a vector or a matrix.

[0060] Specifically, a part of the characters in the sample text can be randomly selected as the unmasked character, and the remaining characters can be selected as the masked character. Then, the unmasked character is coded to obtain the sample text coding feature of the unmasked character, and the characters in the sample text corresponding to the masked character can be vectorized based on the word embedding technology to obtain the text vector of the masked character.

[0061] S104, pre-train the target detection model according to the sample image coding feature, the sequence feature, the sample text coding feature and the text vector.

[0062] Optionally, the sample image encoding features, sequence features, sample text encoding features, and text vectors can be fused based on a feature fusion rule to obtain fused features. For example, the sample image encoding features, sequence features, sample text encoding features, and text vectors can be spliced, and the spliced features can be used as the fused features. Then, the fused features can be used to pre-train the target detection model. Specifically, the fused features can be input into the target detection model, and the model can be used to process the fused features to obtain predicted target categories in the sample image. Then, based on a pre-set loss function, the training loss can be calculated according to the predicted target categories and the sample text. The target detection model can be pre-trained according to the training loss until the training loss is stable within a set range or the number of training reaches a set number, and then the pre-training is stopped. The set range and the set number can be set by a person skilled in the art according to actual needs.

[0063] The technical solution provided by the embodiments of the present disclosure can obtain a sample image-text pair, then determine sample image encoding features of an unmasked image region in the sample image and sequence features of a masked image region in the sample image, further determine sample text encoding features of an unmasked character in the sample text and a text vector of a masked character in the sample text, and finally pre-train a target detection model according to the sample image encoding features, the sequence features, the sample text encoding features, and the text vector. The above technical solution introduces text features, pre-trains the target detection model in combination with image features and text features, can learn more general and effective information, obtains a target detection model with better performance, and thus improves the accuracy of target detection.

[0064] On the basis of the above embodiments, as an optional mode of the present disclosure, the sample image encoding features of the unmasked region in the sample image can be determined by determining an unmasked image region from the sample image, and performing feature extraction on the unmasked region to obtain the sample image encoding features of the unmasked image region.

[0065] Specifically, the sample image can be divided into at least two image regions, then a first set proportion of image regions are randomly selected from the at least two image regions as unmasked image regions, and then feature extraction is performed on the unmasked image regions to obtain the sample image encoding features of the unmasked image regions. For example, the unmasked image regions can be encoded based on an image encoding network to obtain the sample image encoding features of the unmasked image regions. The first set proportion can be set by a person skilled in the art according to actual needs, for example, 70%. The image encoding network can be a network based on deep learning, such as Resnet, Transformer, Swim, etc.

[0066] It can be understood that the extraction manner of the sample image coding feature can more flexibly and reasonably extract the coding features of the unmasked image region in the sample image, thereby laying a foundation for subsequent pre-training of the target detection model.

[0067] On the basis of the above-mentioned embodiments, as an optional manner of the present disclosure, the sample text coding features of the unmasked characters in the sample text are determined, comprising: determining the unmasked characters from the sample text; and performing feature extraction on the unmasked characters to obtain the sample text coding features of the unmasked characters.

[0068] Specifically, a second set proportion of characters are randomly selected from the sample text as unmasked characters; and then the unmasked characters are subjected to feature extraction to obtain the sample text coding features of the unmasked characters. For example, the unmasked characters can be subjected to feature coding based on a text coding network to obtain the sample text coding features of the unmasked characters. The second set proportion can be set by a person skilled in the art according to actual needs, for example, 15%. The image coding network can be a deep learning-based network, such as a TextRcnn, Bert, or other network model.

[0069] It can be understood that the extraction manner of the sample text coding feature can more flexibly and reasonably extract the coding features of the unmasked characters in the sample text, thereby laying a foundation for subsequent pre-training of the target detection model.

[0070] Figure 2 is a flowchart of another pre-training method of a target detection model according to an embodiment of the present disclosure. The present embodiment further optimizes the “pre-training of the target detection model according to the sample image coding features, the sequence features, the sample text coding features, and the text vector” on the basis of the above-mentioned embodiments, and provides an optional manner. As shown in Figure 2 The pre-training method of the target detection model of the present embodiment can comprise:

[0071] S201, obtaining a sample image-text pair.

[0072] The sample image-text pair comprises a sample image and a sample text, and the sample text is a label of a target in the sample image.

[0073] S202, determining sample image coding features of unmasked image regions in the sample image and sequence features of masked image regions in the sample image.

[0074] S203, determining sample text coding features of unmasked characters in the sample text and text vectors of masked characters in the sample text.

[0075] S204, determining a contrast loss according to the sample image coding features and the sample text coding features.

[0076]

[0076] In this embodiment, the contrast loss refers to a loss determined based on the image encoding feature and the text encoding feature.

[0077] Specifically, the contrast loss can be determined based on a preset contrast loss determination manner according to the sample image encoding feature and the sample text encoding feature. For example, the cosine similarity between the sample image encoding feature and the sample text encoding feature can be determined, and the cosine similarity is taken as the contrast loss of the sample image-text pair. For another example, the contrast loss can be determined based on metric learning according to the sample image encoding feature and the sample text encoding feature.

[0078] S205, determining a pixel reconstruction loss according to the sample image encoding feature and the sequence feature.

[0079] In this embodiment, the pixel reconstruction loss refers to a loss calculated based on the pixel feature of the covered region in the sample image after reconstruction.

[0080] An optional manner can be based on an image decoding network to determine the pixel feature of the covered image region in the sample image according to the sample image encoding feature of the un-covered image region in the sample image; and then based on a preset loss function, the pixel reconstruction loss is determined according to the pixel feature of the covered image region in the sample image and the sequence feature, for example, the pixel reconstruction loss can be determined based on the mean square error loss function according to the pixel feature of the covered image region in the sample image and the sequence feature. The image decoding network can be a transformer, a fully-connected layer fully-connect, etc.

[0081] S206, determining a text loss according to the sample text encoding feature and the text vector.

[0082] In this embodiment, the text loss refers to a loss calculated based on the predicted character of the covered character in the sample text.

[0083] An optional manner can be based on a text decoding network to determine the predicted character feature of the covered character in the sample text according to the sample text encoding feature of the un-covered character in the sample text; and then based on a preset loss function, the text loss is determined according to the predicted character feature of the covered character in the sample text and the text vector, for example, the text loss can be determined based on the cross-entropy loss function according to the predicted character feature of the covered character in the sample text and the text vector. The text decoding network can be a transformer, a fully-connected layer fully-connect, etc.

[0084] S207, pre-training the target detection model according to the contrast loss, the pixel reconstruction loss and the text loss.

[0085] Specifically, the training loss can be determined according to the contrast loss, the pixel reconstruction loss and the text loss based on a preset loss fusion manner, and then the target detection model can be pre-trained by using the training loss. For example, the contrast loss, the pixel reconstruction loss and the text loss can be added, the result of the addition is taken as the training loss, and then the target detection model can be pre-trained by using the training loss. For another example, the contrast loss, the pixel reconstruction loss and the text loss can be weighted and summed, the result of the summation is taken as the training loss, and then the target detection model can be pre-trained by using the training loss.

[0086] The technical solution provided by the embodiments of the present disclosure comprises the following steps: obtaining a sample image-text pair, then determining a sample image encoding feature of an unmasked image region in the sample image and a sequence feature of a masked image region in the sample image, and determining a sample text encoding feature of an unmasked character in the sample text and a text vector of a masked character in the sample text, then determining a contrast loss according to the sample image encoding feature and the sample text encoding feature, determining a pixel reconstruction loss according to the sample image encoding feature and the sequence feature, and determining a text loss according to the sample text encoding feature and the text vector, and finally pre-training the target detection model according to the contrast loss, the pixel reconstruction loss and the text loss. The above technical solution introduces the contrast loss, which can help the target detection model to model the correlation between the image and the text at the same time, well grasp the similarity between the images with similar text candidate boxes, and enable the model to learn more general and effective information, thereby making the target detection more accurate.

[0087] On the basis of the above-mentioned embodiments, as an optional manner of the present disclosure, the determination of the pixel reconstruction loss according to the sample image encoding feature and the sequence feature can comprise the following steps: fusing the sample image encoding feature and the sequence feature to obtain an image fusion feature; determining a reconstructed pixel value of the masked image region according to the image fusion feature; and determining the pixel reconstruction loss according to the reconstructed pixel value and an original pixel value of the masked image region.

[0088] The image fusion feature refers to a feature obtained by fusing the sample image encoding feature and the sequence feature, which can be represented in the form of a vector or a rectangle. The reconstructed pixel value refers to a predicted pixel value of the masked region obtained by reconstructing the masked image region in the sample image. The original pixel value refers to the true pixel value of the masked image region.

[0089] Specifically, the sample image coding feature of the unmasked image region and the sequence feature of the masked image region can be fused based on a preset feature fusion manner, for example, the sample image coding feature and the sequence feature can be superimposed or spliced to obtain pixel fusion features, and then the reconstruction pixel values of the masked image region in the sample image can be obtained based on the image decoding network according to the pixel fusion features; and then, the pixel reconstruction loss can be determined based on the preset loss function according to the reconstruction pixel values and the original pixel values of the masked image region, for example, the pixel reconstruction loss can be determined based on the mean square error loss function according to the reconstruction pixel values and the original pixel values of the masked image region.

[0090] It can be understood that the pixel reconstruction loss can be determined by reconstructing the pixels of the masked image region, which can help the model learn the fine-grained information of the region of the image itself.

[0091] On the basis of the above-mentioned embodiments, as an optional manner of the present disclosure, the text loss can be determined according to the sample text coding feature and the text vector by fusing the sample text coding feature and the text vector to obtain text fusion features; the predicted character of the masked character is determined according to the text fusion features; and the text loss is determined according to the predicted character and the original character of the masked character.

[0092] The text fusion features refer to the features obtained by fusing the sample text coding feature and the text vector, which can be represented in the form of a vector or a rectangle. The predicted character refers to the result predicted for the masked character in the sample text. The original character refers to the true character of the masked character in the sample text.

[0093] Specifically, the sample text coding feature of the unmasked character and the text vector of the masked character can be fused based on a preset feature fusion manner, for example, the sample text coding feature and the text vector can be superimposed or spliced to obtain text fusion features, and then the predicted character of the masked character can be predicted based on the text decoding network according to the text fusion features; and then, the text loss can be determined based on the preset loss function according to the predicted character and the original character of the masked character, for example, the text loss can be determined based on the cross-entropy loss function according to the predicted character and the original character of the masked character.

[0094] It can be understood that the text loss can be determined by predicting the masked character, which can help the model learn the semantic information carried by the sample text.

[0095] Figure 3is a flowchart of a training method of a target detection model according to an embodiment of the present disclosure. The embodiment is applicable to the case of how to train a target detection model in a target detection scenario. The apparatus can be implemented in software and / or hardware, and can be integrated into an electronic device that carries the training function of the target detection model, such as a server. As shown in Figure 3 The training method of the target detection model of the embodiment can include the following steps.

[0096] S301, serializing and converting a sample to-be-detected image to obtain an image sequence.

[0097] In the embodiment, the sample to-be-detected image refers to an image including at least one object for fine-tuning the target detection model; it should be noted that the sample to-be-detected image can be small-scale image data in a downstream target detection business scenario. The image sequence refers to the result obtained after serializing the sample to-be-detected image.

[0098] Specifically, the sample to-be-detected image can be divided into at least two image regions, and then the at least two image regions are serialized to obtain an image sequence, for example, a fully connected network can be used to convert and process the at least two image regions to obtain the image sequence.

[0099] S302, fine-tuning the target detection model according to the image sequence.

[0100] The target detection model is trained based on the pre-training method of the target detection model provided in the above embodiments of the present disclosure.

[0101] An optional way is to input the image sequence into the target detection model to adjust the model parameters in the target detection model.

[0102] The technical solution provided by the embodiments of the present disclosure is to serialize and convert a sample to-be-detected image to obtain an image sequence, and then fine-tune the target detection model according to the image sequence. The above technical solution has better detection performance for the model obtained by fine-tuning the pre-trained target detection model, thereby laying a foundation for the target detection task.

[0103] On the basis of the above embodiments, as an optional way of the present disclosure, fine-tuning the target detection model according to the image sequence can be extracting features from the image sequence to obtain to-be-detected image features; inputting the to-be-detected image features into a position detection network and a category recognition network of the target detection model respectively to obtain sample detection results of the sample to-be-detected image; and fine-tuning the target detection model according to the sample detection results and sample label data of the sample to-be-detected image.

[0104] The image features to be detected are features obtained by performing feature extraction on the image sequence of the sample image to be detected, and can be represented in the form of a vector or a matrix. The position detection network is used to predict the bounding box information of the target in the sample image to be detected, including but not limited to the center position, height and width of the bounding box, etc. The category recognition network is used to predict the category information of the target in the sample image to be detected. The sample detection result is the result of detecting the target in the sample image to be detected, including but not limited to the sample bounding box information and sample category information of the target in the sample image to be detected. The sample label data is the real bounding box information and real category information of the target in the sample image to be detected.

[0105] Specifically, the image encoding network and the image decoding network of the target detection model can be used to extract features from the image sequence to obtain the image features to be detected of the sample image to be detected. Then, the image features to be detected are input into the position detection network and the category recognition network of the target detection model to obtain the sample detection result of the sample image to be detected. Further, the fine-tuning loss can be determined based on the sample detection result of the sample image to be detected and the sample label data according to a preset loss function, and then the target detection model is trained using the fine-tuning loss to adjust the parameters in the target detection model.

[0106] It should be noted that, in the fine-tuning process of the target detection model, only the image encoding network and the image decoding network of the target detection model are fine-tuned, and the text encoding network and the text decoding network of the target detection model are not used, so that the target detection model can be fine-tuned more quickly and efficiently.

[0107] It can be understood that, after the target detection model is pre-trained and learned based on large-scale data, the target detection model is fine-tuned using small-scale business scenario data (only image data), so that a practical target detection model can be obtained efficiently and quickly, thereby improving the accuracy of target detection in the target detection scene task.

[0108] Figure 4 is a flowchart of a target detection method according to an embodiment of the present disclosure. The embodiment is applicable to the case of how to detect the target detection in the target detection scene. The apparatus can be realized in the form of software and / or hardware, and can be integrated in an electronic device carrying a target detection function, such as a server. As shown in Figure 4 The target detection method of the embodiment can include:

[0109] S401, obtaining a target image to be detected.

[0110] In the embodiment, the target image to be detected is an image that needs to be detected in a specific target detection scene.

[0111] Specifically, a target to-be-detected image that needs to be detected by the demander can be acquired. The demander refers to a party that needs to perform image target detection, and includes but is not limited to individuals, institutions, and the like.

[0112] In S402, a target detection model is used to detect the target to-be-detected image, to obtain a target detection result of the target to-be-detected image.

[0113] The target detection model is trained according to the training method of the target detection model provided in the embodiments of the present disclosure.

[0114] The target detection result refers to a result of detecting the target to-be-detected image, and can include but is not limited to bounding box position information and category information of a target in the target to-be-detected image.

[0115] Specifically, the target to-be-detected image can be input into the target detection model, and the target to-be-detected image is detected by the target detection model to obtain the target detection result of the target to-be-detected image.

[0116] The technical scheme of the embodiments of the present disclosure acquires a target to-be-detected image, and then uses a target detection model to detect the target to-be-detected image, to obtain a target detection result of the target to-be-detected image. The above technical scheme uses a pre-trained target detection model to perform target detection, which can improve the accuracy of target detection.

[0117] Figure 5 FIG. 1 is a structural schematic diagram of a pre-training device of a target detection model according to an embodiment of the present disclosure. The embodiments of the present disclosure are applicable to the case of how to pre-train a target detection model in a target detection scenario. The device can be implemented in software and / or hardware, and can be integrated in an electronic device that carries the pre-training function of the target detection model, such as a server. As shown in FIG. 1, the pre-training device 500 of the target detection model of the present embodiment can include: Figure 5

[0118] A sample image-text pair acquisition module 501 is configured to acquire a sample image-text pair. The sample image-text pair includes a sample image and a sample text, and the sample text is a label of a target in the sample image.

[0119] A sample image feature determination module 502 is configured to determine a sample image encoding feature of an unmasked image region in the sample image, and a sequence feature of a masked image region in the sample image.

[0120] A sample text feature determination module 503 is configured to determine a sample text encoding feature of an unmasked character in the sample text, and a text vector of a masked character in the sample text.

[0121] ​The target detection model training module 504 is configured to pre-train the target detection model according to the sample image coding feature, the sequence feature, the sample text coding feature, and the text vector.

[0122] The technical solution provided by the embodiments of the present disclosure comprises the following steps: obtaining a sample image-text pair, determining a sample image coding feature of an unmasked image region in the sample image and a sequence feature of a masked image region in the sample image, further determining a sample text coding feature of an unmasked character in the sample text and a text vector of a masked character in the sample text, and finally pre-training the target detection model according to the sample image coding feature, the sequence feature, the sample text coding feature, and the text vector. The technical solution introduces text features, pre-trains the target detection model in combination with image features and text features, can learn more general and effective information, obtains a target detection model with better performance, and thus improves the accuracy of target detection.

[0123] Further, the sample image feature determination module 502 is specifically configured to:

[0124] determine an unmasked image region from the sample image;

[0125] perform feature extraction on the unmasked region to obtain a sample image coding feature of the unmasked image region.

[0126] Further, the sample text feature determination module 503 is specifically configured to:

[0127] determine an unmasked character from the sample text;

[0128] perform feature extraction on the unmasked character to obtain a sample text coding feature of the unmasked character.

[0129] Further, the target detection model training module 503 comprises:

[0130] a contrast loss determination unit configured to determine a contrast loss according to the sample image coding feature and the sample text coding feature;

[0131] a pixel reconstruction loss determination unit configured to determine a pixel reconstruction loss according to the sample image coding feature and the sequence feature;

[0132] a text loss determination unit configured to determine a text loss according to the sample text coding feature and the text vector;

[0133] a target detection model training unit configured to pre-train the target detection model according to the contrast loss, the pixel reconstruction loss, and the text loss.

[0134] Further, the pixel reconstruction loss determination unit is specifically configured to:

[0135] Fusing the sample image coding feature and the sequence feature to obtain an image fusion feature;

[0136] According to the image fusion feature, a reconstructed pixel value of a masked image region is determined.

[0137] According to the reconstructed pixel value and an original pixel value of the masked image region, a pixel reconstruction loss is determined.

[0138] Further, the text loss determination unit is specifically configured to:

[0139] Fusing the sample text coding feature and the text vector to obtain a text fusion feature;

[0140] According to the text fusion feature, a predicted character of a masked character is determined.

[0141] According to the predicted character and an original character of the masked character, a text loss is determined.

[0142] Figure 6 is a structural schematic diagram of a training device of a target detection model according to an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to the case of how to train the target detection model in the target detection scene. The device can be realized in the form of software and / or hardware, and can be integrated in an electronic device that carries the training function of the target detection model, such as a server. As shown in Figure 6 The training device 600 of the target detection model of the embodiment can include:

[0143] An image sequence determination module 601 is configured to perform sequence conversion on a sample to-be-detected image to obtain an image sequence.

[0144] A target detection model fine-tuning module 602 is configured to fine-tune the target detection model according to the image sequence; wherein the target detection model is trained based on the pre-training method of the target detection model of any embodiment of the present disclosure.

[0145] The technical solution provided by the embodiment of the present disclosure converts the sample to-be-detected image into an image sequence, and then fine-tunes the target detection model according to the image sequence. The above technical solution has a model with better detection performance obtained by fine-tuning the pre-trained target detection model, thereby laying a foundation for the target detection task.

[0146] Further, the target detection model fine-tuning module 602 is specifically configured to:

[0147] Extracting features from the image sequence to obtain to-be-detected image features;

[0148] Inputting the to-be-detected image features into a position detection network and a category recognition network of the target detection model respectively to obtain a sample detection result of the sample to-be-detected image.

[0149] According to the sample detection result and the sample label data of the sample to-be-detected image, the target detection model is fine-tuned.

[0150] Figure 7 is a structural schematic diagram of a target detection device provided according to an embodiment of the present disclosure. The embodiment of the present disclosure is suitable for the case of how to detect target detection in a target detection scene. The device can be realized in a software and / or hardware manner, and can be integrated in an electronic device carrying a target detection function, such as a server. As shown in Figure 7 The target detection device 700 of the embodiment can include:

[0151] The target to-be-detected image acquisition module 701 is configured to acquire a target to-be-detected image.

[0152] The target detection result determination module 702 is configured to detect the target to-be-detected image by using a target detection model to obtain a target detection result of the target to-be-detected image. The target detection model is obtained by training according to the training method of the target detection model of any embodiment of the present disclosure.

[0153] The technical solution of the embodiment of the present disclosure acquires a target to-be-detected image, and then detects the target to-be-detected image by using a target detection model to obtain a target detection result of the target to-be-detected image. The above technical solution can improve the accuracy of target detection by using a pre-trained target detection model for target detection.

[0154] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0155] Figure 8 is a block diagram of an electronic device for implementing the pre-training method, the training method or the target detection method of the target detection model of the embodiment of the present disclosure. Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the present disclosure described and / or claimed in this document.

[0156] As Figure 8As shown, the electronic device 800 includes a computing unit 801 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 802 or a computer program loaded into a random access memory (RAM) 803 from a storage unit 808. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0157] A plurality of components in the electronic device 800 are connected to the I / O interface 805, including an input unit 806 such as a keyboard, a mouse, and the like, an output unit 807 such as various types of displays, a speaker, and the like, a storage unit 808 such as a magnetic disk, an optical disk, and the like, and a communication unit 809 such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0158] The computing unit 801 can be various general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, and the like. The computing unit 801 performs various methods and processes described above, such as the pre-training method, the training method, or the object detection method of the object detection model. For example, in some embodiments, the pre-training method, the training method, or the object detection method of the object detection model can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the pre-training method, the training method, or the object detection method of the object detection model described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the pre-training method, the training method, or the object detection method of the object detection model by any other appropriate means, such as by means of firmware.

[0159] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0160] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0161] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0162] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0163] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0164] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0165] Artificial intelligence is a discipline that studies enabling computers to simulate some human thinking processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), both hardware and software technologies. Artificial intelligence hardware technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing, etc.; artificial intelligence software technology mainly includes computer vision technology, speech recognition technology, natural language processing technology, and machine learning / deep learning technology, big data processing technology, knowledge graph technology, etc. several major directions.

[0166] Cloud computing refers to a technology system that accesses a pool of elastic scalable shared physical or virtual resources through a network, the resources can include servers, operating systems, networks, software, applications and storage devices, etc., and the resources can be deployed and managed in a self-service manner as needed. Through cloud computing technology, efficient and powerful data processing capabilities can be provided for artificial intelligence, blockchain and other technical applications and model training.

[0167] It should be understood that the various forms of flow shown above can be used to reorder, add or delete steps. For example, the steps described in the present disclosure can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, which is not limited herein.

[0168] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A pre-training method of an object detection model, comprising: obtaining a sample image-text pair, wherein the sample image-text pair comprises a sample image and a sample text, and the sample text is a label of an object in the sample image; determining a sample image encoding feature of an unmasked image region in the sample image and a sequence feature of a masked image region in the sample image, wherein the unmasked image region is obtained by randomly masking an image block in the sample image; determining a sample text encoding feature of an unmasked character in the sample text and a text vector of a masked character in the sample text, wherein the unmasked character is obtained by randomly masking a character in the sample text; pre-training an object detection model according to the sample image encoding feature, the sequence feature, the sample text encoding feature and the text vector; wherein the pre-training of the object detection model according to the sample image encoding feature, the sequence feature, the sample text encoding feature and the text vector comprises: splicing the sample image encoding feature, the sequence feature, the sample text encoding feature and the text vector, taking the spliced features as fused features, and pre-training the object detection model using the fused features.

2. The method of claim 1, wherein, determining the sample image encoding feature of the unmasked region in the sample image comprises: determining the unmasked image region from the sample image; extracting features of the unmasked region to obtain the sample image encoding feature of the unmasked image region.

3. The method of claim 1, wherein, determining the sample text encoding feature of the unmasked character in the sample text comprises: determining the unmasked character from the sample text; extracting features of the unmasked character to obtain the sample text encoding feature of the unmasked character.

4. The method of claim 1, wherein, the pre-training of the object detection model according to the sample image encoding feature, the sequence feature, the sample text encoding feature and the text vector comprises: determining a contrast loss according to the sample image encoding feature and the sample text encoding feature; determining a pixel reconstruction loss according to the sample image encoding feature and the sequence feature; determining a text loss according to the sample text encoding feature and the text vector; pre-training the object detection model according to the contrast loss, the pixel reconstruction loss and the text loss.

5. The method of claim 4, wherein, the determination of the pixel reconstruction loss according to the sample image encoding feature and the sequence feature comprises: fusing the sample image encoding feature and the sequence feature to obtain image fused features; determining reconstructed pixel values of the masked image region according to the image fused features; determining a pixel reconstruction loss according to the reconstructed pixel values and original pixel values of the masked image region.

6. The method of claim 4, wherein, the determination of the text loss according to the sample text encoding feature and the text vector comprises: fusing the sample text encoding feature and the text vector to obtain text fused features; determining predicted characters of the masked character according to the text fused features; Determine a text loss according to the predicted character and the original character of the masked character.

7. A training method of an object detection model, comprising: serializing a sample to-be-detected image to obtain an image sequence; fine-tuning an object detection model according to the image sequence; wherein the object detection model is trained based on the pre-training method of the object detection model in any one of claims 1-6.

8. The method of claim 7, wherein, The fine-tuning of the object detection model according to the image sequence comprises: extracting features from the image sequence to obtain to-be-detected image features; inputting the to-be-detected image features into a position detection network and a category recognition network of the object detection model respectively to obtain a sample detection result of the sample to-be-detected image; fine-tuning the object detection model according to the sample detection result and a sample label data of the sample to-be-detected image.

9. An object detection method, comprising: obtaining a target to-be-detected image; detecting the target to-be-detected image by using an object detection model to obtain a target detection result of the target to-be-detected image; wherein the object detection model is trained based on the training method of the object detection model in any one of claims 7-8.

10. A pre-training device of an object detection model, comprising: a sample image-text pair acquisition module configured to acquire a sample image-text pair; wherein the sample image-text pair comprises a sample image and a sample text, and the sample text is a label of an object in the sample image; a sample image feature determination module configured to determine a sample image coding feature of an unmasked image region in the sample image and a sequence feature of a masked image region in the sample image; the unmasked image region refers to an image region obtained by randomly masking an image block in the sample image; a sample text feature determination module configured to determine a sample text coding feature of an unmasked character in the sample text and a text vector of a masked character in the sample text; the unmasked character refers to a character obtained by randomly masking a character in the sample text; an object detection model training module configured to pre-train an object detection model according to the sample image coding feature, the sequence feature, the sample text coding feature and the text vector; wherein the object detection model training module is specifically configured to: splice the sample image coding feature, the sequence feature, the sample text coding feature and the text vector, take the spliced features as fusion features, and pre-train the object detection model by using the fusion features.

11. The apparatus of claim 10, wherein, The sample image feature determination module is specifically configured to: determine the unmasked image region from the sample image; extract features from the unmasked image region to obtain the sample image coding feature of the unmasked image region.

12. The apparatus of claim 10, wherein, The sample text feature determination module is specifically configured to: determine the unmasked character from the sample text; extract features from the unmasked character to obtain the sample text coding feature of the unmasked character.

13. The apparatus of claim 10, wherein, The object detection model training module comprises: a contrast loss determination unit, configured to determine a contrast loss according to the sample image encoding feature and the sample text encoding feature; a pixel reconstruction loss determination unit, configured to determine a pixel reconstruction loss according to the sample image encoding feature and the sequence feature; a text loss determination unit, configured to determine a text loss according to the sample text encoding feature and the text vector; a target detection model training unit, configured to pre-train the target detection model according to the contrast loss, the pixel reconstruction loss and the text loss.

14. The apparatus of claim 13, wherein, The pixel reconstruction loss determination unit is specifically configured to: fuse the sample image encoding feature and the sequence feature to obtain image fused features; determine reconstructed pixel values of the masked image region according to the image fused features; determine a pixel reconstruction loss according to the reconstructed pixel values and original pixel values of the masked image region.

15. The apparatus of claim 13, wherein, The text loss determination unit is specifically configured to: fuse the sample text encoding feature and the text vector to obtain text fused features; determine predicted characters of the masked characters according to the text fused features; determine a text loss according to the predicted characters and original characters of the masked characters.

16. A training apparatus of a target detection model, comprising: an image sequence determination module, configured to perform sequence conversion on a sample to-be-detected image to obtain an image sequence; a target detection model fine-tuning module, configured to fine-tune a target detection model according to the image sequence; wherein the target detection model is trained based on the pre-training method of the target detection model in any one of claims 1-6.

17. The apparatus of claim 16, wherein, The target detection model fine-tuning module is specifically configured to: perform feature extraction on the image sequence to obtain to-be-detected image features; input the to-be-detected image features into a position detection network and a category recognition network of the target detection model respectively to obtain sample detection results of the sample to-be-detected image; fine-tune the target detection model according to the sample detection results and sample label data of the sample to-be-detected image.

18. A target detection apparatus, comprising: a target to-be-detected image acquisition module, configured to acquire a target to-be-detected image; a target detection result determination module, configured to detect the target to-be-detected image by using a target detection model to obtain a target detection result of the target to-be-detected image; wherein the target detection model is trained according to the training method of the target detection model in any one of claims 7-8.

19. An electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the pre-training method of the target detection model in any one of claims 1-6, the training method of the target detection model in any one of claims 7-8, or the target detection method in claim 9.

20. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to make the computer execute the pre-training method of the target detection model according to any one of claims 1-6, the training method of the target detection model according to any one of claims 7-8, or the target detection method according to claim 9.

21. A computer program product comprising a computer program which, when executed by a processor, implements the pre-training method of the target detection model according to any one of claims 1-6, the training method of the target detection model according to any one of claims 7-8, or the target detection method according to claim 9.

Images