Cross-modal multi-scale fusion detection model training and detection method, device and equipment

Abstract

The present disclosure relates to a cross-modal multi-scale fusion detection model training, detection method, device and equipment. The cross-modal multi-scale fusion detection model training method comprises: obtaining the class label and position information corresponding to each target object in the sample image; determining the image region corresponding to each target object; for each target object, calculating a first similarity, a second similarity and a third similarity through the cross-modal multi-scale fusion detection model to be trained; calculating a loss value according to the first similarity, the second similarity, the third similarity and a preset loss function; and training the cross-modal multi-scale fusion detection model to be trained according to the loss value. According to the present disclosure, the accuracy of the target object recognition result in the image can be improved through the cross-modal multi-scale fusion detection model obtained by training.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a cross-modal multi-scale fusion detection model training and detection method, device and equipment. BACKGROUND

[0002] With the popularization of image acquisition technology, the use range of images is wider and wider. In order to better use images to perform some tasks such as automatic driving tasks, further processing of images is needed, such as recognition of target objects in images.

[0003] The prior art can recognize target objects in images through target recognition, so as to obtain a recognition result of the target objects. For example, the recognition result includes the category of the target objects and the position of the target objects in the image. The main principle of the existing object detection algorithm is to traverse the image in a sliding window manner to obtain a candidate region in the image, then extract image features of each candidate region, and based on the extracted image features, recognize the category and position of the target objects in the candidate region.

[0004] However, the accuracy of the recognition result obtained by the prior art is low. SUMMARY

[0005] In order to solve the above technical problems, the present disclosure provides a cross-modal multi-scale fusion detection model training and detection method, device and equipment.

[0006] A first aspect of the embodiments of the present disclosure provides a cross-modal multi-scale fusion detection model training method, comprising:

[0007] Obtaining a category label and position information corresponding to each target object in a plurality of target objects in a sample image;

[0008] Determining an image region corresponding to each target object based on the position information corresponding to each target object;

[0009] For each target object in the plurality of target objects, calculating a first image feature of the image region corresponding to the target object and a first text feature of the category label corresponding to the target object by using a cross-modal multi-scale fusion detection model to be trained;

[0010] Calculating a first similarity of the first image feature and the first text feature, and a second similarity of the first image feature and each second text feature in at least one second text feature, each second text feature being a text feature of a category label corresponding to any target object other than the target object;

[0011] a third similarity between the first image feature and each of at least one second text feature, each of the at least one second text feature being a text feature of a category label corresponding to any of the target objects other than the target object;

[0012] a loss value is calculated according to the first similarity, the second similarity, the third similarity, and a preset loss function;

[0013] The cross-modal multi-scale fusion detection model is trained according to the loss value.

[0014] A second aspect of the embodiments of the present disclosure provides a detection method, comprising:

[0015] obtaining a target image;

[0016] inputting the target image into the cross-modal multi-scale fusion detection model, so that the cross-modal multi-scale fusion detection model outputs a recognition result of at least one target object in the target image, the recognition result of each target object comprising a category label and position information of the target object, and the cross-modal multi-scale fusion detection model being trained by the training method of the first aspect.

[0017] A third aspect of the embodiments of the present disclosure provides a cross-modal multi-scale fusion detection model training apparatus, comprising:

[0018] a data acquisition module configured to acquire a category label and position information corresponding to each of a plurality of target objects in a sample image;

[0019] a region determination module configured to determine an image region corresponding to each of the plurality of target objects based on the position information corresponding to each of the plurality of target objects;

[0020] a first calculation module configured to calculate, for each of the plurality of target objects, a first image feature of the image region corresponding to the target object and a first text feature of the category label corresponding to the target object by using a cross-modal multi-scale fusion detection model to be trained;

[0021] a second calculation module configured to calculate a first similarity between the first image feature and the first text feature, and a second similarity between the first image feature and each of at least one second text feature, each of the at least one second text feature being a text feature of a category label corresponding to any of the target objects other than the target object;

[0022] a third calculation module configured to calculate a third similarity between the first text feature and each of at least one second image feature, each of the at least one second image feature being an image feature of an image region corresponding to any of the target objects other than the target object;

[0023] The fourth computing module is configured to compute a loss value according to the first similarity, the second similarity, the third similarity, and a preset loss function.

[0024] The model training module is configured to train the cross-modal multi-scale fusion detection model to be trained according to the loss value.

[0025] A fourth aspect of the embodiments of the present disclosure provides a detection device, which comprises:

[0026] The image acquisition module is configured to acquire a target image.

[0027] The object recognition module is configured to input the target image into the cross-modal multi-scale fusion detection model, so that the cross-modal multi-scale fusion detection model outputs a recognition result of at least one target object in the target image, the recognition result of each target object comprising a class label and position information of the target object, and the cross-modal multi-scale fusion detection model being trained by the training method of the first aspect.

[0028] A fifth aspect of the embodiments of the present disclosure provides an electronic device, which comprises:

[0029] The processor;

[0030] The memory is configured to store executable instructions.

[0031] The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the cross-modal multi-scale fusion detection model training method of the first aspect and the detection method of the second aspect.

[0032] The technical solutions provided by the embodiments of the present disclosure have the following advantages compared with the prior art:

[0033] The cross-modal multi-scale fusion detection model training method provided by the embodiments of the present disclosure comprises the following steps: obtaining a plurality of target objects in a sample image, and obtaining a class label and position information corresponding to each target object in the plurality of target objects; determining an image region corresponding to each target object based on the position information corresponding to each target object; for each target object in the plurality of target objects, calculating a first image feature of the image region corresponding to the target object and a first text feature of the class label corresponding to the target object by using a cross-modal multi-scale fusion detection model to be trained; calculating a first similarity between the first image feature and the first text feature, and a second similarity between the first image feature and each of at least one second text feature, wherein each second text feature is a text feature of a class label corresponding to any target object other than the target object; calculating a third similarity between the first text feature and each of at least one second image feature, wherein each second image feature is an image feature of an image region corresponding to any target object other than the target object; calculating a loss value based on the first similarity, the second similarity, the third similarity, and a preset loss function; and training the cross-modal multi-scale fusion detection model to be trained based on the loss value. Since the text feature and the image feature of the sample image are considered in the process of training the cross-modal multi-scale fusion detection model to be trained, the accuracy of the identification result of the target object in the target image is improved when the target image is processed based on the cross-modal multi-scale fusion detection model. BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the accompanying drawings required to be used in the embodiments or prior art description will be briefly introduced. Obviously, those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0036] Figure 1 is a flowchart of a cross-modal multi-scale fusion detection model training method provided by the embodiments of the present disclosure;

[0037] Figure 2 is a schematic diagram of a sample image provided by the embodiments of the present disclosure;

[0038] Figure 3 is a schematic diagram of a cross-modal multi-scale fusion detection model provided by the embodiments of the present disclosure;

[0039] Figure 4is a flowchart of a detection method provided by an embodiment of the present disclosure.

[0040] Figure 5 is a structural schematic diagram of a cross-modal multi-scale fusion detection model training device provided by an embodiment of the present disclosure.

[0041] Figure 6 is a structural schematic diagram of a detection device provided by an embodiment of the present disclosure.

[0042] Figure 7 is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] In order to more clearly understand the above-mentioned purposes, features and advantages of the present disclosure, the schemes of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0044] In the following description, many specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure can also be implemented in other different manners from those described herein; obviously, the embodiments described in the specification are only a part of the embodiments of the present disclosure, and not all the embodiments.

[0045] It should be understood that each step recorded in the method embodiments of the present disclosure can be executed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present disclosure is not limited in this respect.

[0046] It should be noted that, in this document, relational terms such as "first" and "second", and the like, are used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between or among the entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the phrase "comprising a... " does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0047] It should be noted that the modification of "one" and "multiple" mentioned in the present disclosure is illustrative and not restrictive, and those skilled in the art should understand that, unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0048] Generally, the main principle of the object detection algorithm is to traverse the image in a sliding window manner to obtain candidate regions in the image, then extract image features of each candidate region, and based on the extracted image features, identify the class and position of the target object in the candidate region. However, the accuracy of the recognition result obtained by the prior art is low. In view of this problem, the present embodiment provides a cross-modal multi-scale fusion detection model training method, which will be introduced below in combination with specific embodiments.

[0049] Figure 1 is a flowchart of a cross-modal multi-scale fusion detection model training method provided by the present embodiment. The method can be executed by a cross-modal multi-scale fusion detection model training device, which can be realized in software and / or hardware. The cross-modal multi-scale fusion detection model training device can be configured in an electronic device, such as a server or a terminal.

[0050] The cross-modal multi-scale fusion detection model training method shown in Figure 1 will be introduced below, which can be executed by an electronic device. As shown in Figure 1 , the cross-modal multi-scale fusion detection model training method provided by the present embodiment includes the following steps.

[0051] S110, obtaining the class label and position information corresponding to each of the plurality of target objects in the sample image.

[0052] In the present embodiment, when the electronic device needs to train the cross-modal multi-scale fusion detection model to be trained, the class label and position information corresponding to each of the plurality of target objects in the sample image are obtained.

[0053] In the present embodiment, the sample image is an image used for training, and the number of sample images is multiple. There is at least one target object in each sample image, and the target object is an object existing in the image.

[0054] The position information can be used to represent the position of the target object in the image, such as the pixel coordinates of the target object in the image.

[0055] Figure 2 is a schematic diagram of a sample image provided by the present embodiment, as shown in Figure 2 , there are two target objects in the image, which are basketball 001 and house 002, where "basketball" and "house" are the class labels of the target objects.

[0056] Specifically, the electronic device obtains sample images, and class labels and position information corresponding to each target object in the sample images from a preset database when needing to train a cross-modal multi-scale fusion detection model to be trained, or receives a training instruction.

[0057] Exemplarily, the preset database can be a Flickr website database, an MS-COCO database, or the like, and a plurality of sample images, and class labels and position information corresponding to each target object in the plurality of target objects in the sample images can be obtained from different databases respectively, and the cross-modal multi-scale fusion detection model to be trained is trained, which is not limited herein.

[0058] In the embodiment of the present disclosure, the class label and the position information corresponding to each target object can be represented in the form of a triplet data as wherein, x e X is an image region corresponding to the i th target object, t e T is a class label corresponding to the i th target object, y e Y is two coordinates, which represents the position information of the i th target object in the sample image, and each target object corresponds to a triplet data.

[0059] In S120, an image region corresponding to each target object is determined based on the position information corresponding to each target object.

[0060] In the embodiment of the present disclosure, after obtaining the class label and the position information corresponding to each target object in the plurality of target objects in the sample images, the electronic device determines an image region corresponding to each target object based on the position information corresponding to each target object.

[0061] Specifically, the position information can be pixel coordinates of a top-left corner and a bottom-right corner corresponding to each target object in an image, and the electronic device can determine an image region corresponding to each target object based on the pixel coordinates of the top-left corner and the bottom-right corner corresponding to each target object in the image.

[0062] In S130, for each target object in the plurality of target objects, a first image feature of an image region corresponding to the target object is calculated by the cross-modal multi-scale fusion detection model to be trained, and a first text feature of a class label corresponding to the target object is calculated.

[0063] In the embodiment of the present disclosure, after determining the image region corresponding to each target object, for each target object in the plurality of target objects, a first image feature of an image region corresponding to the target object is calculated by the cross-modal multi-scale fusion detection model to be trained, and a first text feature of a class label corresponding to the target object is calculated.

[0064] In the embodiments of the present disclosure, the cross-modal multi-scale fusion detection model to be trained can be a model capable of processing images and detecting target objects in images, such as an image processing model, without limitation.

[0065] Specifically, after determining the image region corresponding to each target object, the electronic device inputs each target object into the cross-modal multi-scale fusion detection model to be trained, and calculates a first image feature of the image region corresponding to the target object and a first text feature of the category label corresponding to the target object through the cross-modal multi-scale fusion detection model to be trained.

[0066] S140, calculate a first similarity of the first image feature and the first text feature, and a second similarity of the first image feature and each of at least one second text feature, each second text feature being a text feature of a category label corresponding to any target object other than the target object.

[0067] In the embodiments of the present disclosure, after obtaining the first image feature and the first text feature corresponding to each target object, the electronic device calculates, for each target object, a first similarity of the first image feature and the first text feature, and a second similarity of the first image feature and each of at least one second text feature through the cross-modal multi-scale fusion detection model to be trained.

[0068] Each second text feature is a text feature of a category label corresponding to any target object other than the target object.

[0069] For example, in the case of three target objects, target object 1, target object 2 and target object 3, the electronic device first calculates the similarity of the first image feature and the first text feature of target object 1 to obtain a first similarity, and then calculates the similarity of the first image feature of target object 1 and the second text feature of target object 2, and the similarity of the first image feature of target object 1 and the second text feature of target object 3.

[0070] S150, calculate a third similarity of the first text feature and each of at least one second image feature, each second image feature being an image feature of an image region corresponding to any target object other than the target object.

[0071] In the embodiments of the present disclosure, after obtaining the first image feature and the first text feature corresponding to each target object, the electronic device calculates, for each target object, a third similarity of the first text feature and each of at least one second image feature through the cross-modal multi-scale fusion detection model to be trained.

[0072] Each second image feature is an image feature of an image region corresponding to any target object other than the target object.

[0073] For example, in the case of three target objects, target object 1, target object 2 and target object 3, the electronic device calculates the similarity of the first text feature of target object 1 with the second image feature of target object 2 and the similarity of the first text feature of target object 1 with the second image feature of target object 3, respectively.

[0074] S160, calculating a loss value according to the first similarity, the second similarity, the third similarity and a preset loss function.

[0075] In the embodiments of the present disclosure, after obtaining the first similarity, the second similarity and the third similarity, the electronic device calculates a loss value according to the first similarity, the second similarity, the third similarity and a preset loss function.

[0076] Optionally, the preset loss function can be a standard cross-entropy loss formula.

[0077] Specifically, after obtaining the first similarity, the second similarity and the third similarity, the electronic device inputs the first similarity, the second similarity and the third similarity into the preset loss function for calculation, and the obtained value is the loss value.

[0078] S170, training the cross-modal multi-scale fusion detection model to be trained according to the loss value.

[0079] In the embodiments of the present disclosure, after obtaining the loss value, the electronic device adjusts the parameters of the cross-modal multi-scale fusion detection model to be trained according to the loss value, so as to train the cross-modal multi-scale fusion detection model to be trained, until the cross-modal multi-scale fusion detection model to be trained converges, and a trained cross-modal multi-scale fusion detection model is obtained.

[0080] In the embodiment of the present disclosure, by obtaining the class label and position information corresponding to each target object in the plurality of target objects in the sample image, determining the image region corresponding to each target object based on the position information corresponding to each target object, for each target object in the plurality of target objects, calculating the first image feature of the image region corresponding to the target object and the first text feature of the class label corresponding to the target object through the cross-modal multi-scale fusion detection model to be trained, calculating the first similarity of the first image feature and the first text feature, and the second similarity of the first image feature and each second text feature in the at least one second text feature, each second text feature being the text feature of the class label corresponding to any target object other than the target object, calculating the third similarity of the first text feature and each second image feature in the at least one second image feature, each second image feature being the image feature of the image region corresponding to any target object other than the target object, calculating the loss value according to the first similarity, the second similarity, the third similarity and the preset loss function, and then training the cross-modal multi-scale fusion detection model to be trained according to the loss value, since the text feature and the image feature of the sample image are considered in the process of training the cross-modal multi-scale fusion detection model to be trained, when the target image is processed based on the cross-modal multi-scale fusion detection model, the accuracy of the target object recognition result in the target image is improved.

[0081] On the basis of the above-mentioned embodiment of the present disclosure, the first image feature of the image region corresponding to the target object in S130 can specifically include: encoding the image region corresponding to the target object to obtain a representation vector corresponding to the image region; and performing feature extraction on the representation vector corresponding to the image region to obtain the first image feature.

[0082] In the embodiment of the present disclosure, the electronic device can encode the image region corresponding to the target object based on the preset image encoder to obtain a representation vector corresponding to the image region, and then input the representation vector into the feature extraction module to perform feature extraction on the representation vector to obtain the first image feature.

[0083] Figure 3 is a schematic diagram of a cross-modal multi-scale fusion detection model provided by the embodiment of the present disclosure, as Figure 3The shown cross-modal multi-scale fusion detection model can be a cross-modal multi-scale fusion detection model to be trained. The cross-modal multi-scale fusion detection model to be trained includes an image input layer configured to input an image region corresponding to a target object, a text input layer configured to input a category label corresponding to the target object, an encoder configured to encode the image region corresponding to the target object input by the image input layer, a first self-attention mechanism network configured to extract features from a representation vector obtained after processing by the encoder, a second self-attention mechanism network configured to extract features from the category label corresponding to the target object input by the text input layer, a cross-attention network configured to fuse image features and text features obtained by the first self-attention mechanism network and the second self-attention mechanism network respectively, a first feedforward neural network configured to further extract features from the fused features, and a second feedforward neural network configured to further extract features from the first text features. The first self-attention mechanism network, the cross-attention network, and the first feedforward neural network constitute a transformer module, i.e., a first transformer module, the second self-attention mechanism network and the second feedforward neural network constitute a transformer module, i.e., a second transformer module, and the first transformer module and the second transformer module can share parameters.

[0084] Optionally, the encoder can be an encoder in a contrastive text-image pair pre-training model (CLIP).

[0085] Exemplarily, the electronic device can determine the first text features based on Figure 3 The encoder in the cross-modal multi-scale fusion detection model to be trained encodes the image region corresponding to the target object to obtain a representation vector corresponding to the image region, and then extracts features from the representation vector based on the first self-attention mechanism network to obtain first image features.

[0086] In the embodiments of the present disclosure, the image region corresponding to the target object can be encoded based on a preset image encoder to obtain a representation vector corresponding to the image region, and then the representation vector can be input into a feature extraction module to extract features from the representation vector to obtain first image features, thereby improving the accuracy of the obtained first image features.

[0087] In the embodiments of the present disclosure, the first text features of the category label corresponding to the target object in S130 can specifically include: constructing at least one sentence according to the category label corresponding to the target object, the sentence including the category label; and determining the first text features according to feature information of each sentence in the at least one sentence.

[0088] In the embodiments of the present disclosure, the electronic device can construct a sentence for the category label corresponding to the target object by a preset sentence construction manner, to obtain at least one sentence corresponding to each category label.

[0089] Optionally, the preset sentence construction manner can be a prompt template manner.

[0090] For example, the category label is "apple", and the constructed sentence can be "I am an apple phone".

[0091] Specifically, the electronic device can construct a sentence for the category label corresponding to the target object by a preset sentence construction manner, to obtain a sentence containing the category label, and then perform feature extraction on the constructed sentence by a feature extraction module such as the second self-attention mechanism network shown in Figure 3 to obtain the text feature corresponding to each target object.

[0092] Further, the first similarity between the first image feature and the first text feature in S140 includes: performing fusion processing on the first image feature and the first text feature to obtain a fusion feature; and determining the similarity between the fusion feature and the first text feature as the first similarity between the first image feature and the first text feature.

[0093] In some embodiments of the present disclosure, the electronic device can perform fusion processing on the first image feature and the first text feature by a feature fusion module in the cross-modal multi-scale fusion detection model to be trained, such as the cross-attention network shown in Figure 3 to obtain a fusion feature, and then calculate the similarity between the fusion feature and the first text feature based on the fusion feature, and determine the similarity as the first similarity between the first image feature and the first text feature.

[0094] In other embodiments of the present disclosure, the electronic device can further perform feature extraction on the fusion feature by a first feedforward neural network in the first transformer module, as shown in Figure 3 to obtain a target fusion feature, and at the same time perform further feature extraction on the first text feature based on a second feedforward neural network in the second transformer module to obtain a target text feature, and then calculate the similarity between the target fusion feature and the target text feature, and determine the similarity as the first similarity between the first image feature and the first text feature.

[0095] For example, after processing the image region corresponding to the target object by the first transformer module as shown in Figure 3 to obtain the fusion feature, the specific formula is as follows:

[0096]

[0097] denotes the fusion feature of the i-th target object, which is in the form of a vector;

[0098] After the class label corresponding to the target object is processed by the second transformer module as shown in formula (3), a first text feature is obtained, and the specific formula is as follows: Figure 3

[0099]

[0100] denotes the first text feature of the i-th target object, which is in the form of a vector; Further, based on the fusion feature of the i-th target object, a first similarity and a second similarity are calculated, and the specific formula is as follows:

[0101]

[0102] wherein,

[0103] denotes the similarity of the i-th image to the j-th text; denotes the similarity of the i-th text to the j-th image; denotes the transposed feature vector corresponding to the fusion feature of the i-th target object; denotes the transposed feature vector corresponding to the first text feature of the i-th target object. In formula (4), when j is the same as i, the obtained similarity is the first similarity; when j is different from i, the obtained similarity is the second similarity.

[0104] In formula (5), when j is the same as i, the obtained similarity is the first similarity; when j is different from i, the obtained similarity is the third similarity.

[0105] In formula (5), when j is the same as i, the obtained similarity is the first similarity; when j is different from i, the obtained similarity is the third similarity. In the embodiments of the present disclosure, when the electronic device trains the cross-modal multi-scale fusion detection model to be trained, the fusion feature obtained by performing fusion processing on the first image feature and the first text feature corresponding to the target object is calculated with the first text feature corresponding to the target object, so as to improve the performance of the obtained cross-modal multi-scale fusion detection model and improve the accuracy of image recognition.

[0106]

[0107] ​​Further, the calculating the loss value according to the first similarity, the second similarity, the third similarity and the preset loss function in S160 can specifically include: performing normalization processing on the first similarity and the second similarity to obtain a first probability; performing normalization processing on the first similarity and the third similarity to obtain a second probability; calculating a mean value of the first probability and the second probability to obtain a target probability; and inputting the target probability into the preset loss function to obtain the loss value.

[0108] In the embodiments of the present disclosure, the normalization processing can be a softmax operation.

[0109] The first probability is a probability of image to text, and the second probability is a probability of text to image.

[0110] For example, the first similarity and the second similarity are calculated based on the first similarity, the second similarity and the third similarity, and the specific formula is as follows:

[0111]

[0112] wherein, denotes a probability of image to text; denotes a probability of text to image;

[0113] respectively represent the normalization processing process of image to text and text to image.

[0114] Further, after obtaining the first probability and the second probability, the first probability and the second probability are added and averaged to obtain a target probability, and the target probability is input into the preset loss function to obtain the loss value.

[0115] In the embodiments of the present disclosure, the first probability of image to text and the second probability of text to image can be calculated respectively, and then the loss value is obtained based on the first probability and the second probability, which improves the accuracy of the obtained loss value, and further improves the accuracy of the cross-modal multi-scale fusion detection model obtained by training.

[0116] Figure 4 is a flowchart of a detection method provided by the embodiments of the present disclosure, as shown in Figure 4 The detection method can be executed by an electronic device, and specifically includes the following steps.

[0117] S410, obtaining a target image.

[0118] In the embodiments of the present disclosure, the electronic device obtains the target image from a preset database when it is necessary to identify a target object of the target image.

[0119] S420. Input the target image into the cross-modal multi-scale fusion detection model, so that the cross-modal multi-scale fusion detection model outputs the recognition result of at least one target object in the target image, and the recognition result of each target object includes the category label and location information of the target object.

[0120] In this embodiment, the cross-modal multi-scale fusion detection model is trained using the cross-modal multi-scale fusion detection model training method described in the above embodiments.

[0121] The location information refers to the position of the target object in the target image, such as pixel coordinates.

[0122] In this embodiment of the disclosure, after acquiring the target image, the electronic device inputs the target image into a cross-modal multi-scale fusion detection model, so that the cross-modal multi-scale fusion detection model can identify the target image and output the identification result of at least one target object in the target image.

[0123] In this embodiment of the disclosure, by inputting the acquired target image into the cross-modal multi-scale fusion detection model, the cross-modal multi-scale fusion detection model can output the category label and location information of at least one target object in the target image. Since the cross-modal multi-scale fusion detection model combines the image features and text features corresponding to the target image, it can detect the target object by using information from multiple aspects of the target image, thereby improving the accuracy of the acquired category label and location information of the target object.

[0124] Figure 5 This is a schematic diagram of the structure of a cross-modal multi-scale fusion detection model training device provided in an embodiment of this disclosure.

[0125] In this embodiment, the cross-modal multi-scale fusion detection model training device can be housed within an electronic device and is understood as a functional module within the aforementioned electronic device. Specifically, the electronic device can be a server or a terminal, and no limitation is made herein.

[0126] like Figure 5 As shown, the cross-modal multi-scale fusion detection model training device 500 may include a data acquisition module 510, a region determination module 520, a first calculation module 530, a second calculation module 540, a third calculation module 550, a fourth calculation module 560, and a model training module 570.

[0127] The data acquisition module 510 can be used to acquire the category label and location information of each target object in a sample image.

[0128] The region determination module 520 can be used to determine the image region corresponding to each target object based on the location information corresponding to each target object.

[0129] The first calculation module 530 can be configured to calculate, for each target object in the plurality of target objects, a first image feature of an image region corresponding to the target object and a first text feature of a category label corresponding to the target object by the cross-modal multi-scale fusion detection model to be trained.

[0130] The second calculation module 540 can be configured to calculate a first similarity of the first image feature and the first text feature and a second similarity of the first image feature and each of at least one second text feature, each of the at least one second text feature being a text feature of a category label corresponding to any target object other than the target object.

[0131] The third calculation module 550 can be configured to calculate a third similarity of the first text feature and each of at least one second image feature, each of the at least one second image feature being an image feature of an image region corresponding to any target object other than the target object.

[0132] The fourth calculation module 560 can be configured to calculate a loss value according to the first similarity, the second similarity, the third similarity, and a preset loss function.

[0133] The model training module 570 can be configured to train the cross-modal multi-scale fusion detection model to be trained according to the loss value.

[0134] In the embodiments of the present disclosure, by obtaining the category label and the position information corresponding to each of the plurality of target objects in the sample image, determining the image region corresponding to each of the plurality of target objects based on the position information corresponding to each of the plurality of target objects, for each of the plurality of target objects, calculating the first image feature of the image region corresponding to the target object and the first text feature of the category label corresponding to the target object by using the cross-modal multi-scale fusion detection model to be trained, calculating the first similarity of the first image feature and the first text feature and the second similarity of the first image feature and each of the at least one second text feature, each of the at least one second text feature being the text feature of the category label corresponding to any target object other than the target object, calculating the third similarity of the first text feature and each of the at least one second image feature, each of the at least one second image feature being the image feature of the image region corresponding to any target object other than the target object, calculating the loss value according to the first similarity, the second similarity, the third similarity and the preset loss function, and then training the cross-modal multi-scale fusion detection model to be trained according to the loss value, since the text feature and the image feature of the sample image are considered in the process of training the cross-modal multi-scale fusion detection model to be trained, when the cross-modal multi-scale fusion detection model is used to process the target image, the accuracy of the target object recognition result in the target image is improved.

[0135] In some embodiments of the present disclosure, the first calculation module 530 can include an encoding unit 5301 and a first extraction unit 5302.

[0136] The encoding unit 5301 can be configured to encode the image region corresponding to the target object to obtain a representation vector corresponding to the image region.

[0137] The first extraction unit 5302 can be configured to perform feature extraction on the representation vector corresponding to the image region to obtain the first image feature.

[0138] In some embodiments of the present disclosure, the first calculation module 530 can further include a sentence construction unit 5303 and a second extraction unit 5304.

[0139] The sentence construction unit 5303 can be configured to construct at least one sentence according to the category label corresponding to the target object, the sentence including the category label.

[0140] The second extraction unit 5304 can be configured to determine the first text feature according to the feature information of each of the at least one sentence.

[0141] In some embodiments of the present disclosure, the second calculation module 540 can include a fusion processing unit 5401 and a similarity determination unit 5402.

[0142] The fusion processing unit 5401 can be configured to perform fusion processing on the first image feature and the first text feature to obtain a fusion feature.

[0143] The similarity determination unit 5402 can be configured to determine a similarity between the fusion feature and the first text feature as a first similarity between the first image feature and the first text feature.

[0144] In some embodiments of the present disclosure, the fourth calculation module 560 can include a first processing unit 5601, a second processing unit 5602, a mean calculation unit 5603, and a loss value determination unit 5604.

[0145] The first processing unit 5601 can be configured to perform normalization processing on the first similarity and the second similarity to obtain a first probability.

[0146] The second processing unit 5602 can be configured to perform normalization processing on the first similarity and the third similarity to obtain a second probability.

[0147] The mean calculation unit 5603 can be configured to calculate a mean of the first probability and the second probability to obtain a target probability.

[0148] The loss value determination unit 5604 can be configured to input the target probability into a preset loss function to obtain a loss value.

[0149] It should be noted that, Figure 5 The cross-modal multi-scale fusion detection model training apparatus 500 shown can perform each step of the above method embodiments, and achieve each process and effect of the above method embodiments, which are not repeated here.

[0150] Figure 6 FIG. 6 is a structural schematic diagram of a detection apparatus provided by an embodiment of the present disclosure.

[0151] In an embodiment of the present disclosure, the detection apparatus can be arranged in an electronic device and is understood as a part of the function modules of the electronic device. Specifically, the electronic device can be a server or a terminal, which is not limited here.

[0152] As shown in FIG. 6, Figure 6 The detection apparatus 600 can include an image acquisition module 610 and an object recognition module 620.

[0153] The image acquisition module 610 can be configured to acquire a target image.

[0154] The object recognition module 620 can be configured to input the target image into the cross-modal multi-scale fusion detection model, so that the cross-modal multi-scale fusion detection model outputs a recognition result of at least one target object in the target image, and the recognition result of each target object includes a category label and position information of the target object, and the cross-modal multi-scale fusion detection model is trained by using the cross-modal multi-scale fusion detection model training method in the above embodiments.

[0155] In the embodiments of the present disclosure, the target image obtained can be input into the cross-modal multi-scale fusion detection model, so that the cross-modal multi-scale fusion detection model outputs the category label and the position information of at least one target object in the target image. Since the image features and the text features corresponding to the target image are combined in the cross-modal multi-scale fusion detection model, the target object is detected by using the multi-aspect information of the target image, and thus the accuracy of the category label and the position information of the target object obtained is improved.

[0156] Figure 7 FIG. 1 is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure.

[0157] In the embodiments of the present disclosure, Figure 7 The electronic device shown in the figure can be a server or a terminal, which is not limited herein.

[0158] As shown in the figure, Figure 7 The electronic device can include a processor 710 and a memory 720 storing computer program instructions.

[0159] Specifically, the processor 710 can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or can be configured as one or more integrated circuits implementing the embodiments of the present disclosure.

[0160] The memory 720 can include mass storage for information or instructions. For example, and without limitation, the memory 720 can include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. Storage memory 720, where appropriate, can include removable or non-removable (or fixed) media. Storage memory 720 can be internal or external to the integrated gateway device, as appropriate. In particular embodiments, storage memory 720 is non-volatile solid-state memory. In particular embodiments, the memory 720 includes read-only memory (ROM). Where appropriate, this ROM can be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0161] The processor 710 performs the steps of the cross-modal multi-scale fusion detection model training method and detection method provided by the embodiments of the present disclosure by reading and executing the computer program instructions stored in the memory 720.

[0162] In one example, the electronic device can also include a transceiver 730 and a bus 740. Wherein, as shown, the processor 710, the memory 720 and the transceiver 730 are connected through the bus 740 and complete the communication between each other. Figure 7

[0163] ​Bus 740 includes a hardware, software, or both that couples components of computer system 700 to each other. By way of example, and not limitation, bus 740 can include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side BUS (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand™ interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or another suitable bus or a combination of two or more of these. Where appropriate, bus 740 can include one or more buses of the same type or buses of different types.

[0164] The embodiment of the present disclosure further provides a computer readable storage medium, which can store a computer program. When the computer program is executed by a processor, the processor implements the cross-modal multi-scale fusion detection model training method and the detection method provided by the embodiment of the present disclosure.

[0165] The storage medium described above can for example include a memory 720 storing computer program instructions, and the above instructions can be executed by the processor 710 of the electronic device to complete the cross-modal multi-scale fusion detection model training method and the detection method provided by the embodiment of the present disclosure. Optionally, the storage medium can be a non-transitory computer readable storage medium, for example, the non-transitory computer readable storage medium can be a ROM, a Random Access Memory (RAM), a Compact Disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0166] The foregoing is merely illustrative of the various implementations of the present disclosure and the general principles thereof. Numerous modifications can be made to these illustrations, and equivalents can be substituted therefor, without departing from the scope of the present disclosure. The specific embodiments commensurate with the specific application are intended to be illustrative only and not limiting of the scope of the application as set forth in the following claims.

Claims

1. A method for training a cross-modal multi-scale fusion detection model, characterized in that, The method comprises: obtaining a class label and position information corresponding to each target object in a plurality of target objects in a sample image; determining an image region corresponding to each target object based on the position information corresponding to each target object; for each target object in the plurality of target objects, calculating a first image feature of the image region corresponding to the target object and a first text feature of the class label corresponding to the target object by using a cross-modal multi-scale fusion detection model to be trained; calculating a first similarity between the first image feature and the first text feature, and a second similarity between the first image feature and each of at least one second text feature, wherein each second text feature is a text feature of a class label corresponding to any target object other than the target object; calculating a third similarity between the first text feature and each of at least one second image feature, wherein each second image feature is an image feature of an image region corresponding to any target object other than the target object; calculating a loss value according to the first similarity, the second similarity, the third similarity, and a preset loss function; training the cross-modal multi-scale fusion detection model to be trained according to the loss value.

2. The method of claim 1, wherein, The method further comprises: encoding the image region corresponding to the target object to obtain a representation vector corresponding to the image region; extracting a feature of the representation vector corresponding to the image region to obtain the first image feature.

3. The method of claim 1, wherein, The method further comprises: constructing at least one sentence according to the class label corresponding to the target object, wherein the sentence comprises the class label; determining the first text feature according to feature information of each sentence in the at least one sentence.

4. The method of claim 1, wherein, The method further comprises: fusing the first image feature and the first text feature to obtain a fused feature; determining the first similarity between the first image feature and the first text feature as a similarity between the fused feature and the first text feature.

5. The method of claim 1, wherein, The method further comprises: normalizing the first similarity and the second similarity to obtain a first probability; normalizing the first similarity and the third similarity to obtain a second probability; calculating a mean value of the first probability and the second probability to obtain a target probability; inputting the target probability into the preset loss function to obtain the loss value.

6. A method of detection, characterized in that The method comprises: obtaining a target image; inputting the target image into a cross-modal multi-scale fusion detection model, so that the cross-modal multi-scale fusion detection model outputs a recognition result of at least one target object in the target image, the recognition result of each target object comprising a class label and position information of the target object, the cross-modal multi-scale fusion detection model being trained by the training method in any one of claims 1-5.

7. A cross-modal multi-scale fusion detection model training apparatus, characterized in that, Comprising: a data acquisition module for acquiring a class label and position information corresponding to each of a plurality of target objects in a sample image; a region determination module for determining an image region corresponding to each of the target objects based on the position information corresponding to each of the target objects; a first calculation module for calculating, for each of the target objects, a first image feature of the image region corresponding to the target object and a first text feature of the class label corresponding to the target object by a cross-modal multi-scale fusion detection model to be trained; a second calculation module for calculating a first similarity between the first image feature and the first text feature and a second similarity between the first image feature and each of at least one second text feature, the second text feature being a text feature of a class label corresponding to any target object other than the target object; a third calculation module for calculating a third similarity between the first text feature and each of at least one second image feature, the second image feature being an image feature of an image region corresponding to any target object other than the target object; a fourth calculation module for calculating a loss value according to the first similarity, the second similarity, the third similarity, and a preset loss function; a model training module for training the cross-modal multi-scale fusion detection model to be trained according to the loss value.

8. The apparatus of claim 7, wherein, The first calculation module comprises an encoding unit and a first extraction unit; The encoding unit is configured to encode the image region corresponding to the target object to obtain a representation vector corresponding to the image region; The first extraction unit is configured to perform feature extraction on the representation vector corresponding to the image region to obtain the first image feature.

9. A detection device, characterized in that Comprising: an image acquisition module for acquiring a target image; an object recognition module for inputting the target image into a cross-modal multi-scale fusion detection model, so that the cross-modal multi-scale fusion detection model outputs a recognition result of at least one target object in the target image, the recognition result of each target object comprising a class label and position information of the target object, the cross-modal multi-scale fusion detection model being trained by the training method in any one of claims 1-5.

10. An electronic device, comprising: Comprising: a processor; a memory for storing executable instructions; wherein the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the cross-modal multi-scale fusion detection model training method in any one of claims 1-5 and the detection method in claim 6.

Images