Cross-modal target detection method, training method and related equipment

Abstract

The invention discloses a cross-modal target detection method, a training method and related equipment. The method comprises the steps of obtaining a to-be-detected image and a description text; performing feature extraction on the to-be-detected image and the description text to obtain image features and text features; performing matching processing on the image features and the text features by using the matching branches to obtain the category of the target; and performing positioning processing on the image features by using the positioning branches to obtain a positioning map of the target. According to the scheme, the accuracy of target detection can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of target detection, and in particular to a cross-modal target detection method, training method and related equipment. Background Art

[0002] As one of the most important tasks in the field of computer vision, object detection technology has achieved rapid development with the rapid development of deep learning methods.

[0003] At present, cross-modal object detection technology can be applied to object detection tasks. The cross-modal object detection task mainly combines multiple different modal information to perform object detection tasks. However, combining multiple different modal information may affect the accuracy of the task, resulting in low accuracy of object detection. Summary of the invention

[0004] The main technical problem solved by this application is to provide a cross-modal target detection method, training method and related equipment, which can improve the accuracy of target detection.

[0005] The first aspect of the present application provides a cross-modal target detection method, which includes: obtaining an image to be detected and a description text; performing feature extraction on the image to be detected and the description text respectively to obtain image features and text features; using a matching branch to match the image features and the text features to obtain the category of the target; and using a positioning branch to locate the image features to obtain a positioning map of the target.

[0006] A second aspect of the present application provides a method for training a target detection model, wherein the target detection model includes at least a matching branch and a positioning branch, and the method includes: obtaining an image sample to be detected and a description text sample; performing feature extraction on the image sample to be detected and the description text sample respectively to obtain image features and text features; using the matching branch to match the image features and the text features to obtain the category of the target, and obtaining an alignment loss value based on the category of the target; and using the positioning branch to position the image features to obtain a positioning map of the target, and obtaining a positioning loss value based on the positioning map of the target; and training the target detection model based on the alignment loss value and the positioning loss value to obtain a trained target detection model.

[0007] A third aspect of the present application provides a computer device, which includes a memory and a processor coupled to each other, wherein the memory stores program data, and the processor is used to execute the program data to implement any step of the above-mentioned cross-modal target detection method and the target detection model training method.

[0008] In a fourth aspect, the present application provides a computer-readable storage medium, which stores program data that can be executed by a processor, and the program data is used to implement any step of the above-mentioned cross-modal target detection method and target detection model training method.

[0009] The above scheme obtains the image to be detected and the description text, extracts features from the image to be detected and the description text respectively, and obtains image features and text features. Then, the matching branch and the positioning branch are explicitly separated. For the matching branch that requires strong semantic features, the matching branch is used to match the image features and text features to obtain the category of the target. The matching branch can fully perceive the semantic information of the text and image. A large amount of semantic information is added to the features of the matching branch, which is conducive to improving the accuracy of the matching task; while the positioning branch does not perceive the semantic features of the text. The positioning branch is used to position the image features to obtain the positioning map of the target, which can improve the accuracy of the positioning task and the accuracy of target detection as a whole.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in this application, the following briefly introduces the drawings required in the description of the embodiments. Obviously, the drawings described below are only some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them: Figure 1 It is a flowchart of an embodiment of a cross-modal target detection method of the present application; Figure 2 It is a structural diagram of an embodiment of the target detection model of the present application; Figure 3 This application Figure 1 A schematic diagram of a flow chart of an embodiment of step S14; Figure 4 It is a flowchart of the first embodiment of the training method of the target detection model of the present application; Figure 5 It is a structural schematic diagram of another embodiment of the target detection model of the present application; Figure 6 This application Figure 4 A schematic diagram of a flow chart of an embodiment of step S24; Figure 7 It is a flowchart of the second embodiment of the training method of the target detection model of the present application; Figure 8 This application Figure 7A schematic diagram of a flow chart of an embodiment of step S32; Fig. 9 is a structural schematic diagram of an embodiment of a cross-modal target detection device of the present application; Fig.10 It is a structural schematic diagram of an embodiment of a training device for a target detection model of the present application; Fig.11 It is a structural schematic diagram of an embodiment of a computer device of the present application; Fig.12 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0013] The terms "first" and "second" in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" can explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0014] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0015] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0016] The present application provides the following embodiments, and each embodiment is described in detail below.

[0017] It is understandable that the cross-modal target detection method and the target detection model training method in this application can be executed by a computer device, which can be any device with processing capabilities, such as a mobile device, a computer, a server, etc., and this application does not limit this. In some possible implementations, the cross-modal target detection method and the target detection model training method can also be implemented by a processor calling program data stored in a memory. This application does not limit the execution device.

[0018] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of a cross-modal target detection method of the present application. The method may include the following steps: S11: Obtain the image to be detected and the description text.

[0019] The cross-modal target detection of the present application can be implemented using a target detection model, where cross-modality can refer to the process of converting or associating information between different modalities. For example, visual modality, text modality and / or auditory modality, etc., the present application does not limit cross-modality. Optionally, the present application uses visual modality and text modality as examples for illustration.

[0020] An image to be detected and a description text for target detection may be obtained, wherein the description text is text content related to the image to be detected. For example, the description text is a description text of the image to be detected (such as target, background, color, feature, category, etc.). Optionally, the description text may include at least one character or word. This application does not limit the description text.

[0021] Optionally, the image to be detected may be a visible light image. The image to be detected may contain a target, for example, the target includes but is not limited to: an object, a living body, a moving object, a scene, a text, etc. This application does not limit the image to be detected.

[0022] For example, if the description text is "A dog holds a Frisbee", the image to be detected contains a dog and a Frisbee.

[0023] S12: Extract features of the image to be detected and the description text respectively to obtain image features and text features.

[0024] The object detection model may include a feature extraction network. The feature extraction network may be used to extract features of the image to be detected and the description text respectively to obtain image features and text features.

[0025] Optionally, an image feature extraction network can be used to extract features from the image to be detected to obtain image features. The image feature extraction network may include, but is not limited to: image encoders (such as Visual Encoder), convolutional neural networks (such as LeNet, ResNet, VGGNet, etc.), graph neural networks, etc. This application uses an image encoder as an example for illustration, and this application does not limit the image feature extraction network.

[0026] Optionally, a text feature extraction network can be used to extract features from the description text to obtain text features. The text feature extraction network may include, but is not limited to: a text encoder (such as a Text Encoder), a recurrent neural network, a convolutional neural network, a Transformer, etc. The following description of this application takes a text encoder as an example, and this application does not limit the text feature extraction network.

[0027] S13: Use the matching branch to match the image features and text features to obtain the category of the target.

[0028] See also Figure 2 The target detection model may include at least a matching branch and a positioning branch. The matching branch is used to obtain the category of the target. The positioning branch is used to obtain the positioning map of the target, and the positioning map may represent the positioning box / detection box / positioning coordinates of the target in the image to be detected.

[0029] The image features and text features are input into the matching branch, and the image features and text features are matched by the matching branch to obtain the category of the target. Optionally, a deep fusion module (such as Deep Fusion) can be used to deeply fuse the image features and text features to perform cross-modal fusion of the image features and text features to obtain cross-modal fused text fusion features and image fusion features. The text fusion features represent text features fused with image features, and the image fusion features represent image features fused with text features. Then, the matching module is used to perform feature matching on the text fusion features and the image fusion features to perform category recognition or matching to obtain the category of the target.

[0030] In some implementations, image features and text features may be deeply integrated. Specifically, the image features extracted above are and text features Afterwards, the image features can be and text features Perform multimodal fusion (such as Fusion) to obtain fusion features, such as text features fused with image features and image features fused with text features , to input the branch of text features and the branch of image features respectively.

[0031] Then, using text features (such as ) and fusion features ( ) to perform text encoding (such as text encoding by TextEncoder) and generate fused text fusion features . Using image features (such as ) and fusion features (such as ) to perform image encoding (such as using the image encoder DyHead Module to perform image encoding) and generate fused image fusion features .

[0032] Refer to the above method to perform multiple deep fusions, and perform the i-th (i is an integer greater than 1) text feature and fusion features Perform text encoding to obtain the final text fusion features . For the i-th image feature and fusion features ) to encode the image and obtain the final image fusion features Wherein, M and N are integers greater than 1. This application does not limit the number of deep fusions.

[0033] Through the last method, we can get the text fusion feature ( , , …, ) and image fusion features ( , , …, ), the text fusion feature can represent the features of N texts (such as word segmentation, etc.), and the image fusion feature can represent the features of M regions. By using the alignment network to perform feature matching on the text fusion feature and the image fusion feature respectively, each text and each region can be matched to obtain the matching similarity of each matching pair, thereby obtaining the alignment degree of each matching pair (such as Word-Region Alignment Score), where each matching pair can represent the category of a target. Matching pairs with an alignment degree greater than the preset alignment degree can be obtained to obtain the category of the target. Among them, the Word-Region Alignment Score is an indicator used to measure the degree of alignment between words in the text and regions in the image. Specifically, it evaluates the effect of alignment by calculating the similarity or matching degree between text phrases and features of different regions in the image.

[0034] S14: Use the positioning branch to perform positioning processing on the image features to obtain a positioning map of the target.

[0035] Only the image features are input into the positioning branch, and the image features are positioned to obtain the positioning map of the target. The positioning map is also the detection box / positioning box of the target in the image to be detected, which can represent the location information of the target.

[0036] In some embodiments, see Figure 3 , step S14 of the above embodiment can be further expanded. The positioning branch is used to perform positioning processing on the image features to obtain a positioning map of the target. This embodiment may include the following steps: S141: Perform transition processing on the image features to obtain transition features.

[0037] Continue reading Figure 2 , the positioning branch can include a positioning transition layer, a first positioning network, a dimension transformation network, a feature separation network and a second positioning network, etc.

[0038] Use the positioning transition layer to perform transition processing on image features to obtain transition features Among them, the transition characteristics , W, H, and C are the width, height, and number of channels of the feature, respectively. The positioning transition layer can be a neural network or a convolutional network, etc., and this application does not limit this.

[0039] Optionally, the image features may not be subjected to transition processing, that is, the image features may be referred to as transition features of this step, so that the following steps may be directly executed, and this application does not impose any limitation on this.

[0040] S142: Perform a first positioning on the transition feature to obtain a first positioning map.

[0041] The first localization network is used to localize the transition features For the first positioning, a rough border positioning can be performed to obtain the first positioning map , where N is the number of output channels of the first positioning. In some embodiments, the first positioning network performs a transition feature Perform the first positioning to obtain the first positioning map The process can be expressed as: .

[0042] in, Represents the first positioning network.

[0043] Since the first positioning map only retains positioning-related information in principle, in other words, for the first positioning map, features irrelevant to positioning will be gradually filtered out during the training process, so the first positioning map can be used as a positioning feature filter factor (eg, can represent a relatively pure positioning feature filter factor).

[0044] S143: Using the first positioning map to perform feature separation on the transition feature to obtain a separation feature.

[0045] The first positioning map can be used Transition characteristics Perform feature separation to separate transition features The features that are not related to positioning are used to obtain separation features This process can make it possible to locate the transition features of the transition layer The positioning-irrelevant features in the image are filtered out to a certain extent, and what remains is mostly positioning-aware feature information, making subsequent positioning more accurate.

[0046] In some embodiments, the first positioning map can be dimensionally transformed through a dimension transformation network to transform the first positioning map into a feature with the same dimension as the transition feature to obtain a feature separation factor Among them, the dimension of the feature separation factor is related to the transition feature The dimensions are the same.

[0047] Optionally, the dimension transformation network may include: a network of an anti-prediction layer + an activation function (such as a sigmoid activation function), which is not limited in this application.

[0048] Exemplarily, for the first positioning map Perform dimension transformation to obtain feature separation factor The process can be expressed as follows: .

[0049] in, It represents the anti-prediction layer, which can be composed of two convolutional layers. After the anti-prediction layer convolution processing, it is processed by the sigmoid activation function to obtain the feature separation factor , which is a positioning feature filtering separation factor. Since the first positioning map The positioning-irrelevant features in the image are gradually filtered out, so the feature filter factors generated by the anti-prediction layer in principle only contain positioning-related activation information.

[0050] Then, a feature separation network is used to utilize the feature separation factor , for transition characteristics Perform feature separation (such as multiplication) to obtain separated features .

[0051] Optionally, a feature separation factor can be used And the learnable bias term bias, for transition features By performing feature separation, we can obtain the separation features .

[0052] Exemplary, separation features It can be expressed by the following formula: .

[0053] in, It is expressed as pixel-by-pixel multiplication, and bias is a learnable bias term. Re-separation, positioning the transition characteristics of the transition layer The positioning-irrelevant features in the image are filtered out to a certain extent, that is, what remains is mostly positioning-aware feature information.

[0054] S144: Perform a second positioning on the separated features to obtain a second positioning map, wherein the second positioning map serves as a positioning map of the target in the image to be detected.

[0055] After separating the features that are not related to positioning, a second positioning network (such as a refined coordinate regressor) is used to perform a second positioning on the separated features, that is, coordinate regression can be performed to obtain a second positioning map with more accurate positioning. Among them, the second positioning map It can be used as the final positioning map, that is, the second positioning map can be used as the positioning map of the target in the image to be detected.

[0056] Thus, the target category of the matching branch and the positioning map of the target of the positioning branch are comprehensively matched to obtain a cross-modal target detection result.

[0057] The above scheme obtains the image to be detected and the description text, extracts features from the image to be detected and the description text respectively, and obtains image features and text features. Then, the matching branch and the positioning branch are explicitly separated. For the matching branch that requires strong semantic features, the matching branch is used to match the image features and text features to obtain the category of the target. The matching branch can fully perceive the semantic information of the text and image. A large amount of semantic information is added to the features of the matching branch, which is conducive to improving the accuracy of the matching task. The positioning branch does not perceive the semantic features of the text, but requires more detailed edge features. The positioning branch is used to position the image features to obtain the positioning map of the target, which can improve the accuracy of the positioning task and the accuracy of target detection as a whole.

[0058] In some embodiments, the above-mentioned object detection model may be trained so that the trained object detection model performs cross-modal object detection. The training process of the object detection model may refer to the following embodiments.

[0059] See also Figure 4 , Figure 4 1 is a flow chart of the first embodiment of the training method of the target detection model of the present application. The method may include the following steps: S21: Obtain image samples to be detected and description text samples.

[0060] A number of to-be-detected image samples and description text samples may be obtained. The to-be-detected image samples and description text samples may refer to the description of the above embodiment and will not be described in detail here.

[0061] S22: Extract features from the image sample to be detected and the description text sample respectively to obtain image features and text features.

[0062] See also Figure 5 , the target detection model can include a feature extraction network (such as a text encoder Text Encoder, an image encoder Visual Encoder), a matching branch, and a positioning branch.

[0063] A text encoder can be used to extract features from the description text sample to obtain text features, and an image encoder can be used to extract features from the image sample to be detected to obtain image features.

[0064] S23: Use the matching branch to match the image features and the text features to obtain the category of the target, and obtain the alignment loss value based on the category of the target.

[0065] The image features and text features are input into the matching branch to match the image features and text features using the matching branch to obtain the category of the target. Then, the loss is calculated based on the category of the target to obtain the alignment loss value of the matching branch.

[0066] In some implementations, the image features and text features are deeply fused using a matching branch to obtain cross-modal fused text fusion features and image fusion features. Then, feature matching is performed on the text fusion features and the image fusion features to obtain the alignment degree of each category of the target. Then, the alignment loss value can be obtained according to the alignment degree.

[0067] S24: Use the positioning branch to perform positioning processing on the image features to obtain a positioning map of the target, and obtain a positioning loss value based on the positioning map of the target.

[0068] Only the image features are input into the positioning branch, and the positioning branch is used to perform positioning processing on the image features to obtain the positioning map of the target. Then, the loss is calculated based on the positioning map of the target to obtain the positioning loss value of the positioning branch.

[0069] In some embodiments, see Figure 6 , step S24 of the above embodiment can be further expanded. The positioning branch is used to perform positioning processing on the image features to obtain a positioning map of the target, and based on the positioning map of the target, a positioning loss value is obtained. This embodiment may include the following steps: S241: Perform transition processing on the image features to obtain transition features.

[0070] Continue reading Figure 5 , the positioning branch can include a positioning transition layer, a first positioning network, a dimension transformation network, a feature separation network and a second positioning network, etc.

[0071] In this step, the image features are processed using the positioning transition layer to obtain transition features .

[0072] S242: Perform a first positioning on the transition feature to obtain a first positioning map; and obtain a first positioning loss value using the first positioning map.

[0073] The first localization network is used to localize the transition features Perform the first positioning and obtain the first positioning map Using the first positioning map By performing loss calculation, a first positioning loss value can be obtained. The first positioning loss value can be used to supervise the learning process. Since the first positioning map is supervised by the label corresponding to the positioning frame, in principle, only positioning-related information is retained in the first positioning map. For the first positioning map, features not related to positioning will be gradually filtered out during the training process.

[0074] S243: Using the first positioning map to perform feature separation on the transition feature to obtain a separated feature.

[0075] Optionally, the feature separation network uses the first positioning map to separate the transition features to obtain separation features

[0076] Optionally, a dimension transformation network is used to transform the first positioning map Perform dimension transformation to obtain feature separation factor Among them, the feature separation factor Dimensions and transition characteristics The dimension of is the same as that of . Then, the feature separation network uses the feature separation factor , for the transition characteristics Perform feature separation to obtain separation features .

[0077] S244: Perform a second positioning on the separation feature to obtain a second positioning map; and obtain a second positioning loss value using the second positioning map.

[0078] The second localization network is used to separate features Perform the second positioning and obtain the second positioning map .

[0079] Using the second positioning map By calculating the loss, a second positioning loss value can be obtained, so that the second positioning loss value can be used to supervise the learning process.

[0080] S245: Obtain a self-distillation loss value using the transition characteristics and the separation characteristics.

[0081] In some embodiments, the self-distillation loss value may also be obtained. Specifically, the transition feature may be used and separation features , and obtain the self-distillation loss value. In this way, the separation characteristics can be achieved While achieving separation, it facilitates the positioning of transition features of the transition layer It can also be separated as much as possible, so that the features sent to the second positioning network can be better separated and filtered for features irrelevant to positioning.

[0082] In some embodiments, the transition characteristics are obtained and separation features The pixel similarity between each pixel point. In response to the pixel similarity being less than the preset similarity, a pixel loss value is determined based on the pixel similarity of the pixel point; otherwise, the pixel loss value of the pixel point is determined to be zero. Then, the pixel loss values ​​of each pixel point can be combined to obtain the transition feature and separation features The corresponding self-distillation loss value.

[0083] Optionally, the cosine similarity or cosine distance between the pixels may be used to represent the pixel similarity, wherein the smaller the cosine distance, the more similar the two pixels are in direction; and the larger the cosine distance, the less similar the two pixels are in direction.

[0084] Exemplary, transitional features and separation features Pixel loss value between each pixel (i, j) It can be expressed as: .

[0085] Among them, (i, j) represents a pixel point at the coordinate position (i, j) on the feature map. Represents separation features A pixel on Represents transition characteristics A pixel on top. Represents a preset parameter, which is also a control parameter. When the cosine distance between two features is less than or equal to the preset parameter When (that is, the pixel similarity is not less than the preset similarity), it is considered that the two features are very close at this pixel point and there is no need to shorten the distance between them.

[0086] For example, the pixel loss value of each pixel point (i, j) is integrated , the final self-distillation loss value can be obtained ,as follows: .

[0087] Among them, W and H represent the width and height of the feature respectively.

[0088] S246: Obtain a positioning loss value by integrating the first positioning loss value, the second positioning loss value and / or the self-distillation loss value.

[0089] Optionally, the first positioning loss value and the second positioning loss value may be combined to obtain the positioning loss value of the positioning branch.

[0090] Optionally, the first positioning loss value, the second positioning loss value and the self-distillation loss value may be integrated to obtain the positioning loss value of the positioning branch.

[0091] In some embodiments, during training, in order to avoid separating features Transition Features Reverse influence, resulting in separation characteristics The feature separation is not obvious. That is, the generated filter factors may be too trivial (the value of each pixel and each channel is very close to 1), so the feature separation Before filtering out completely irrelevant features, they degenerate into transition features. are very similar, thus losing the function of feature separation. To this end, this application also adopts the following two training strategies.

[0092] In some embodiments, for the self-distillation loss value , during the training process, the separation features Gradient isolation is performed. That is, when the self-distillation loss value is returned, it is only returned to the transition feature. , therefore, the self-distillation loss value will not affect the transition characteristics Causes influence, thus avoiding separation features Transition Features Reverse impact.

[0093] In some implementations, the matching branch and the positioning branch, or the first positioning map in the positioning branch, can be respectively and the second positioning map Use two positioning loss values, and use different positive samples to obtain positioning loss values ​​to supervise the training process of the two regressions. In this way, avoid the first positioning map Become a trivial (invalid) prediction positioning map, thereby avoiding the generated filter factors from being too trivial.

[0094] Continue reading Figure 4 After the above step S24, the following steps may also be included: S25: Comprehensively train the target detection model based on the alignment loss value and the positioning loss value to obtain a trained target detection model.

[0095] The target detection model is trained based on the alignment loss value of the matching branch and the positioning loss value of the positioning branch, and the network parameters of each branch of the target detection model are adjusted to obtain the trained target detection model.

[0096] For the above scheme, the present application introduces the idea of ​​feature separation into cross-modal target detection, explicitly separates the matching branch and the positioning branch, and generates different features for the two branches respectively. For the matching branch that requires strong semantic features, the connection between the two modalities is increased through the deep fusion module, and a text-image fusion feature rich in semantic information is generated and input into the matching module for matching. For the positioning task, semantic information is not required but only edge texture information is required. There is also a positioning branch for feature self-distillation. The feature self-distillation method is used to gradually separate and filter the features that are not related to positioning, generate positioning features rich in detailed edge information, and enhance the accuracy of positioning. Through the idea of ​​feature separation, different branches can perform their tasks separately and generate task-aware features respectively, so as to achieve task separation, reduce mutual interference between irrelevant features, and improve network effects.

[0097] For step S24 of the above embodiment, corresponding training positive samples may be used to obtain loss values ​​for different branches, or for the first positioning map and the second positioning map in the positioning branch.

[0098] See also Figure 7 , Figure 7 1 is a flow chart of the second embodiment of the training method of the target detection model of the present application. The method may include the following steps: S31: Obtain candidate positive samples.

[0099] Candidate positive samples can be obtained based on the image samples to be detected.

[0100] Optionally, an anchor box may be set on each pixel of the image sample to be detected, and candidate boxes may be obtained, and all candidate boxes may be used as candidate positive samples.

[0101] Optionally, partitions may be randomly selected from the image sample to be detected, each partition is used as a candidate frame, and all candidate frames are used as candidate positive samples. The present application does not limit the method of obtaining candidate positive samples.

[0102] S32: Perform task separation sampling on the candidate positive samples to determine the alignment positive samples selected by the matching branch and the positioning positive samples selected by the positioning branch; wherein the categories of the alignment positive samples and the target are used to obtain the alignment loss value, and the positioning positive samples and the positioning map of the target are used to obtain the positioning loss value.

[0103] Continue reading Figure 5 In the training process of the target detection model, a task separation sampling module is also included. The task separation sampling module is used to perform task separation sampling on candidate positive samples to respectively determine the alignment positive samples selected by the matching branch and the positioning positive samples selected by the positioning branch. Among them, the categories of the alignment positive samples and the targets are used to obtain the alignment loss value, and the positioning positive samples and the positioning maps of the targets are used to obtain the positioning loss value.

[0104] In some embodiments, see Figure 8 , step S32 may include the following steps: S321: Obtain the matching degree between the candidate positive sample and the real label frame to obtain a reference matching value.

[0105] Each candidate positive sample can be matched with the true label frame to obtain the matching degree of each candidate positive sample with the true label frame, and obtain the reference matching degree. The matching degree can be expressed by ‌IOU (Intersection over Union), and the training positive samples of each branch can be selected using the ‌IOU selection strategy.

[0106] For example, the matching degree (‌IOU) of each candidate positive sample and the real label box is obtained, and the reference matching value IoU corresponding to each candidate positive sample is obtained. d .

[0107] S322: Determine the aligned positive sample selected by the matching branch from the candidate positive samples by using the reference matching value and the category similarity of the category.

[0108] For the matching branch, the reference matching value and the category similarity of the category can be used to determine the aligned positive sample selected by the matching branch from the candidate positive samples. Among them, the category similarity of the category is obtained based on the matching similarity of the category corresponding to the true label box.

[0109] In some embodiments, the matching similarity can be obtained for the image fusion features and text fusion features at all pixels to obtain a corresponding similarity matrix. Each matching similarity can correspond to a category. Then, according to the real label box corresponding to each matching similarity, the matching similarity of the category corresponding to the real label box is selected to obtain the category similarity of each category. The category similarity S is expressed by the following formula: .

[0110] in, (i is less than or equal to N) represents text fusion features, (j is less than or equal to M) represents the image fusion feature.

[0111] In some implementations, the reference matching value IoU can be used d The category similarity S of the category is used to obtain the matching sample evaluation value of the matching branch. Specifically, the first weight parameter can be used , control the reference matching value IoU d and the contribution between the category similarity S. Therefore, the matching sample evaluation value of the matching branch It can be expressed as: .

[0112] Then, select the matching sample evaluation value The candidate positive samples that meet the matching selection conditions are used to obtain the aligned positive samples. The matching selection conditions include: matching sample evaluation value Greater than the first threshold , and the first number before sorting is K (K is an integer). That is, the evaluation value of the matching sample can be Greater than the first threshold Among all candidate samples, select the first K candidate positive samples to obtain the aligned positive samples of the matching branch , which can be expressed as follows: .

[0113] in, Indicates the matching selection condition, that is, matching sample evaluation value Greater than the first threshold Among all candidate samples, select the first K candidate positive samples as the alignment positive samples.

[0114] Optionally, the matching branch can use the aligned positive samples and the corresponding categories to obtain the alignment loss value.

[0115] S323: Determine the positioning positive sample selected by the positioning branch from the candidate positive samples by using the reference matching value and the positioning similarity of the positioning map.

[0116] For the positioning branch, the reference matching value IoU can be used d The positioning similarity of the positioning map is used to determine the positioning positive sample selected by the positioning branch from the candidate positive samples. Among them, the positioning similarity of the positioning map is obtained based on the positioning map and the true label box.

[0117] Optionally, the positioning map includes a first positioning map and / or a second positioning map. Correspondingly, the first positioning similarity can be obtained based on the similarity between the first positioning map and the corresponding true label frame. The second positioning similarity can be obtained based on the similarity between the second positioning map and the corresponding true label frame. In this way, positioning positive samples can be selected according to different positioning maps or any of the positioning maps. As the positioning features are gradually separated, the positive samples of the first positioning and the second positioning are also different.

[0118] In some embodiments, taking the selection of positive samples for the first positioning map and / or the second positioning map as an example, the positioning positive samples include the first positioning positive samples and / or the second positioning positive samples, wherein the first positioning positive samples and the first positioning map are used to obtain the first positioning loss value, and the second positioning positive samples and the second positioning map are used to obtain the second positioning loss value.

[0119] In some implementations, for the first positioning map, the reference matching value IoU d The first positioning similarity with the first positioning graph is used to obtain the first positioning evaluation value of the positioning branch. Specifically, the second trade-off parameter , control the reference matching value IoU d The contribution between the similarity of the first positioning and the first positioning. Therefore, the first positioning evaluation value of the positioning branch is It can be expressed as: .

[0120] in, Represents the second trade-off parameter, controlling the reference matching value IoU d Similarity with the first position Contribution of the first positioning similarity Represents the similarity (such as IOU) between the first positioning map and the true label box.

[0121] Then, a candidate positive sample whose first positioning evaluation value satisfies the first positioning selection condition is selected to obtain a first positioning positive sample. The first positioning selection condition includes: Greater than the second threshold and ranked first by the second quantity.

[0122] Exemplarily, the first positioning positive sample It can be expressed as: .

[0123] in, Indicates the first positioning selection condition, that is, the first positioning evaluation value Greater than the second threshold (such as ) among all candidate samples, select the first second number (such as K) candidate positive samples as the first positioning positive samples.

[0124] In some implementations, for the second positioning map, the reference matching value IoU d , the first positioning similarity of the first positioning graph and the second positioning similarity of the second positioning graph, to obtain the second positioning evaluation value of the positioning branch. Specifically, the first weight parameter can be used to control the reference matching value IoU d The contribution of the first positioning similarity is controlled by the second weight parameter, and the first weight parameter and the second weight parameter are used to control the second positioning similarity and the reference matching value IoU d , and the first positioning similarity to obtain a second positioning evaluation value of the positioning branch.

[0125] Exemplarily, the second positioning evaluation value of the positioning branch It can be expressed as follows: .

[0126] in, and Respectively represent the first weight parameter and the second weight parameter, both of which are weight parameters less than 1. The first positioning similarity Represents the similarity between the first positioning map and the true label box (such as IOU). Second positioning similarity Represents the similarity (such as IOU) between the second positioning map and the true label box.

[0127] Then, the candidate positive sample whose second positioning evaluation value satisfies the second positioning selection condition is selected to obtain the second positioning positive sample. The second positioning selection condition includes: the second positioning evaluation value Greater than the third threshold and ranked third quantity.

[0128] Exemplarily, the second positioning positive sample It can be expressed as: .

[0129] in, Indicates the second positioning selection condition, that is, the second positioning evaluation value Greater than the third threshold (such as ) among all candidate samples, select the first third number (such as K) candidate positive samples as the second positioning positive samples.

[0130] Optionally, the first threshold, the second threshold and the third threshold referred to above may be different thresholds or the same thresholds, and the first quantity, the second quantity and the third quantity may be different quantities or the same quantity. This application takes the same threshold and the same quantity as an example for illustration, and this application does not impose any restrictions on this.

[0131] In order to select training positive samples that are more suitable for matching tasks and positioning tasks respectively in the training optimization stage, the above scheme can select different training positive samples for the matching branch and the first and second positioning of the positioning branch to obtain corresponding loss values ​​through the above task separation sampling strategy, so as to differentiate the two branches in the parameter learning process and realize the separation of matching features and positioning features. In this way, the mutual influence between the two features in the cross-modal detection task is reduced during the reasoning process of the network, and the corresponding pixel points can be accurately responded to, thereby improving the performance of the network.

[0132] In addition, the task separation sampling module can select training positive samples that are more suitable for matching tasks and positioning tasks respectively during the training optimization stage. In the matching branch, matching-aware positive samples are selected according to the matching task characteristics to enhance the learning of semantic features of the matching branch. In the positioning branch, appropriate positive samples are allocated to the positioning task in two steps according to the progress of the positioning task to supervise the learning of details and edge features. Through the task separation sampling module, the separation of the features of the two branches is supervised from the perspective of network parameter learning, so that task-aware features can be learned in the two branches respectively.

[0133] It is understandable that in the above method of the specific implementation mode, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0134] In some embodiments, the present application also provides a cross-modal target detection device for implementing the cross-modal target detection method of any of the above embodiments.

[0135] See also Fig. 9 , Fig. 9 4 is a schematic diagram of a cross-modal target detection device according to an embodiment of the present invention. The cross-modal target detection device 40 includes an acquisition module 41, a feature extraction module 42, a matching branch module 43 and a positioning branch module 44. The modules are interconnected.

[0136] The acquisition module 41 is used to acquire the image to be detected and the description text.

[0137] The feature extraction module 42 is used to extract features from the image to be detected and the description text respectively, to obtain image features and text features.

[0138] The matching branch module 43 is used to use the matching branch to match the image features and the text features to obtain the category of the target.

[0139] The positioning branch module 44 is used to perform positioning processing on the image features using the positioning branch to obtain a positioning map of the target.

[0140] In some embodiments, the present application also provides a target detection model training device for implementing the target detection model training method of any of the above embodiments.

[0141] See also Fig.10 , Fig.101 is a schematic diagram of the structure of an embodiment of a training device for a target detection model of the present application. The training device 50 for the target detection model includes a sample acquisition module 51, a feature extraction module 52, an alignment loss module 53, a positioning loss module 54 and a training module 55. The modules are interconnected. The target detection model includes at least a matching branch and a positioning branch.

[0142] The sample acquisition module 51 is used to acquire the image samples to be detected and the description text samples; The feature extraction module 52 is used to extract features from the image samples to be detected and the description text samples respectively, to obtain image features and text features.

[0143] The alignment loss module 53 is used to use the matching branch to match the image features and the text features to obtain the category of the target, and obtain the alignment loss value based on the category of the target.

[0144] The positioning loss module 54 is used to perform positioning processing on the image features using the positioning branch to obtain a positioning map of the target, and obtain a positioning loss value based on the positioning map of the target.

[0145] The training module 55 is used to comprehensively consider the alignment loss value and the positioning loss value, train the target detection model, and obtain a trained target detection model.

[0146] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this application does not limit this.

[0147] For the above embodiment, the present application provides a computer device, see Fig.11 , Fig.11 1 is a schematic diagram of the structure of an embodiment of a computer device of the present application. The computer device 60 includes a memory 61 and a processor 62, wherein the memory 61 and the processor 62 are coupled to each other, the memory 61 stores program data, and the processor 62 is used to execute the program data to implement the steps of any embodiment of the cross-modal target detection method and the target detection model training method.

[0148] In this embodiment, the processor 62 may also be referred to as a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip having the ability to process signals. The processor 62 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or the processor 62 may also be any conventional processor, etc.

[0149] The method of the above embodiment can be implemented in the form of a computer program, so the present application proposes a computer readable storage medium, see Fig.12 , Fig.12 1 is a schematic diagram of a computer-readable storage medium of the present application. The computer-readable storage medium 70 stores program data 71 that can be run by a processor, and the program data 71 can be executed by the processor to implement the steps of any embodiment of the cross-modal target detection method and the target detection model training method.

[0150] The computer-readable storage medium 70 of this embodiment may be a medium that can store the program data 71, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or may be a server storing the program data 71. The server may send the stored program data 71 to other devices for execution, or may execute the stored program data 71 by itself.

[0151] In some embodiments, the functions or modules included in the device provided in the above embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, the present application will not go into details here.

[0152] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other. For the sake of brevity, this application will not go into details here.

[0153] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0154] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0155] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0156] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium, which is a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of each implementation method of the present application.

[0157] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a computer-readable storage medium and executed by a computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the present application is not limited to any specific combination of hardware and software.

[0158] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A cross-modal target detection method, characterized in that: include: Get the image to be detected and the description text; Extracting features of the image to be detected and the description text respectively to obtain image features and text features; Using a matching branch to match the image feature with the text feature to obtain a category of a target; and The positioning branch is used to perform positioning processing on the image features to obtain a positioning map of the target.

2. The method according to claim 1, characterized in that The method of using the positioning branch to perform positioning processing on the image features to obtain a positioning map of the target includes: Performing transition processing on the image features to obtain transition features; Performing a first positioning on the transition feature to obtain a first positioning map; Using the first positioning map to perform feature separation on the transition feature to obtain a separation feature; Performing a second positioning on the separation feature to obtain a second positioning map; The second positioning map is used as a positioning map of the target in the image to be detected.

3. The method according to claim 2, characterized in that The step of performing feature separation on the transition feature by using the first positioning map to obtain separated features includes: Performing a dimension transformation on the first positioning map to obtain a feature separation factor; wherein the dimension of the feature separation factor is the same as the dimension of the transition feature; The feature separation factor is used to perform feature separation on the transition feature to obtain the separation feature.

4. The method according to claim 1, characterized in that: The using a matching branch to match the image feature and the text feature to obtain a target category includes: Deeply fusing the image features and the text features to obtain cross-modal fused text fusion features and image fusion features; Feature matching is performed on the text fusion feature and the image fusion feature to obtain the category of the target.

5. A method for training a target detection model, characterized in that: The target detection model includes at least a matching branch and a positioning branch, and the training method includes: Obtain image samples to be detected and description text samples; Extracting features of the image sample to be detected and the description text sample respectively to obtain image features and text features; Using a matching branch to match the image feature and the text feature to obtain a category of the target, and based on the category of the target, obtaining an alignment loss value; and Using the positioning branch to perform positioning processing on the image features to obtain a positioning map of the target, and based on the positioning map of the target, obtaining a positioning loss value; The object detection model is trained based on the alignment loss value and the positioning loss value to obtain a trained object detection model.

6. The method according to claim 5, characterized in that The method of using the positioning branch to perform positioning processing on the image features to obtain a positioning map of the target, and obtaining a positioning loss value based on the positioning map of the target includes: Performing transition processing on the image features to obtain transition features; Performing a first positioning on the transition feature to obtain a first positioning map; and obtaining a first positioning loss value using the first positioning map; Using the first positioning map to perform feature separation on the transition feature to obtain a separation feature; Performing a second positioning on the separation feature to obtain a second positioning map; using the second positioning map to obtain a second positioning loss value; and / or Using the transition characteristic and the separation characteristic, a self-distillation loss value is obtained; The positioning loss value is obtained by integrating the first positioning loss value, the second positioning loss value and / or the self-distillation loss value.

7. The method according to claim 5, characterized in that Also includes: Obtain candidate positive samples; Performing task separation sampling on the candidate positive samples to determine the alignment positive samples selected by the matching branch and the positioning positive samples selected by the positioning branch; The alignment positive sample and the category of the target are used to obtain an alignment loss value, and the positioning positive sample and the positioning map of the target are used to obtain a positioning loss value.

8. The method according to claim 7, characterized in that The performing task separation sampling on the candidate positive samples to determine the alignment positive samples selected by the matching branch and the positioning positive samples selected by the positioning branch includes: Obtaining the matching degree between the candidate positive sample and the real label frame to obtain a reference matching value; Determining the aligned positive sample selected by the matching branch from the candidate positive samples by using the reference matching value and the category similarity of the category; Determine the positioning positive sample selected by the positioning branch from the candidate positive samples by using the positioning similarity of the reference matching value and the positioning map; The category similarity of the category is obtained based on the matching similarity of the category corresponding to the real label frame; and the positioning similarity of the positioning map is obtained based on the positioning map and the real label frame.

9. A computer device, characterized in that: The invention comprises a memory and a processor coupled to each other, wherein the memory stores program data, and the processor is used to execute the program data to implement the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: Program data that can be executed by a processor are stored, and the program data is used to implement the steps of the method according to any one of claims 1 to 8.

Images