Image-text retrieval method, electronic device and storage medium

Abstract

The application discloses a picture-text retrieval method and an electronic device and a storage medium. The method comprises the following steps: analyzing a to-be-retrieved text to determine a main target and a sub-target in the to-be-retrieved text, wherein the sub-target is a conditional target of the main target; performing main target detection on a to-be-retrieved image to obtain a main target region corresponding to the main target in the to-be-retrieved image; determining an associated region of the sub-target in the to-be-retrieved image based on the main target region; performing sub-target detection on the associated region to obtain a sub-target region corresponding to the sub-target; and determining whether the to-be-retrieved image is a retrieval result of the to-be-retrieved text based on the association between the main target region and the sub-target region. The above scheme can improve the accuracy of the overall retrieval result.

Description

Technical Field

[0001] This application relates to the field of image and text retrieval technology, and in particular to an image and text retrieval method, electronic device, and storage medium. Background Technology

[0002] To broaden its application, current multimodal image-text retrieval methods need to support image retrieval of more complex text. Complex text typically includes not only the main target but also sub-targets that are attributes of the main target. Sub-targets in images are usually small in scale, cover a smaller area, and are easily mixed with background information, making them more complex than conventional image-text retrieval. Current image-text retrieval methods suffer from low overall accuracy when dealing with small sub-targets. Summary of the Invention

[0003] This application provides at least one image and text retrieval method, electronic device, and storage medium, which can improve the accuracy of the overall retrieval results when the sub-targets are small.

[0004] The first aspect of this application provides a text-image retrieval method, which includes: parsing the text to be retrieved to determine the main target and sub-targets in the text, wherein the sub-targets are conditional targets of the main target; performing main target detection on the image to be retrieved to obtain the main target region corresponding to the main target in the image to be retrieved; determining the associated region of the sub-target in the image to be retrieved based on the main target region; performing sub-target detection on the associated region to obtain the sub-target region corresponding to the sub-target; and determining whether the image to be retrieved is a retrieval result of the text to be retrieved based on the correlation between the main target region and the sub-target region.

[0005] Among them, determining the associated regions of sub-targets in the image to be retrieved based on the main target region includes: adjusting the main target region to obtain the associated regions, wherein the adjustment method is to expand or shrink.

[0006] Among them, at least one of the adjustment method and the adjustment range of the main target area is determined based on the type of the sub-target; and / or, adjusting the main target area to obtain the associated area includes: determining the size adjustment weight based on the type of the sub-target; adjusting the size parameters corresponding to the main target area using the size adjustment weight to obtain the adjustment range of the main target area; and adjusting the main target area according to the adjustment range to obtain the associated area.

[0007] The process of detecting sub-targets in associated regions to obtain sub-target regions includes: extracting semantic features and detail features from associated regions to obtain semantic feature maps and detail feature maps; generating a spatial attention weight map using the semantic feature maps and detail feature maps; and weighting the detail feature maps using the spatial attention weight map to obtain sub-target regions in associated regions.

[0008] Specifically, the spatial attention weight map is generated using semantic feature maps and detail feature maps, including: concatenating semantic features and edge features to obtain concatenated features; convolving the concatenated features to obtain convolutional features; and activating the convolutional features to obtain the spatial attention weight map.

[0009] Before determining whether the image to be retrieved should be used as the retrieval result of the text to be retrieved based on the correlation between the main target region and the sub-target region, the process also includes: calculating the spatial coverage of the main target region to the sub-target region; determining the positional density between the main target region and the sub-target region; and determining the correlation based on the spatial coverage and positional density.

[0010] The calculation of the spatial coverage of the main target region to the sub-target region includes: obtaining a first area of ​​the intersection between the main target region and the sub-target region, and a second area of ​​the sub-target region, and using the ratio between the first area and the second area as the spatial coverage; and / or determining the positional density between the main target region and the sub-target region includes: obtaining the distance between the center point of the main target region and the center point of the sub-target region; calculating the positional density based on the distance, wherein the positional density is negatively correlated with the distance; and / or determining the correlation based on the spatial coverage and the positional density includes: using the product of the spatial coverage and the positional density as the correlation.

[0011] The process includes parsing the text to be retrieved to determine the main target and sub-targets, including: splitting the text to obtain main target keywords and sub-target keywords; identifying the main target keywords and sub-target keywords to obtain the main target and sub-targets respectively; and / or performing main target detection on the image to be retrieved to obtain the main target region corresponding to the main target in the image to be retrieved, including: performing main target detection on the image to be retrieved to obtain at least one candidate main target region and corresponding target category confidence in the image to be retrieved, wherein the target category confidence represents the confidence that the candidate main target region is the main target; determining the candidate main target region whose target category confidence meets the preset confidence requirement as the main target region; and / or determining whether the image to be retrieved is a retrieval result of the text to be retrieved based on the correlation between the main target region and the sub-target region, including: determining the image to be retrieved as a retrieval result of the text to be retrieved in response to the correlation being greater than a preset correlation threshold.

[0012] The second aspect of this application provides an electronic device including a memory and a processor coupled to each other, the processor being used to execute program instructions stored in the memory to implement the image and text retrieval method of the first aspect described above.

[0013] A third aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the image and text retrieval method described in the first aspect above.

[0014] The above scheme first parses the text to be retrieved to determine the main target and sub-targets. Then, it performs main target detection on the image to be retrieved, obtaining the main target region corresponding to the main target in the image. Based on the relationship between the main target and sub-targets, it determines the associated region in the main target region to reduce invalid or interfering regions in the sub-target detection process. Then, it performs sub-target detection on the associated region to obtain the sub-target region corresponding to the sub-target, thereby improving the accuracy of sub-target detection. Furthermore, based on the correlation between the main target region and the sub-target region, it determines whether the image to be retrieved should be used as a retrieval result of the text to be retrieved, thereby improving the accuracy of the overall retrieval results.

[0015] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0017] Figure 1 This is a flowchart illustrating an embodiment of the image and text retrieval method of this application; Figure 2 This is a flowchart illustrating another embodiment of the image and text retrieval method of this application; Figure 3 This is a schematic diagram of the framework of an embodiment of the image and text retrieval device of this application; Figure 4 This is a schematic diagram of the framework of an embodiment of the image and text retrieval model of this application; Figure 5 This is a schematic diagram of the framework of an embodiment of the electronic device of this application; Figure 6 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0018] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0019] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0020] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0021] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the image and text retrieval method of this application. Specifically, it may include the following steps: Step S110: Parse the text to be retrieved to determine the main target and sub-targets in the text.

[0022] Among them, the sub-objectives are the conditional objectives of the main objective.

[0023] This application is primarily applied to the fields of artificial intelligence and large-scale multimodal models, especially multimodal image and text retrieval. The execution entity in this application is an image and text retrieval model, such as the Chinese-CLIP (Chinese Contrastive Language-Image Pre-training) model and the LLaVA (Large Language and Vision Assistant) model, etc., without specific limitations. This application extracts the main target and sub-targets by parsing the text to be retrieved. The main target is detected and located in the main body region, and the related regions generate the positions of the focused sub-targets. Sub-target detection is performed in the related regions. Finally, the logical connection between the main target region and the sub-target region is verified to improve the sub-target detection rate, thereby improving the overall image and text retrieval accuracy.

[0024] After receiving the text input by the user, the image-text retrieval model analyzes the text to determine the main target and sub-targets to be retrieved. Specifically, the image-text retrieval model can segment the text to obtain main target keywords and sub-target keywords. For example, the jieba word segmentation library can be used to segment the text into independent keyword sequences. Then, the main target keywords and sub-target keywords are identified separately to obtain the main target and sub-target keywords. For example, if the text to be searched is "smokers", the image-text retrieval model is used to break down "smokers" into words, and the main target keyword is "people" and the sub-target keyword is "smoking". Then, "people" and "smoking" are identified, and the main target is "people" and the sub-target is "cigarettes".

[0025] For example, if the text to be searched is "a puppy crossing a zebra crossing", the image-text retrieval model is used to break down "a puppy crossing a zebra crossing" into words, and the main target keyword is "puppy" and the sub-target keyword is "crossing a zebra crossing". Then, "puppy" and "crossing a zebra crossing" are identified, and the main target is "dog" and the sub-target is "zebra crossing".

[0026] For example, if the text to be searched is "a man wearing a black hat and looking down at his phone", the image-text retrieval model is used to break down "a man wearing a black hat and looking down at his phone" into words. The main target keyword is "man", and the sub-target keywords are "wearing a black hat" and "playing on his phone". Then, "man", "wearing a black hat" and "playing on his phone" are identified, and the main target is "man" and the sub-targets are "black hat" and "phone".

[0027] This embodiment improves the accuracy of text parsing by using word segmentation and keyword recognition to extract the main target and sub-targets more accurately, avoiding parsing errors caused by ambiguous text expression.

[0028] As can be seen from the above, there is a relationship between the main objective and the sub-objectives. The number of sub-objectives corresponding to the main objective can be one or more, and no specific limit is made here.

[0029] In addition, before performing image-text retrieval, the image-text retrieval model needs to collect images as the images to be retrieved. The collected images can be real-time images (such as front-end images / video data captured by devices), or image / video data imported by the user.

[0030] Step S120: Perform main target detection on the image to be retrieved to obtain the main target region corresponding to the main target in the image to be retrieved.

[0031] The image to be retrieved can be any image / video data in the base database of the image and text retrieval model, or it can be a set of image / video data specified by the user.

[0032] In one embodiment, after determining the main target and sub-targets to be retrieved, main target detection is performed on the image to be retrieved to obtain at least one candidate main target region and its corresponding target category confidence score in the image, wherein the target category confidence score represents the confidence that the candidate main target region is the main target. Then, candidate main target regions whose target category confidence scores meet a preset confidence requirement are determined as the main target regions.

[0033] If the preset reliability requirement is greater than a preset reliability threshold (e.g., 0.8), then the candidate main target region will be used as the main target region, thereby avoiding low-quality detection results. The main target region must at least include the coordinate bounding box data of the main target.

[0034] By filtering based on the confidence level of the target category, the detection results of the main target region are made more reliable, reducing false detections caused by image noise or background interference, thereby lowering the error rate of main target localization.

[0035] Step S130: Based on the main target region, determine the associated regions of the sub-targets in the image to be retrieved.

[0036] In one embodiment, to improve the detection accuracy of sub-targets, the main target region can be adjusted to obtain an associated region, wherein the adjustment method is expansion or reduction. At least one of the adjustment method and the adjustment range of the main target region is determined based on the type of the sub-target. The main target region refers to the bounding box range obtained from the main target detection, and the associated region refers to the optimized region used for sub-target detection. The main target region is connected to the associated region through expansion or reduction operations, where expansion refers to increasing the bounding box size to cover the actual position of the sub-target, and reduction refers to decreasing the bounding box size to exclude irrelevant background. For example, when the sub-target is a cigarette, expansion is performed to include the hand region, and when the sub-target is a backpack, reduction is performed to focus on the shoulder position.

[0037] By expanding the main target region, the actual location of the sub-target is fully contained within the associated region, thus avoiding the omission of sub-targets due to an excessively small region and further improving the sub-target detection rate. By shrinking the main target region, the associated region is made more compact and focused on the location of the sub-target, reducing background noise interference and thus improving the sub-target detection accuracy. Combined with adaptive adjustment of expansion and shrinkage, the associated region dynamically matches the scale features of the sub-target, thereby further improving the accuracy of image and text retrieval.

[0038] Specifically, the size adjustment weight can be determined based on the type of the sub-target. The sub-target type refers to the attribute target category described in the search text, such as cigarettes or a backpack. The size adjustment weight is a scaling factor used to scale the width and height of the main target region. Then, the size parameters corresponding to the main target region are adjusted using the size adjustment weight to obtain the adjustment range of the main target region. In this embodiment, the size parameters corresponding to the main target region are the height and width of the coordinate frame obtained from the main target detection. The main target region is adjusted according to the adjustment range to obtain the associated region. See the following formula: (1) (2) in, Indicates the associated region. Indicates the main target area. Indicates the adjustment method. Indicates the range of adjustment. , Indicates the size adjustment weight. This parameter represents the width of the main target area. This represents the height parameter of the main target area.

[0039] After analyzing the sub-target types, the image and text retrieval model determines the size adjustment weights according to preset rules. For example, the size adjustment weights correspond to the cigarette type. =0.30, =0.35, weight adjusted according to backpack type and size =-0.20, =-0.20, and then the main target area can be adjusted according to the above formulas (1) and (2). The size adjustment weight is combined with the width and height parameters of the main target area to achieve dynamic scaling of width and height. Functionally, this design ensures that the associated area accurately covers the actual position of the sub-target, avoiding scale deviation caused by fixed adjustment; during the operation, the image and text retrieval model receives the sub-target type, looks up the table or calculates the weight, applies it to the width and height parameters of the main target area, and generates the associated area. The weight can be set based on the cigarette sub-target type. =0.30, =0.35 indicates that the width of the main target area is expanded by 30% and the height by 35%; the weight can be set based on the target type of the backpack. =-0.20, =-0.20 indicates that the width and height of the main target area are each reduced by 20%. In addition, the weight range can be set from -0.5 to 0.5 to adapt to different sub-target scale characteristics.

[0040] By determining the size adjustment weight based on the sub-target type, the associated region dynamically matches the actual scale characteristics of the sub-target, thereby avoiding the omission of sub-targets due to the region being too small and further improving the sub-target detection rate. By using the size adjustment weight to adjust the width and height parameters of the main target region, the adjustment process accurately adapts to the differences in sub-target types, thereby improving the sub-target detection accuracy. By adjusting the main target region according to the adjustment range, the associated region focuses on the sub-target position, reducing background noise interference, thereby further improving the accuracy of image and text retrieval.

[0041] Step S140: Perform sub-target detection on the associated region to obtain the sub-target region corresponding to the sub-target.

[0042] In one implementation, after determining the associated region, since the sub-target may be smaller than the main target, the detection of the sub-target needs to be more detailed. Specifically, semantic feature extraction and detail feature extraction can be performed on the associated region separately to obtain semantic feature maps and detail feature maps. Then, a spatial attention weight map is generated using the semantic feature map and detail feature map. Finally, the detail feature map is weighted using the spatial attention weight map to obtain the sub-target region within the associated region.

[0043] Semantic feature maps can be extracted from deep networks, containing target category information but with low spatial resolution, such as through the output of high-level convolutional layers in ResNet. Detail feature maps can be extracted from shallow networks, preserving edge texture information but containing more noise, such as through the output of low-level convolutional layers in ResNet. This approach provides target category information through semantic feature extraction, preventing misclassification of background as the primary target in small target detection; detail feature extraction preserves edge texture information, thus improving the completeness of small target feature representation. Furthermore, spatial attention weight maps adaptively focus on sub-target regions, making the sub-target detection region more accurately match the actual location; weighted detail feature maps effectively suppress background noise, thereby improving the sub-target detection rate. This feature combination significantly improves the detection accuracy of sub-targets with small scale and limited coverage in the image, thereby optimizing the overall performance of large-scale image-text retrieval models.

[0044] Furthermore, to generate the spatial attention weight map, semantic features and edge features can be concatenated to obtain concatenated features. Next, convolution is performed on the concatenated features to obtain convolutional features. Activation processing is then applied to the convolutional features to obtain the spatial attention weight map. See the following formula for reference: (3) (4) in, Represents semantic features, Indicates edge features, This indicates a feature concatenation operation. It is a 1×1 convolutional layer. This represents the Sigmoid activation function. This is a spatial attention weight map. This is element-wise multiplication (i.e., multiplying elements in the same position directly). This indicates the sub-target region.

[0045] Semantic feature map With detailed feature map A high-dimensional feature tensor (i.e., spliced ​​feature) is formed by channel concatenation and fusion. Then, it is used... Convolutional layers perform dimensionality reduction on the concatenated features and learn cross-channel weights. For example, a VGG (Visual Geometry Group) network can be used to extract features, concatenating a 256-channel semantic feature map with a 512-channel detail feature map to create a 768-channel feature map, which is then processed... The convolutional layer is compressed to 1 channel to generate a spatial attention weight map. Then, the sigmoid activation function is used to generate the spatial attention weight map from the dimensionality-reduced concatenated features. The values ​​of the graph elements range from 0 to 1 and are used to quantify the importance of regions.

[0046] After obtaining the spatial attention weight map Then, the spatial attention weight map can be... With detailed feature map Element-wise multiplication is performed to enhance key regions, thereby obtaining sub-target regions. For example, when detecting cigarette sub-targets, the semantic feature map identifies the human body category, the detail feature map preserves the edge details of the cigarette, and the spatial attention weight map focuses on the hand area and suppresses background noise.

[0047] Step S150: Based on the correlation between the main target region and the sub-target region, determine whether the image to be retrieved should be used as the retrieval result of the text to be retrieved.

[0048] In one embodiment, to obtain the correlation between the main target region and the sub-target region, the spatial coverage of the main target region to the sub-target region can be calculated, and the positional density between the main target region and the sub-target region can be determined. The correlation is then determined based on the spatial coverage and positional density.

[0049] Spatial coverage is defined as the ratio of the overlapping area between the main target region and the sub-target region to the total area of ​​the sub-target region. For example, when detecting a "smoker," if the overlap between the human body region and the cigarette region is 10%, the spatial coverage is 0.1. Location density is calculated based on the Euclidean distance between the center points of the main target region and the sub-target region. Furthermore, a spatial coverage threshold of 0.6 and a location density threshold of 0.5 can be used to determine correlation. For example, a strong correlation is considered when both spatial coverage ≥ 0.6 and location density ≥ 0.5. In practice, spatial coverage is first used to quantify the coverage of sub-targets within the main target region to eliminate background interference. Then, location density is used to assess the spatial proximity between the sub-target and the main target. Finally, both are combined to determine correlation, ensuring a logical connection between the sub-target and the main target.

[0050] The calculation of spatial coverage allows for a quantitative assessment of the coverage of sub-target regions within the main target region, thus avoiding false detections caused by the mixing of sub-targets with background information. The calculation of location density allows for an accurate assessment of the spatial distance relationship between the main target region and the sub-target region, thereby eliminating invalid associations where sub-targets are far from the main target. The determination of association based on the combination of spatial coverage and location density further improves the accuracy of association verification, which is beneficial for improving the recall rate of large-scale image and text retrieval.

[0051] Specifically, to calculate spatial coverage, the first area of ​​the intersection between the main target area and the sub-target area, and the second area of ​​the sub-target area can be obtained first, and the ratio between the first area and the second area can be used as the spatial coverage.

[0052] To calculate the location density, the distance between the center point of the main target region and the center point of the sub-target region can be obtained, and the location density can be calculated based on the distance. The location density is negatively correlated with the distance.

[0053] To determine correlation based on spatial coverage and location density, the product of spatial coverage and location density can be used as the correlation. See the following formula: (5) in, ( ) represents the area calculation formula. This represents the first area of ​​the intersection between the main target region and the sub-target regions. This represents the second area of ​​the sub-target region. For spatial coverage, The coordinates of the center point of the main target area. The coordinates of the center point of the sub-target region. This represents the Euclidean distance between the center point of the main target region and the center point of the sub-target region. Indicates the degree of correlation.

[0054] In the process of calculating correlation, the intersection area is first used... With sub-target area The ratio between them is used to quantify the coverage of the sub-target within the main target. Then, the spatial proximity is evaluated by calculating the distance between the center points. Finally, the product is used to comprehensively judge the strength of the correlation, ensuring the accurate verification of the logical connection between the sub-target and the main target.

[0055] In one embodiment, a threshold can be set to determine whether an image to be retrieved is a search result for the text to be retrieved. Specifically, in response to a relevance greater than a preset relevance threshold, the image to be retrieved is determined to be a search result for the text to be retrieved.

[0056] Please see Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of the image and text retrieval method of this application. Specifically, it may include the following steps: Step S210: Parse the text to be retrieved to determine the main target and sub-targets in the text.

[0057] This step is the same as step S110 above, and will not be repeated here.

[0058] Step S220: Perform main target detection on the image to be retrieved to obtain the main target region corresponding to the main target in the image to be retrieved.

[0059] This step is the same as step S120 above, and will not be repeated here.

[0060] Step S230: Based on the main target region, determine the associated regions of the sub-targets in the image to be retrieved.

[0061] This step is the same as step S130 above, and will not be repeated here.

[0062] Step S240: Perform sub-target detection on the associated region to obtain the sub-target region corresponding to the sub-target.

[0063] This step is the same as step S140 above, and will not be repeated here.

[0064] Step S250: Detect the sub-target region, obtain the detection results, and determine whether to adjust the sub-target region based on the detection results.

[0065] Step S260: Based on the correlation between the main target region and the sub-target region, determine whether the image to be retrieved should be used as the retrieval result of the text to be retrieved.

[0066] In one embodiment, after obtaining the sub-target region, to avoid the sub-target region containing only the features of the sub-target while completely ignoring the features of the main target, it is necessary to detect the sub-target region and obtain the detection result. If the detection result shows that the sub-target region contains some features of the main target, then there is no need to adjust the sub-target region; the correlation between the sub-target region and the main target region can be directly calculated, and the image to be retrieved can be used as the retrieval result of the text to be retrieved based on this correlation. If the detection result shows that the sub-target region does not contain the features of the main target, then the sub-target region needs to be expanded so that it contains some features of the main target. After that, the correlation between the adjusted sub-target region and the main target region is calculated, and the image to be retrieved can be used as the retrieval result of the text to be retrieved based on this correlation.

[0067] This solution avoids situations where the sub-target region only contains the sub-target's own features and is completely detached from the main target's features, ensuring that the sub-target region has a reasonable correlation with the main target. At the same time, it determines whether the sub-target region needs to be adjusted based on the detection results. If it does not contain the main target's features, it is expanded so that the sub-target region contains some of the main target's features, thus ensuring the effectiveness of the region.

[0068] The relevance between sub-targets and the main target is the core basis for determining search results. Step S250 optimizes the sub-target region to ensure that the two have a relevance judgment, avoiding the failure of the relevance judgment due to improper region. By reducing misjudgments caused by the sub-target region deviating from the main target (such as misjudging the sub-target region unrelated to the main target as a valid search result), the rigor of the search logic is ensured.

[0069] This application optimizes the retrieval performance of image-text retrieval models when there are multiple targets or a main target with sub-targets through collaborative association of related targets. It splits the main target and sub-targets (conditional targets) based on the text to be retrieved, and generates corresponding dynamic association regions based on the main target to optimize the actual detection area. Furthermore, based on the generated dynamic association regions, sub-targets (conditional targets) are detected and their features are optimized to obtain more accurate sub-target (conditional target) regions. The detected main target region and sub-target (conditional target) regions are then validated for association constraints, ultimately yielding more accurate image-text retrieval results.

[0070] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0071] Please see Figure 3 , Figure 3This is a schematic diagram of the framework of an embodiment of the image and text retrieval device 300 of this application. The image and text retrieval device 300 includes a parsing module 310, a main target region detection module 320, an association module 330, a sub-target region detection module 340, and a judgment module 350. The parsing module 310 parses the text to be retrieved to determine the main target and sub-targets in the text, wherein the sub-targets are conditional targets of the main target. The main target region detection module 320 performs main target detection on the image to be retrieved to obtain the main target region corresponding to the main target in the image to be retrieved. The association module 330 determines the association region of the sub-targets in the image to be retrieved based on the main target region. The sub-target region detection module 340 performs sub-target detection on the association region to obtain the sub-target region corresponding to the sub-target. The judgment module 350 determines whether the image to be retrieved is a retrieval result of the text to be retrieved based on the association between the main target region and the sub-target region.

[0072] In one embodiment, the association module 330 performs the function of determining the association region of the sub-target in the image to be retrieved based on the main target region, including: adjusting the main target region to obtain the association region, wherein the adjustment method is to expand or shrink.

[0073] In one embodiment, the association module 330 performs at least one of the adjustment method and the adjustment range of the main target area based on the type of the sub-target; and / or adjusts the main target area to obtain the association area, including: determining the size adjustment weight based on the type of the sub-target; adjusting the size parameters corresponding to the main target area using the size adjustment weight to obtain the adjustment range of the main target area; and adjusting the main target area according to the adjustment range to obtain the association area.

[0074] In one embodiment, the sub-target region detection module 340 performs sub-target detection on the associated region to obtain the sub-target region corresponding to the sub-target, including: extracting semantic features and extracting detail features on the associated region to obtain a semantic feature map and a detail feature map; generating a spatial attention weight map using the semantic feature map and the detail feature map; and weighting the detail feature map using the spatial attention weight map to obtain the sub-target region in the associated region.

[0075] In one embodiment, the sub-target region detection module 340 performs the following steps: generating a spatial attention weight map using semantic feature maps and detail feature maps, including: concatenating semantic features and edge features to obtain concatenated features; convolving the concatenated features to obtain convolutional features; and performing activation processing on the convolutional features to obtain a spatial attention weight map.

[0076] In one embodiment, before the determination module 350 determines whether the image to be retrieved is a retrieval result of the text to be retrieved based on the correlation between the main target region and the sub-target region, the method further includes: calculating the spatial coverage of the main target region to the sub-target region; and determining the positional density between the main target region and the sub-target region; and determining the correlation based on the spatial coverage and positional density.

[0077] In one embodiment, the determination module 350 performs calculations to obtain the spatial coverage of the main target region to the sub-target region, including: obtaining a first area of ​​the intersection between the main target region and the sub-target region, and a second area of ​​the sub-target region, and using the ratio between the first area and the second area as the spatial coverage; and / or, determining the positional density between the main target region and the sub-target region, including: obtaining the distance between the center point of the main target region and the center point of the sub-target region; calculating the positional density based on the distance, wherein the positional density is negatively correlated with the distance; and / or, determining the correlation based on the spatial coverage and the positional density, including: using the product between the spatial coverage and the positional density as the correlation.

[0078] In one embodiment, the parsing module 310 performs parsing of the text to be retrieved to determine the main target and sub-targets in the text, including: splitting the text to be retrieved into words to obtain main target keywords and sub-target keywords; identifying the main target keywords and sub-target keywords respectively to obtain the main target and sub-targets; and / or, the main target region detection module 320 performs main target detection on the image to be retrieved to obtain the main target region corresponding to the main target in the image to be retrieved, including: performing main target detection on the image to be retrieved to obtain at least one candidate main target region and corresponding target category confidence in the image to be retrieved, wherein the target category confidence represents the confidence that the candidate main target region is the main target; determining the candidate main target region whose target category confidence meets the preset confidence requirement as the main target region; and / or, the judgment module 350 performs judgment based on the correlation between the main target region and the sub-target region to determine whether the image to be retrieved is the retrieval result of the text to be retrieved, including: determining the image to be retrieved as the retrieval result of the text to be retrieved in response to the correlation being greater than a preset correlation threshold.

[0079] Please see Figure 4 , Figure 4This is a schematic diagram of the framework of an embodiment of the image-text retrieval model 400 of this application. The image-text retrieval model 400 includes a data acquisition module 410, a main target detection module 420, an associated region generation module 430, and an associatedness verification module 440. The data acquisition module 410 acquires real-time images (such as front-end images / video data captured by devices), or image / video data imported by the user. The main target detection module 420 parses the text to be retrieved to determine the main target and sub-targets in the text to be retrieved, and performs main target detection on the image to be retrieved to obtain the main target region corresponding to the main target in the image to be retrieved. The associated region generation module 430 determines the associated regions of the sub-targets in the image to be retrieved based on the main target regions. The associatedness verification module 440 performs sub-target detection on the associated regions to obtain the sub-target regions corresponding to the sub-targets, and determines whether the image to be retrieved is a retrieval result of the text to be retrieved based on the associated regions and the associated regions of the main target regions and the sub-target regions.

[0080] Please see Figure 5 , Figure 5 This is a schematic diagram of an embodiment of the electronic device 50 of this application. The electronic device 50 includes a memory 51 and a processor 52 coupled to each other. The processor 52 is used to execute program instructions stored in the memory 51 to implement the steps in any of the above-described embodiments of the image and text retrieval method. In a specific implementation scenario, the electronic device 50 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 50 may also include mobile devices such as laptops and tablets, which are not limited here.

[0081] Specifically, processor 52 controls itself and memory 51 to implement the steps in any of the above-described image and text retrieval method embodiments. Processor 52 can also be referred to as a CPU (Central Processing Unit). Processor 52 may be an integrated circuit chip with signal processing capabilities. Processor 52 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 52 can be implemented using integrated circuit chips.

[0082] Please see Figure 6 , Figure 6This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium 60 of this application. The computer-readable storage medium 60 stores program instructions 601 that can be executed by a processor. The program instructions 601 are used to implement the steps in any of the above-described embodiments of the image and text retrieval method.

[0083] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0084] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0085] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0086] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0087] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for image and text retrieval, characterized in that, include: The text to be retrieved is parsed to determine the main target and sub-targets in the text to be retrieved, wherein the sub-targets are conditional targets of the main target; Perform main target detection on the image to be retrieved to obtain the main target region corresponding to the main target in the image to be retrieved; Based on the main target region, the associated regions of the sub-targets in the image to be retrieved are determined; Sub-target detection is performed on the associated region to obtain the sub-target region corresponding to the sub-target; Based on the correlation between the main target region and the sub-target region, it is determined whether the image to be retrieved should be used as a retrieval result for the text to be retrieved.

2. The method according to claim 1, characterized in that, The step of determining the associated region of the sub-target in the image to be retrieved based on the main target region includes: The main target area is adjusted to obtain the associated area, wherein the adjustment is performed by expanding or shrinking.

3. The method according to claim 2, characterized in that, The adjustment method and the adjustment range of the main target area are determined based on the type of the sub-target; And / or, adjusting the main target region to obtain the associated region includes: The size adjustment weight is determined based on the type of the sub-target; The size parameters corresponding to the main target area are adjusted using the size adjustment weight to obtain the adjustment range of the main target area; The main target region is adjusted according to the adjustment range to obtain the associated region.

4. The method according to claim 1, characterized in that, The step of performing sub-target detection on the associated region to obtain the sub-target region corresponding to the sub-target includes: Semantic feature extraction and detail feature extraction are performed on the associated regions respectively to obtain semantic feature maps and detail feature maps; Using the semantic feature map and detail feature map, a spatial attention weight map is generated; The spatial attention weight map is used to weight the detail feature map to obtain the sub-target region in the associated region.

5. The method according to claim 4, characterized in that, The step of generating a spatial attention weight map using the semantic feature map and detail feature map includes: The semantic features and edge features are concatenated to obtain the concatenated features; The spliced ​​features are convolved to obtain convolutional features; The convolutional features are activated to obtain the spatial attention weight map.

6. The method according to claim 1, characterized in that, Before determining whether the image to be retrieved should be used as a retrieval result for the text to be retrieved based on the correlation between the main target region and the sub-target region, the method further includes: The spatial coverage of the main target region over the sub-target region is calculated; and, Determine the positional density between the main target region and the sub-target region; The correlation is determined based on the spatial coverage and the location density.

7. The method according to claim 6, characterized in that, The calculation of the spatial coverage of the main target region over the sub-target region includes: Obtain the first area of ​​the intersection between the main target region and the sub-target region, and the second area of ​​the sub-target region; use the ratio between the first area and the second area as the spatial coverage. And / or, determining the positional density between the main target region and the sub-target region includes: Obtain the distance between the center point of the main target region and the center point of the sub-target region; The location density is calculated based on the distance, wherein the location density is negatively correlated with the distance; And / or, determining the correlation based on the spatial coverage and the location density includes: The product of the spatial coverage and the location density is taken as the correlation.

8. The method according to claim 1, characterized in that, The step of parsing the text to be retrieved to determine the main target and sub-targets in the text includes: The text to be retrieved is split into words to obtain the main target keywords and sub-target keywords; The main target keyword and the sub-target keyword are identified separately to obtain the main target and the sub-target; And / or, the step of performing main target detection on the image to be retrieved to obtain the main target region corresponding to the main target in the image to be retrieved includes: The image to be retrieved is subjected to main target detection to obtain at least one candidate main target region and corresponding target category confidence in the image to be retrieved, wherein the target category confidence represents the confidence that the candidate main target region is the main target; The candidate main target regions whose target category confidence scores meet the preset confidence requirements are determined as the main target regions; And / or, determining whether the image to be retrieved should be used as a retrieval result for the text to be retrieved based on the correlation between the main target region and the sub-target region includes: In response to the correlation being greater than a preset correlation threshold, the image to be retrieved is determined as the retrieval result of the text to be retrieved.

9. An electronic device, characterized in that, The method includes a memory and a processor coupled to each other, the processor being used to execute program instructions stored in the memory to implement the image and text retrieval method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the image and text retrieval method according to any one of claims 1 to 8.

Images