Image-text correlation detection method and device, electronic equipment and storage medium

Abstract

The invention provides an image-text correlation detection method and device, electronic equipment and a storage medium, and relates to the artificial intelligence fields of natural language processing technologies, deep learning, computer vision and the like. The method comprises the following steps: analyzing a target document to be processed to obtain a picture in the target document and structured context information corresponding to the picture; obtaining a hierarchical text vector corresponding to the target document; and for any picture, respectively determining a target visual vector corresponding to the picture, and determining a correlation judgment result between the picture and the target document according to the hierarchical text vector, the target visual vector and the structured context information. By applying the scheme provided by the invention, the accuracy and the like of the obtained correlation judgment result can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of natural language processing, deep learning and computer vision, and especially to methods, devices, electronic devices and storage media for detecting image-text relevance. Background Technology

[0002] In scenarios such as document content review, generated content quality assessment, advertising compliance review, and search and recommendation ranking, image-text relevance detection is typically required. This involves determining whether images in a document are relevant to the document content (especially surrounding paragraphs). For example, it's necessary to determine whether illustrations in news articles match the paragraph content and are not clickbait or misleading images. Another example is determining whether images are embedded in irrelevant text to circumvent review procedures during document content review. Summary of the Invention

[0003] This disclosure provides methods, apparatus, electronic devices, and storage media for detecting the correlation between images and text.

[0004] A method for detecting the relevance of images and text includes:

[0005] The target document to be processed is parsed to obtain the images and the corresponding structured context information of the images;

[0006] Obtain the hierarchical text vector corresponding to the target document;

[0007] For any given image, the target visual vector corresponding to the image is determined, and based on the hierarchical text vector, the target visual vector, and the structured context information, the correlation determination result between the image and the target document is determined.

[0008] A text-image correlation detection device includes: a first processing module and a second processing module;

[0009] The first processing module is used to parse the target document to be processed to obtain the images therein and the structured context information corresponding to the images;

[0010] The second processing module is used to obtain the hierarchical text vector corresponding to the target document, and for any image, determine the target visual vector corresponding to the image, and determine the correlation judgment result between the image and the target document based on the hierarchical text vector, the target visual vector and the structured context information.

[0011] An electronic device, comprising:

[0012] At least one processor; and

[0013] A memory communicatively connected to the at least one processor; wherein,

[0014] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described above.

[0015] A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0016] A computer program product includes a computer program / instructions that, when executed by a processor, implement the method described above.

[0017] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0018] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0019] Figure 1 This is a flowchart of the first embodiment of the image-text correlation detection method described in this disclosure;

[0020] Figure 2 This is a flowchart illustrating an embodiment of the method for generating comprehensive matching evaluation results as described in this disclosure;

[0021] Figure 3 This is a flowchart of the second embodiment of the image-text correlation detection method described in this disclosure;

[0022] Figure 4 This is a schematic diagram of the composition structure of Embodiment 400 of the image-text correlation detection device described in this disclosure;

[0023] Figure 5 A schematic block diagram of an electronic device 500 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0024] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0025] Furthermore, it should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0026] Figure 1 This is a flowchart of the first embodiment of the image-text relevance detection method described in this disclosure. Figure 1 As shown, the specific implementation methods are as follows.

[0027] In step 101, the target document to be processed is parsed to obtain the images and their corresponding structured context information.

[0028] In step 102, the hierarchical text vector corresponding to the target document is obtained.

[0029] In step 103, for any image, the target visual vector corresponding to the image is determined, and the relevance judgment result between the image and the target document is determined based on the hierarchical text vector, the target visual vector and the structured context information.

[0030] Traditionally, keywords are extracted directly from the target document, and then the relevance of the image to the target document is determined based on these keywords. However, this method is usually not very accurate.

[0031] The above-described method allows for the parsing of the target document to obtain each image and its corresponding structured context information. It also allows for the acquisition of the hierarchical text vector corresponding to the target document and the target visual vector corresponding to each image. Furthermore, by combining the hierarchical text vector, the target visual vector, and the structured context information, the required relevance determination result can be determined, thereby improving the accuracy of the obtained relevance determination result.

[0032] It should be noted that the target documents, images, structured context information, and relevance determination results in the embodiments described in this disclosure are not targeted at any specific user and are not intended to reflect the personal information of any specific user. The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solutions of this disclosure all comply with relevant laws and regulations and do not violate public order and good morals.

[0033] The target document described in this disclosure can be a rich text document, such as a HyperText Markup Language (HTML) document or a Markdown document. A rich text structure parser or rich text parsing rules can be used to parse the target document's structure, constructing a hierarchical structural representation, and generating multi-dimensional structured contextual information for each image, thereby providing accurate input for subsequent processing.

[0034] In some embodiments of this disclosure, the structured context information corresponding to any image may include: the chapter in which the image is located, the context paragraph of the image, and the image caption information. The context paragraph includes M adjacent paragraphs before the image and N adjacent paragraphs after the image, where M and N are both positive integers and their specific values ​​can be determined according to actual needs, such as both being 3.

[0035] For example, the target document can be converted into a Document Object Model (DOM) tree, recognizing nodes such as heading tags (h1 / h2 / h3), paragraphs (p / div), lists (ul / li), and images (img / figure). A hierarchical structure of "document → chapter → paragraph" can be constructed based on heading levels, aggregating consecutive paragraph nodes under each chapter. Furthermore, the location and chapter of an image within the target document can be determined, and the three paragraphs before and after the image can be extracted as its context paragraphs. Additionally, image caption information can be obtained, such as extracting caption information from image-related caption content, alt attributes, title attributes, and adjacent text content.

[0036] In practical applications, in addition to the chapter where the image is located, the paragraphs surrounding the image, and the image caption information, the structured context information can also include other content, the specific content of which can be determined according to actual needs.

[0037] By extracting structured contextual information, the structural relationship between images and target documents can be clarified, and irrelevant paragraphs can be reduced, thereby reducing the workload of subsequent processing and improving the accuracy of subsequent processing results.

[0038] In addition, the target document may contain one image or multiple images. If it contains multiple images, each image can be processed in the same way to obtain the corresponding relevance judgment results.

[0039] After obtaining each image and its corresponding structured context information, the hierarchical text vector corresponding to the target document can be obtained.

[0040] In some embodiments of this disclosure, the hierarchical text vector may include: target paragraph embeddings of each paragraph in the target document, target chapter embeddings of each chapter in the target document, and target document embeddings of the target document. Accordingly, the method of obtaining the hierarchical text vector corresponding to the target document may include: obtaining the initial paragraph embeddings of each paragraph, the initial chapter embeddings of each chapter, and the initial document embeddings of the target document, and determining the target paragraph embeddings, target chapter embeddings, and target document embeddings based on the initial paragraph embeddings, initial chapter embeddings, and initial document embeddings.

[0041] In some embodiments of this disclosure, the basic paragraph vectors of each paragraph can be determined separately, and the initial chapter vectors of each chapter can be determined separately based on the basic paragraph vectors belonging to the same chapter, and the initial document vectors can be determined based on the initial chapter vectors. In addition, the basic paragraph vectors can be adjusted based on the initial document vectors to obtain the initial paragraph vectors of each paragraph.

[0042] For example, a pre-trained language model can be used to encode the word (token) sequence of the target text to obtain each token embedding. Then, for any paragraph in the target document, based on all the token embeddings, a learnable attention network can be used to automatically calculate the importance weight of each token in the paragraph to the overall semantics of the paragraph. Then, all the token embeddings can be weighted and summed according to these weights to obtain a fixed-dimensional basic paragraph vector that can focus on key information.

[0043] Pre-trained language models can be models such as Bidirectional Encoder Representations from Transformers (BERT) or Enhanced Representation through kNowledge IntEgration (ERNIE).

[0044] Then, the self-attention weighted aggregation of all basic paragraph vectors under the same chapter can be performed to obtain the initial chapter vector of each chapter, and the self-attention weighted aggregation of the initial chapter vectors of all chapters can be performed to obtain the initial document vector of the target document.

[0045] Furthermore, a top-down context enhancement mechanism can be introduced, such as using the initial document vector as a condition and adjusting each basic paragraph vector through a lightweight adapter to obtain the initial paragraph vector of each paragraph, so that each initial paragraph vector can simultaneously include local semantics and global context information.

[0046] In addition, through the above-mentioned hierarchical text encoding process, text information of different granularities such as target documents, chapters, and paragraphs can be encoded into a unified vector space, while retaining hierarchical structure information to meet the matching requirements of different levels, and taking into account both local accuracy and global consistency.

[0047] In some embodiments of this disclosure, each initial paragraph vector, each initial chapter vector, and each initial document vector can be directly determined as each target paragraph vector, each target chapter vector, and each target document vector. Alternatively, the domain to which the target document belongs can be determined, and the domain vector corresponding to the domain can be obtained. Then, each initial paragraph vector, each initial chapter vector, and each initial document vector can be adjusted according to the domain vector to obtain each target paragraph vector, each target chapter vector, and each target document vector.

[0048] For example, assuming the target document belongs to domain a, then the business configuration information of domain a and the predetermined historical statistical values ​​of domain a can be used as inputs. The domain vector can be generated by meta-learning algorithms such as parametric conditional layer, model-agnostic meta-learning (MAML), and crawler meta-learning (Reptile). The dimension of the domain vector can be a predetermined value, such as 64.

[0049] After generating the domain vectors, the initial paragraph vectors, initial chapter vectors, and initial document vectors can be adjusted using the domain vectors. This involves applying a linear transformation or a multi-layer perceptron (MLP) transformation to obtain the target paragraph vectors, target chapter vectors, and target document vectors, thus achieving domain adaptation and further improving the accuracy of the target paragraph vectors, target chapter vectors, and target document vectors.

[0050] For any image in the target document, the corresponding target visual vector can be determined separately. In some embodiments of this disclosure, the global image vector of the image can be obtained, and salient region detection can be performed on the image to obtain the region vectors of each detected salient region. Furthermore, text information in the image can be recognized to obtain the information vector of the recognized text information. Then, the global image vector, region vector, and information vector can be fused to obtain an initial visual vector, and the target visual vector can be determined based on the initial visual vector.

[0051] For example, a pre-trained image coding model can be used to generate global image vectors. The pre-trained image coding model can be a Contrastive Language-Image Pre-training (CLIP) image encoder or a Vision Transformer (ViT), etc.

[0052] For example, a saliency detection model can be used to detect salient regions in an image. The saliency detection model can be a MobileNet Single Shot multibox Detector model, a lightweight general segmentation model (SAM, Segment Anything Model), or a faster region-based convolutional neural network model (Faster R-CNN, Faster Region-Based Convolutional Neural Network). For each detected salient region, the corresponding region vector can be obtained.

[0053] In addition, text information in images can be recognized. For example, text information in images can be recognized through optical character recognition (OCR) technology, and text information can be encoded using a text encoder to obtain information vectors.

[0054] Furthermore, the global image vector, region vector, and information vector can be fused using MLP to obtain an initial visual vector. In some embodiments of this disclosure, the initial visual vector can be directly determined as the target visual vector, or the initial visual vector can be adjusted based on the neighborhood vector to obtain the target visual vector.

[0055] Through the above processing, images can be encoded into multi-scale target visual vectors, which simultaneously include global image information, salient region information, and text information, thereby improving the accuracy of subsequent processing results based on target visual vectors.

[0056] For any given image, after obtaining the hierarchical text vector and the target visual vector, the relevance between the image and the target document can be determined based on the hierarchical text vector, the target visual vector, and the structured context information corresponding to the image.

[0057] In some embodiments of this disclosure, the comprehensive matching degree evaluation result between the image and the target document can be determined first based on the hierarchical text vector, the target visual vector, and the structured context information, and then the relevance determination result can be determined based on the comprehensive matching degree evaluation result.

[0058] This allows for the integration of multi-source information to form an objective comprehensive matching degree assessment result. Subsequently, a judgment can be made based on the comprehensive matching degree assessment result, thereby separating the matching process from the decision-making process and improving the reliability of the relevance judgment result.

[0059] In some embodiments of this disclosure, the method for determining the comprehensive matching degree evaluation result between the image and the target document may include: determining a first-level matching degree evaluation result based on the target document vector and the target visual vector; determining a second-level matching degree evaluation result based on the target chapter vector and the target visual vector of the chapter in which the image is located; determining a third-level matching degree evaluation result based on the target paragraph vector and the target visual vector of each context paragraph; determining a fourth-level matching degree evaluation result based on the caption vector, information vector, and target visual vector corresponding to the caption information; and determining a comprehensive matching degree evaluation result based on the matching degree evaluation results of each level.

[0060] Specifically, in some embodiments of this disclosure, a first correlation between the target document vector and the target visual vector can be obtained, and the first correlation can be determined as the first-level matching degree evaluation result; and / or, a second correlation between the target chapter vector of the chapter where the image is located and the target visual vector can be obtained, and the second correlation can be determined as the second-level matching degree evaluation result; and / or, a third correlation between the target paragraph vector and the target visual vector of each context paragraph can be obtained, and the maximum value among the third correlations can be determined as the third-level matching degree evaluation result; and / or, a fourth correlation between the caption vector and the target visual vector can be obtained, and a fifth correlation between the information vector (the vector corresponding to the text information identified from the image) and the target visual vector can be obtained, and the fourth correlation and the fifth correlation are weighted and added together to obtain the fourth-level matching degree evaluation result.

[0061] As can be seen, the scheme described in this disclosure can perform four levels of matching between images and target documents: document level matching, chapter level matching, paragraph level matching, and entity level matching. Through four-level matching, multi-granularity evaluation of image-text relationships can be achieved, grasping both overall thematic consistency and capturing local detailed relationships. Furthermore, through the complementarity and constraints between levels, the robustness and interpretability of the matching are significantly improved, avoiding potential omissions or misjudgments that might occur with single-level matching. In practical applications, to reduce computational costs, the four-level matching can also be simplified to a two-level matching of "chapter level + paragraph level".

[0062] In document-level matching, the first relevance between the target document vector and the target visual vector is obtained, and this first relevance is determined as the first-level matching evaluation result. This first-level matching evaluation result reflects the degree of matching between the entire document and the image. In chapter-level matching, the second relevance between the target chapter vector and the target visual vector of the chapter containing the image is obtained, and this second relevance is determined as the second-level matching evaluation result. This second-level matching evaluation result reflects the degree of matching between the theme of the chapter containing the image and the image. In paragraph-level matching, the third relevance between the target paragraph vector and the target visual vector of each context paragraph is obtained, and the maximum value of each third relevance is determined as the third-level matching evaluation result. This third-level matching evaluation result reflects the degree of matching between the paragraph and the image. In entity-level matching, the fourth relevance between the caption vector and the target visual vector, and the fifth relevance between the information vector and the target visual vector are obtained, and the fourth and fifth relevances are weighted and summed to obtain the fourth-level matching evaluation result. This fourth-level matching evaluation result reflects the degree of fine-grained information matching.

[0063] In some embodiments of this disclosure, each relevance can be calculated using a cross-modal matching network model. For example, the following calculations can be performed using a cross-modal matching network model:

[0064] Score = w1 * cos(ht, hi) + w2 *<ht,hi> +w3*MLP([ht ; hi ]); (1)

[0065] Where ht represents the text vector; for example, when calculating the first relevance, ht represents the target document vector; when calculating the second relevance, ht represents the target chapter vector of the chapter containing the image; hi represents the target visual vector of the image; and cos(ht, hi) represents the cosine similarity.<ht,hi> The dot product similarity is represented by MLP([ht;hi]), which represents the bimodal concatenation features. w1, w2, and w3 are all learnable weights that can be optimized through training samples.

[0066] After obtaining the first-level matching degree evaluation results, the second-level matching degree evaluation results, the third-level matching degree evaluation results, and the fourth-level matching degree evaluation results, the comprehensive matching degree evaluation result can be determined based on the matching degree evaluation results of each level.

[0067] In some embodiments of this disclosure, the consistency penalty value corresponding to the image can be obtained, and the fusion result of the matching degree evaluation results at each level can be obtained. Then, the comprehensive matching degree evaluation result can be determined based on the fusion result and the consistency penalty value.

[0068] In some embodiments of this disclosure, the method for obtaining the consistency penalty value corresponding to the image may include: obtaining the mean of the second-level matching degree evaluation result and the third-level matching degree evaluation result, obtaining the first difference between the mean and the first tolerance threshold, obtaining the absolute value of the second difference between the second-level matching degree evaluation result and the third-level matching degree evaluation result, and determining the consistency penalty value based on the comparison result between the absolute value and the second tolerance threshold and the comparison result between the first-level matching degree evaluation result and the first difference.

[0069] Generally speaking, the first-level matching score should not be significantly lower than the average of the second-level and third-level matching scores, i.e.:

[0070] sdoc ≥avg(schap,spara)−δ1; (2)

[0071] Where δ1 represents the first tolerance threshold, such as 0.1, sdoc represents the first-level matching degree evaluation result, schap represents the second-level matching degree evaluation result, and spara represents the third-level matching degree evaluation result.

[0072] Furthermore, the matching evaluation results of the second-level and third-level levels need to be close, that is:

[0073] |schap−spara|≤δ2; (3)

[0074] Where δ2 represents the second tolerance threshold, such as 0.15.

[0075] Accordingly, after obtaining the first difference (i.e., avg(schap, spara)−δ1), the first penalty value can be determined by comparing the first level matching degree evaluation result with the first difference, according to the predetermined rules and the requirements in formula (2). After obtaining the absolute value of the second difference (i.e. |schap−spara|), the second penalty value can be determined by comparing the absolute value with the second tolerance threshold, according to the predetermined rules and the requirements in formula (3). Then, the first penalty value and the second penalty value can be weighted and added together to obtain the required consistency penalty value.

[0076] In addition, in some embodiments of this disclosure, the method for obtaining the fusion result of the matching degree evaluation results at each level may include: using a weighted fusion model to fuse the matching degree evaluation results at each level to obtain the fusion result.

[0077] Accordingly, the method for determining the comprehensive matching degree evaluation result based on the fusion result and the consistency penalty value can be as follows:

[0078] sfinal =f(sdoc,schap,spara,sentity)*(1−λ*pconsistency); (4)

[0079] Wherein, sfinal represents the overall matching evaluation result, sentence represents the fourth-level matching evaluation result, the values ​​of sdoc, schap, spara and sentence are usually between [0,1], f() represents the weighted fusion model, pconsistency represents the consistency penalty value, which is usually also between [0,1], and λ represents the penalty coefficient, such as 0.3.

[0080] By adopting the above processing method, the comprehensive matching degree evaluation result can be determined by combining the fusion result of the matching degree evaluation results of each level and the consistency penalty value. This not only integrates multi-granularity text and image association information, but also constrains the contradictions between levels through the penalty value, reducing local noise and cross-level mismatch interference.

[0081] Based on the above introduction, Figure 2 This is a flowchart illustrating an embodiment of the method for generating comprehensive matching evaluation results as described in this disclosure. Figure 2 As shown, the specific implementation methods are as follows.

[0082] In step 201, the first relevance between the target document vector and the target visual vector is obtained, and the first relevance is determined as the first-level matching degree evaluation result.

[0083] In step 202, the second correlation between the target chapter vector and the target visual vector of the chapter in which the image is located is obtained, and the second correlation is determined as the second-level matching degree evaluation result.

[0084] In step 203, the third relevance between the target paragraph vector and the target visual vector of each context paragraph is obtained, and the maximum value of each third relevance is determined as the third-level matching degree evaluation result.

[0085] In step 204, the fourth correlation between the image annotation vector and the target visual vector is obtained, and the fifth correlation between the information vector and the target visual vector is obtained. The fourth correlation and the fifth correlation are weighted and added together to obtain the fourth-level matching degree evaluation result.

[0086] In step 205, the consistency penalty value corresponding to the image is determined.

[0087] For example, the mean of the second-level matching degree evaluation result and the third-level matching degree evaluation result can be obtained, and the first difference between the mean and the first tolerance threshold can be obtained. The absolute value of the second difference between the second-level matching degree evaluation result and the third-level matching degree evaluation result can be obtained. The consistency penalty value is determined based on the comparison result between the absolute value and the second tolerance threshold and the comparison result between the first-level matching degree evaluation result and the first difference.

[0088] In step 206, the fusion results of the matching degree evaluation results at each level are obtained.

[0089] In step 207, the overall matching degree evaluation result is determined based on the fusion result and the consistency penalty value.

[0090] Based on the comprehensive matching evaluation results, the relevance between the image and the target document can be determined.

[0091] In some embodiments of this disclosure, the relevance determination result may include: a basic determination result and a relevance probability. The basic determination result is used to indicate whether there is a relevance between the image and the target document, and the relevance probability is used to indicate the relevance probability value between the image and the target document. Accordingly, the method of determining the relevance determination result based on the comprehensive matching degree evaluation result may include: determining the target threshold corresponding to the target document based on the matching degree evaluation results at each level and the domain vector; fusing the comprehensive matching degree evaluation result, the consistency penalty value, the target threshold, and the matching degree evaluation results at each level to obtain target reference information; and determining the basic determination result and the relevance probability based on the target reference information.

[0092] In practical applications, the target threshold can be a pre-set empirical value, but preferably, it can be a determined dynamic threshold. In some embodiments of this disclosure, the target threshold can be generated using a threshold prediction model based on the matching degree evaluation results at each level and the neighborhood vector.

[0093] As mentioned above, the domain to which the target document belongs can be determined, and the domain vector corresponding to the domain can be obtained. Accordingly, the target threshold can be generated by a threshold prediction model based on the matching degree evaluation results of each level and the domain vector.

[0094] If possible:

[0095] T=g(d,sdoc,schap,spara,sentity); (5)

[0096] Where g() represents the threshold prediction model, which can be a lightweight MLP (such as 2-3 layers), sdoc, schap, spara and sentenceity represent the matching degree evaluation results of the first layer, the second layer, the third layer and the fourth layer respectively, and d represents the neighborhood vector.

[0097] Through the above processing, adaptive dynamic threshold determination can be achieved, overcoming the drawbacks of fixed threshold "one-size-fits-all" and improving the adaptability of the solution described in this disclosure in different fields.

[0098] By fusing the comprehensive matching evaluation results, consistency penalty value, target threshold, and matching evaluation results at each level, target reference information can be obtained. The method of fusion is not limited; linear weighting or attention fusion methods can be used. Then, based on the target reference information, a relevance evaluation model can be used to generate basic judgment results and relevance probabilities. The relevance evaluation model can be a small neural network model or a linear model, etc. The target reference information can be input into the relevance evaluation model to obtain the output basic judgment results and relevance probabilities. The basic judgment result indicates whether it is (relevant) or not (irrelevant), and the relevance probability is a specific numerical value that can range from [0,1].

[0099] In some embodiments of this disclosure, the relevance determination result may further include: a certainty level and evidence information, wherein the evidence information is used to explain the reason for giving the basic determination result, and the certainty level is used to explain the certainty of the basic determination result. Accordingly, the method of determining the relevance determination result based on the comprehensive matching degree evaluation result may further include: determining the certainty level and evidence information based on the matching degree evaluation results of each level, the consistency penalty value, the relevance probability and the target threshold.

[0100] There are no restrictions on how the deterministic level is generated. For example, the deterministic level can be determined according to predetermined rules, such as the greater the difference between the matching evaluation results of each level, the higher the uncertainty; the greater the consistency penalty value, the higher the uncertainty; and the closer the correlation probability is to the target threshold, the higher the uncertainty.

[0101] There are no restrictions on how evidence information is generated. For example, evidence information can be generated by combining rules with model output. Evidence information can be in structured formats such as JSON (JavaScript Object Notation) for easy front-end display. Evidence information may include phrases such as "good consistency across levels" (relevant) or "relevant only to the global topic, irrelevant to paragraph content" (irrelevant). The specific content included can be determined according to actual needs. In addition, when relevant, the most relevant paragraph content can also be displayed.

[0102] In some embodiments of this disclosure, the level of certainty may include high confidence, medium confidence, and low confidence. After determining the correlation between the image and the target document, in response to determining that the level of certainty is high confidence, the basic judgment result, the correlation probability, and evidence information can be output (e.g., fed back to the user). In response to determining that the level of certainty is medium confidence or low confidence, the manual review result of the target document can be obtained. In response to determining that no modification is needed to the basic judgment result and evidence information based on the manual review result, the basic judgment result, the correlation probability, and evidence information can be output. In response to determining that modification is needed to the basic judgment result and / or evidence information based on the manual review result, the unmodified content and the modified content in the basic judgment result, the correlation probability, and evidence information can be output.

[0103] If the certainty level is high confidence, the relevance determination result is relatively reliable. Therefore, the basic determination result, relevance probability, and evidence information can be directly output. If necessary, the certainty level can also be output simultaneously. If the certainty level is medium confidence or low confidence, the relevance determination result is not very reliable. Therefore, manual review is required. If the manual review determines that the relevance determination result is correct, the basic determination result, relevance probability, and evidence information can be output. If necessary, the certainty level can also be output simultaneously. If the manual review determines that the relevance determination result has problems, the basic determination result and / or evidence information can be modified based on the manual review result. Assuming that both the basic determination result and evidence information have been modified, the modified basic determination result, relevance probability, and modified evidence information can be output. If necessary, the certainty level can also be output simultaneously.

[0104] By leveraging uncertainty levels, decision-making risks can be automatically classified, high-confidence results can be directly output to improve processing efficiency, while guiding manual processing of medium-confidence and low-confidence results. Evidence information provides transparent judgment criteria and traceable explanatory information, which not only enhances the credibility and acceptability of relevance judgment results, but also significantly reduces the understanding cost and operational threshold of manual review, achieving efficient human-machine collaboration.

[0105] In addition, manual review can compensate for the shortcomings of models at the "cognitive edge," correct errors, and further improve the accuracy of the relevance judgment results fed back to users.

[0106] In some embodiments of this disclosure, after determining the relevance between the image and the target document, in response to determining that the hard example mining trigger condition is met, hard example mining can be performed based on the feedback results of image-text relevance detection conducted within the most recent predetermined time period. Training samples can be constructed based on the hard example mining results, and then the training samples can be used to optimize the various models involved in the image-text relevance detection process. These models may include cross-modal matching network models, weighted fusion models, threshold prediction models, relevance evaluation models, etc. The specific value of the predetermined time period can be determined according to actual needs.

[0107] The triggering conditions for difficult case mining can refer to the passage of a predetermined period or the collection of a predetermined number of feedback results. During online service, feedback information such as the re-evaluation results of the operation / strategy system, cases of image-text discrepancies in user complaints / negative reviews, and cases of problems discovered by manual review can be collected. Difficult cases can be mined from these, such as high-confidence errors, boundary cases close to the target threshold, and cases with high consistency penalty values ​​but still judged as relevant or irrelevant. Cluster analysis can be performed on difficult cases to identify common error patterns, and corresponding training samples can be constructed. These training samples can then be used to optimize the various models involved in the image-text relevance detection process, thereby achieving an online learning and feedback loop and continuously improving model performance.

[0108] In addition, the constructed training samples can be mixed with some historical samples to form training samples for optimizing each model, thereby preserving historical knowledge and improving the adaptability of new samples.

[0109] Online learning strategies can be selected based on actual resources. For example, when resources are sufficient, online incremental updates can be used, while when resources are limited, it can be simplified to "difficult example sampling + periodic full retraining". In addition, incremental updates can use small batches and low learning rates to avoid the model overfitting to new samples.

[0110] Based on the above introduction, Figure 3 This is a flowchart of the second embodiment of the image-text relevance detection method described in this disclosure. Figure 3 As shown, the specific implementation methods are as follows.

[0111] In step 301, the target document to be processed is parsed to obtain the images and their corresponding structured context information. The structured context information may include: the chapter where the image is located, the context paragraph of the image, and the image caption information.

[0112] Assuming the target document is a product details page on an e-commerce platform, it may include HTML-formatted details text, main product image, detailed images, product title, attribute parameters (brand, model, specifications, etc.).

[0113] It can parse detailed text, extract chapter structure and paragraphs under each chapter, and determine the chapter where each image is located, the context paragraph of each image, and the image caption information.

[0114] In step 302, the hierarchical text vector corresponding to the target document is obtained, including: the target paragraph vector of each paragraph in the target document, the target chapter vector of each chapter in the target document, and the target document vector of the target document.

[0115] For example, the basic paragraph vector of each paragraph in the target document can be determined separately. Based on the basic paragraph vectors of each paragraph belonging to the same chapter, the initial chapter vector of each chapter in the target document can be determined separately. Based on the initial chapter vector, the initial document vector of the target document can be determined separately. Based on the initial document vector, the basic paragraph vectors can be adjusted to obtain the initial paragraph vector of each paragraph.

[0116] Then, the initial paragraph vector, initial chapter vector, and initial document vector can be directly determined as the target paragraph vector, target chapter vector, and target document vector. Alternatively, the domain to which the target document belongs can be determined, and the domain vector corresponding to the domain can be obtained. Then, the initial paragraph vector, initial chapter vector, and initial document vector can be adjusted according to the domain vector to obtain the target paragraph vector, target chapter vector, and target document vector respectively.

[0117] In step 303, each image in the target document is processed according to the methods shown in steps 304-311.

[0118] In step 304, the target visual vector corresponding to the image is determined.

[0119] For example, the global image vector of an image can be obtained, and salient regions can be detected in the image to obtain the region vectors of each detected salient region. Furthermore, text information in the image can be recognized to obtain the information vector of the recognized text information. Then, the global image vector, region vector, and information vector can be fused to obtain the initial visual vector. The initial visual vector can then be directly determined as the target visual vector, or the initial visual vector can be adjusted based on the neighborhood vector to obtain the target visual vector.

[0120] In step 305, the first-level matching degree evaluation result is determined based on the target document vector and the target visual vector; the second-level matching degree evaluation result is determined based on the target chapter vector and the target visual vector of the chapter in which the image is located; the third-level matching degree evaluation result is determined based on the target paragraph vector and the target visual vector of each context paragraph; and the fourth-level matching degree evaluation result is determined based on the caption vector corresponding to the caption information, the information vector corresponding to the text information in the image, and the target visual vector.

[0121] In step 306, the consistency penalty value corresponding to the image is obtained, and the comprehensive matching evaluation result is determined based on the matching degree evaluation results of each level and the consistency penalty value.

[0122] In step 307, the target threshold corresponding to the target document is determined based on the matching degree evaluation results of each level and the domain vector corresponding to the domain to which the target document belongs.

[0123] In step 308, the comprehensive matching degree evaluation results, consistency penalty value, target threshold and matching degree evaluation results at each level are fused to obtain target reference information. Based on the target reference information, the basic judgment result and relevance probability are determined, and the certainty level and evidence information are determined based on the matching degree evaluation results at each level, consistency penalty value, relevance probability and target threshold.

[0124] In step 309, it is determined whether the certainty level is high confidence. If so, step 310 is executed; otherwise, step 311 is executed.

[0125] The level of certainty can be high confidence, medium confidence, or low confidence.

[0126] In step 310, the basic judgment result, relevance probability, and evidence information are output.

[0127] In step 311, the manual review results of the target document are obtained, and the modified basic judgment results, relevance probability and modified evidence information are output based on the manual review results.

[0128] In this embodiment, it is assumed that both the basic judgment result and the evidence information have been modified.

[0129] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure. Furthermore, for parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0130] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0131] Figure 4 This is a schematic diagram of the structural composition of Embodiment 400 of the image-text correlation detection device described in this disclosure. Figure 4 As shown, it includes: a first processing module 401 and a second processing module 402.

[0132] The first processing module 401 is used to parse the target document to be processed and obtain the images and the corresponding structured context information of the images.

[0133] The second processing module 402 is used to obtain the hierarchical text vector corresponding to the target document, and for any image, determine the target visual vector corresponding to the image, and determine the correlation judgment result between the image and the target document based on the hierarchical text vector, the target visual vector and the structured context information.

[0134] In some embodiments of this disclosure, the structured context information corresponding to any image may include: the chapter in which the image is located, the context paragraph of the image, and the image caption information. The context paragraph includes M adjacent paragraphs before the image and N adjacent paragraphs after the image, where M and N are both positive integers.

[0135] In some embodiments of this disclosure, the hierarchical text vector may include: target paragraph vectors for each paragraph in the target document, target chapter vectors for each chapter in the target document, and target document vectors for the target document. Correspondingly, the second processing module 402 may obtain the hierarchical text vectors corresponding to the target document by: obtaining the initial paragraph vectors for each paragraph, the initial chapter vectors for each chapter, and the initial document vector for the target document, and determining the target paragraph vector, target chapter vector, and target document vector based on the initial paragraph vectors, initial chapter vectors, and initial document vectors.

[0136] In some embodiments of this disclosure, the second processing module 402 can determine the basic paragraph vector of each paragraph, and can determine the initial chapter vector of each chapter based on the basic paragraph vectors belonging to the same chapter, and can determine the initial document vector based on the initial chapter vector. In addition, it can adjust each basic paragraph vector based on the initial document vector to obtain the initial paragraph vector of each paragraph.

[0137] In some embodiments of this disclosure, the second processing module 402 can directly determine each initial paragraph vector, each initial chapter vector, and the initial document vector as each target paragraph vector, each target chapter vector, and the target document vector. Alternatively, it can determine the domain to which the target document belongs and obtain the domain vector corresponding to the domain. Then, it can adjust each initial paragraph vector, each initial chapter vector, and the initial document vector according to the domain vector to obtain each target paragraph vector, each target chapter vector, and the target document vector respectively.

[0138] For any image in the target document, the corresponding target visual vector can be determined. In some embodiments of this disclosure, the second processing module 402 can obtain the global image vector of the image, perform salient region detection on the image, obtain the region vector of each detected salient region, and recognize the text information in the image to obtain the information vector of the recognized text information. Then, the global image vector, region vector, and information vector can be fused to obtain an initial visual vector, and the target visual vector can be determined based on the initial visual vector.

[0139] In some embodiments of this disclosure, the second processing module 402 may directly determine the initial visual vector as the target visual vector, or it may adjust the initial visual vector according to the neighborhood vector to obtain the target visual vector.

[0140] For any given image, after obtaining the hierarchical text vector and the target visual vector, the second processing module 402 can determine the correlation between the image and the target document based on the hierarchical text vector, the target visual vector, and the structured context information corresponding to the image.

[0141] In some embodiments of this disclosure, the second processing module 402 may first determine the comprehensive matching degree evaluation result between the image and the target document based on the hierarchical text vector, the target visual vector and the structured context information, and then determine the relevance determination result based on the comprehensive matching degree evaluation result.

[0142] In some embodiments of this disclosure, the second processing module 402 may determine the comprehensive matching degree evaluation result between the image and the target document in the following ways: determining a first-level matching degree evaluation result based on the target document vector and the target visual vector; determining a second-level matching degree evaluation result based on the target chapter vector and the target visual vector of the chapter in which the image is located; determining a third-level matching degree evaluation result based on the target paragraph vector and the target visual vector of each context paragraph; determining a fourth-level matching degree evaluation result based on the caption vector, information vector, and target visual vector corresponding to the caption information; and determining a comprehensive matching degree evaluation result based on the matching degree evaluation results of each level.

[0143] Specifically, in some embodiments of this disclosure, the second processing module 402 may obtain a first correlation between the target document vector and the target visual vector, and determine the first correlation as the first-level matching degree evaluation result; and / or, obtain a second correlation between the target chapter vector of the chapter where the image is located and the target visual vector, and determine the second correlation as the second-level matching degree evaluation result; and / or, obtain a third correlation between the target paragraph vector and the target visual vector of each context paragraph, and determine the maximum value among the third correlations as the third-level matching degree evaluation result; and / or, obtain a fourth correlation between the caption vector and the target visual vector, and obtain a fifth correlation between the information vector (the vector corresponding to the text information identified from the image) and the target visual vector, and weight the fourth correlation and the fifth correlation to obtain a fourth-level matching degree evaluation result.

[0144] In addition, in some embodiments of this disclosure, the second processing module 402 can obtain the consistency penalty value corresponding to the image and the fusion result of the matching degree evaluation results at each level. Then, the comprehensive matching degree evaluation result can be determined based on the fusion result and the consistency penalty value.

[0145] In some embodiments of this disclosure, the second processing module 402 may obtain the consistency penalty value corresponding to the image by: obtaining the average of the second-level matching degree evaluation result and the third-level matching degree evaluation result, obtaining the first difference between the average and the first tolerance threshold, obtaining the absolute value of the second difference between the second-level matching degree evaluation result and the third-level matching degree evaluation result, and determining the consistency penalty value based on the comparison result between the absolute value and the second tolerance threshold and the comparison result between the first-level matching degree evaluation result and the first difference.

[0146] In some embodiments of this disclosure, the relevance determination result may include: a basic determination result and a relevance probability. The basic determination result is used to indicate whether there is a relevance between the image and the target document, and the relevance probability is used to indicate the relevance probability value between the image and the target document. Accordingly, the second processing module 402 may determine the relevance determination result based on the comprehensive matching degree evaluation result in the following ways: determining the target threshold corresponding to the target document based on the matching degree evaluation results at each level and the domain vector; fusing the comprehensive matching degree evaluation result, the consistency penalty value, the target threshold, and the matching degree evaluation results at each level to obtain target reference information; and determining the basic determination result and the relevance probability based on the target reference information.

[0147] In some embodiments of this disclosure, the relevance determination result may further include: a certainty level and evidence information. The evidence information is used to explain the reason for giving the basic determination result, and the certainty level is used to explain the certainty of the basic determination result. Accordingly, the second processing module 402 may determine the relevance determination result based on the comprehensive matching degree evaluation result by: determining the certainty level and evidence information based on the matching degree evaluation results of each level, the consistency penalty value, the relevance probability and the target threshold.

[0148] In some embodiments of this disclosure, the level of certainty may include high confidence, medium confidence, and low confidence. After the second processing module 402 determines the correlation judgment result between the image and the target document, in response to determining that the level of certainty is high confidence, it can output the basic judgment result, the correlation probability, and the evidence information. In response to determining that the level of certainty is medium confidence or low confidence, it can obtain the manual review result of the target document. In response to determining that no modification is needed to the basic judgment result and the evidence information based on the manual review result, it can output the basic judgment result, the correlation probability, and the evidence information. In response to determining that modification is needed to the basic judgment result and / or the evidence information based on the manual review result, it can output the unmodified content and the modified content in the basic judgment result, the correlation probability, and the evidence information.

[0149] In some embodiments of this disclosure, each of the aforementioned relevance degrees can be calculated using a cross-modal matching network model, and / or, the second processing module 402 may obtain the fusion result of the matching degree evaluation results at each level by using a weighted fusion model to fuse the matching degree evaluation results at each level to obtain the fusion result, and / or, the second processing module 402 may determine the target threshold corresponding to the target document based on the matching degree evaluation results at each level and the domain vector by using a threshold prediction model to generate the target threshold based on the matching degree evaluation results at each level and the domain vector, and / or, the second processing module 402 may determine the basic judgment result and relevance probability based on the target reference information by using a relevance evaluation model to generate the basic judgment result and relevance probability based on the target reference information.

[0150] In addition, in some embodiments of this disclosure, after the second processing module 402 determines the correlation determination result between the image and the target document, in response to determining that the hard case mining trigger condition is met, it can also perform hard case mining based on the feedback result corresponding to the image-text correlation detection performed within the most recent predetermined time period, construct training samples based on the hard case mining results, and optimize each model using the training samples.

[0151] The specific workflow of the above-described device embodiments can be found in the relevant descriptions in the foregoing method embodiments, and will not be repeated here.

[0152] In summary, the solution described in this disclosure can improve the accuracy and interpretability of the relevance determination results, reduce manual review costs, and enable continuous optimization and rapid iteration of the entire image-text relevance detection process. Moreover, the solution described in this disclosure is applicable to various scenarios and has broad applicability. For example, it can be applied to the consistency review of product images and text on e-commerce platforms to prevent misleading consumers with discrepancies between images and text, thereby enhancing the credibility of product information on the platform. It can also be applied to the image-text review of news content platforms to prevent clickbait headlines or misleading images, improve content quality, and reduce the cost of manual content review. Furthermore, it can be applied to the compliance review of advertising placement, i.e., to review the relevance of advertising creative images with landing page copy and promotional keywords to ensure the authenticity and compliance of advertising content. Additionally, it can be applied to search and recommendation ranking scenarios to optimize the display of recommendation lists and improve the click-through rate of recommended content. Finally, it can be applied to enterprise knowledge base management scenarios, such as verifying the consistency between images in technical documents and product manuals and the corresponding chapter content, thereby improving the accuracy and efficiency of the knowledge base.

[0153] The solutions described in this disclosure can be applied to the field of artificial intelligence, particularly natural language processing, deep learning, and computer vision. Artificial intelligence is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It involves both hardware and software technologies. Artificial intelligence hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0154] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0155] Figure 5 A schematic block diagram of an electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0156] like Figure 5As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the electronic device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0157] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0158] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as those described in this disclosure. For example, in some embodiments, the methods described in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the methods described in this disclosure can be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the methods described herein by any other suitable means (e.g., by means of firmware).

[0159] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0160] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0161] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0162] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0163] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0164] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0165] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0166] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for detecting the relevance of images and text, comprising: The target document to be processed is parsed to obtain the images and the corresponding structured context information of the images; Obtain the hierarchical text vector corresponding to the target document; For any given image, the target visual vector corresponding to the image is determined, and based on the hierarchical text vector, the target visual vector, and the structured context information, the correlation determination result between the image and the target document is determined.

2. The method according to claim 1, wherein, The hierarchical text vector includes: the target paragraph vector of each paragraph in the target document, the target chapter vector of each chapter in the target document, and the target document vector of the target document; The step of obtaining the hierarchical text vector corresponding to the target document includes: Obtain the initial paragraph vector for each paragraph, the initial chapter vector for each chapter, and the initial document vector for the target document, respectively. The target paragraph vector, the target chapter vector, and the target document vector are determined based on the initial paragraph vector, the initial chapter vector, and the initial document vector.

3. The method according to claim 2, wherein, The steps of obtaining the initial paragraph vector for each paragraph, the initial chapter vector for each chapter, and the initial document vector for the target document include: Determine the basic paragraph vector for each paragraph; Based on the basic paragraph vectors belonging to the same chapter, the initial chapter vectors for each chapter are determined respectively; The initial document vector is determined based on the initial chapter vector; The initial paragraph vectors of each paragraph are adjusted based on the initial document vector to obtain the initial paragraph vector of each paragraph.

4. The method according to claim 2, wherein, The step of determining the target visual vector corresponding to the image includes: Obtain the global image vector of the image; Perform salient region detection on the image and obtain the region vector of each detected salient region; The text information in the image is recognized, and the information vector of the recognized text information is obtained; The global image vector, the region vector, and the information vector are fused to obtain an initial visual vector; The target visual vector is determined based on the initial visual vector.

5. The method according to claim 4, wherein, The step of determining the target paragraph vector, the target chapter vector, and the target document vector based on the initial paragraph vector, the initial chapter vector, and the initial document vector includes: The initial paragraph vector, the initial chapter vector, and the initial document vector can be directly determined as the target paragraph vector, the target chapter vector, and the target document vector. Alternatively, the domain to which the target document belongs can be determined, and the domain vector corresponding to the domain can be obtained. The initial paragraph vector, the initial chapter vector, and the initial document vector can be adjusted according to the domain vector to obtain the target paragraph vector, the target chapter vector, and the target document vector. The step of determining the target visual vector based on the initial visual vector includes: The initial visual vector can be directly determined as the target visual vector, or the initial visual vector can be adjusted according to the neighborhood vector to obtain the target visual vector.

6. The method according to claim 4, wherein, The step of determining the relevance between the image and the target document based on the hierarchical text vector, the target visual vector, and the structured context information includes: Based on the hierarchical text vector, the target visual vector, and the structured context information, the comprehensive matching degree evaluation result between the image and the target document is determined; The relevance determination result is determined based on the comprehensive matching degree evaluation result.

7. The method according to claim 6, wherein, The structured context information includes: the chapter in which the image is located, the context paragraph of the image, and the caption information of the image. The context paragraph includes M adjacent paragraphs before the image and N adjacent paragraphs after the image, where M and N are both positive integers.

8. The method according to claim 7, wherein, The comprehensive matching degree evaluation result between the image and the target document includes: The first-level matching degree evaluation result is determined based on the target document vector and the target visual vector; The second-level matching degree evaluation result is determined based on the target chapter vector of the chapter in which the image is located and the target visual vector; The third-level matching degree evaluation result is determined based on the target paragraph vector and the target visual vector of each context paragraph; The fourth-level matching degree evaluation result is determined based on the caption vector corresponding to the caption information, the information vector, and the target visual vector. The overall matching degree assessment result is determined based on the matching degree assessment results of each level.

9. The method according to claim 8, wherein, The step of determining the first-level matching degree evaluation result based on the target document vector and the target visual vector includes: obtaining a first correlation between the target document vector and the target visual vector, and determining the first correlation as the first-level matching degree evaluation result; And / or, determining the second-level matching degree evaluation result based on the target chapter vector of the chapter where the image is located and the target visual vector includes: obtaining the second correlation between the target chapter vector of the chapter where the image is located and the target visual vector, and determining the second correlation as the second-level matching degree evaluation result; And / or, determining the third-level matching degree evaluation result based on the target paragraph vector and the target visual vector of each context paragraph includes: obtaining the third relevance between the target paragraph vector and the target visual vector of each context paragraph, and determining the maximum value among the third relevance values ​​as the third-level matching degree evaluation result; And / or, determining the fourth-level matching degree evaluation result based on the image annotation vector corresponding to the image annotation information, the information vector, and the target visual vector includes: obtaining the fourth correlation between the image annotation vector and the target visual vector, obtaining the fifth correlation between the information vector and the target visual vector, and weighting and summing the fourth correlation and the fifth correlation to obtain the fourth-level matching degree evaluation result.

10. The method according to claim 9, wherein, The determination of the comprehensive matching degree evaluation result based on the matching degree evaluation results of each level includes: Obtain the consistency penalty value corresponding to the image; Obtain the fusion result of the matching degree evaluation results at each level; The overall matching degree evaluation result is determined based on the fusion result and the consistency penalty value.

11. The method according to claim 10, wherein, The process of obtaining the consistency penalty value corresponding to the image includes: Obtain the mean of the second-level matching degree evaluation result and the third-level matching degree evaluation result, and obtain the first difference between the mean and the first tolerance threshold; Obtain the absolute value of the second difference between the second-level matching degree evaluation result and the third-level matching degree evaluation result; The consistency penalty value is determined based on the comparison between the absolute value and the second tolerance threshold, and the comparison between the first level matching degree evaluation result and the first difference.

12. The method according to claim 10, wherein, The relevance determination result includes: a basic determination result and a relevance probability. The basic determination result is used to indicate whether the image and the target document are related, and the relevance probability is used to indicate the relevance probability value between the image and the target document. The determination of the relevance judgment result based on the comprehensive matching degree evaluation result includes: The target threshold corresponding to the target document is determined based on the matching degree evaluation results of each level and the domain vector. The comprehensive matching degree evaluation result, the consistency penalty value, the target threshold, and the matching degree evaluation results at each level are fused to obtain target reference information. Based on the target reference information, the basic judgment result and the relevance probability are determined.

13. The method according to claim 12, wherein, The relevance determination result also includes: a certainty level and evidence information, wherein the evidence information is used to explain the reason for giving the basic determination result, and the certainty level is used to explain the certainty of the basic determination result; The step of determining the relevance judgment result based on the comprehensive matching degree evaluation result further includes: Based on the matching degree evaluation results at each level, the consistency penalty value, the relevance probability, and the target threshold, the certainty level and the evidence information are determined.

14. The method according to claim 13, wherein, The levels of certainty include high confidence, medium confidence, and low confidence; The method further includes: After determining the correlation between the image and the target document, in response to determining the certainty level as the high confidence level, the basic determination result, the correlation probability, and the evidence information are output. In response to determining that the certainty level is the medium confidence or the low confidence, the manual review result for the target document is obtained. In response to determining, based on the manual review result, that no modification is required to the basic judgment result and the evidence information, the basic judgment result, the relevance probability, and the evidence information are output. In response to determining, based on the manual review result, that modification is required to the basic judgment result and / or the evidence information, the unmodified content and the modified content in the basic judgment result, the relevance probability, and the evidence information are output.

15. The method according to claim 12, wherein, All correlation coefficients were calculated using a cross-modal matching network model. And / or, the fusion result of obtaining the matching degree evaluation results of each level includes: fusing the matching degree evaluation results of each level using a weighted fusion model to obtain the fusion result; And / or, determining the target threshold corresponding to the target document based on the matching degree evaluation results of each level and the domain vector includes: generating the target threshold using a threshold prediction model based on the matching degree evaluation results of each level and the domain vector; And / or, determining the basic judgment result and the correlation probability based on the target reference information includes: generating the basic judgment result and the correlation probability using a correlation evaluation model based on the target reference information.

16. The method of claim 15, further comprising: After determining the correlation between the image and the target document, in response to determining that the hard example mining trigger condition is met, hard example mining is performed based on the feedback results of the image-text correlation detection performed within the most recent predetermined time period. Training samples are constructed based on the hard example mining results, and the training samples are used to optimize each model.

17. A device for detecting the correlation between images and text, comprising: First processing module and second processing module; The first processing module is used to parse the target document to be processed to obtain the images therein and the structured context information corresponding to the images; The second processing module is used to obtain the hierarchical text vector corresponding to the target document, and for any image, determine the target visual vector corresponding to the image, and determine the correlation judgment result between the image and the target document based on the hierarchical text vector, the target visual vector and the structured context information.

18. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-16.

19. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-16.

20. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the method of any one of claims 1-16.

Images