A method for image retrieval based on NLP complex sentence segmentation

Abstract

A kind of method for image retrieval based on NLP complex sentence segmentation, comprising: 1) complex sentence is segmented;2) multiple simple sentences are sorted;3) abstract sentence for retrieval image;4) query pruning;5) retrieve image;6) evaluate results: the performance of the method is evaluated using accuracy (Accuracy), precision (Precision), recall (Recall) and F1_score.The present application combines NLP with image retrieval, makes full use of the advantages of NLP, carries out detailed component syntax analysis and dependency syntax analysis on the complex sentence proposed by the user, inputs the sentence processed by analysis into the database as query sentence to retrieve image, not only greatly improves the effect of image retrieval, but also expands the application range of NLP.

Description

Technical Field

[0001] This invention belongs to the fields of natural language processing and computer vision, and is an application technology that has received high attention in these fields. Specifically, it relates to a method and application of image retrieval based on NLP complex sentence segmentation. Background Technology

[0002] Natural Language Processing (NLP) focuses on human-computer language interaction, studying how to process and utilize natural language. It was first applied to the Turing Test and has evolved from rule-based research methods to current statistical models and methods. It has also progressed from early traditional machine learning methods based on high-dimensional sparse features to the mainstream deep learning methods that use low-dimensional dense vector features based on neural networks to train models. In recent years, with the development of machine learning and related technologies, NLP research has achieved breakthrough after breakthrough, and researchers are designing various models and methods to solve various NLP problems.

[0003] Image retrieval technology research can be divided into two categories based on the different ways of describing image content: text-based image retrieval and content-based image retrieval. This invention focuses on the former. Text-based image retrieval technology originated in the 1970s. It uses text annotation to describe the content of images, thereby generating keywords for each image that describe its content. Users can provide search keywords based on their interests, and the retrieval system finds images that are annotated with those keywords, finally returning the search results to the user.

[0004] In the combination of traditional NLP and image retrieval, the first step is to formulate a simple question based on the image content, then analyze the semantics of the simple question, and finally derive the answer by combining it with the image content. Summary of the Invention

[0005] This invention aims to overcome the shortcomings of existing technologies and provide an image retrieval method based on NLP complex sentence segmentation.

[0006] The purpose of this invention is to fully integrate information from both Natural Language Processing (NLP) and Computer Vision (CVI) modalities, thereby providing an excellent solution for text-based image retrieval. This invention, in combining these two technologies, places greater emphasis on processing complex sentences. Instead of asking questions based on images, it searches for answers step-by-step in a database based on the given questions, thus realizing a complete architecture for image retrieval based on NLP complex sentence segmentation.

[0007] The objective of this invention is achieved as follows:

[0008] An image retrieval method based on NLP complex sentence segmentation includes the following steps:

[0009] 1) Complex Sentence Segmentation: Based on the complex sentence provided by the user, Part-of-Speech is used to assign part-of-speech tags to each word in the input sentence. This method eliminates ambiguity in word part-of-speech and finds the correct tag for each word. After obtaining the part-of-speech tags, Enhanced++Dependencies is used to obtain the dependency relationships between words. Combined with grammar, the sentence is finally segmented into multiple simple sentences.

[0010] 2) Sorting multiple simple sentences: Sort the multiple simple sentences obtained in 1). According to computational linguistics, a simple sentence consists of a subject, a predicate, and an object. Since the problem involved in this invention is a complex problem, there is continuity between the subject and the predicate. The simple sentences can be sorted according to their characteristics.

[0011] 3) Abstracting the sentences for retrieving images: In a knowledge graph, each entity in an image corresponds to a node. Therefore, we only need to traverse and iterate through the nodes involved in the above sentences to determine the first node for the query. First, we query the first simple sentence to obtain the query result (intermediate result of the compound question). At this point, there may be many results. To ensure the accuracy of the final query, we need to use each result as the first entity node of the next simple sentence for querying. This process is repeated until the final answer to the compound question is obtained.

[0012] 4) Query Pruning: This invention relates to the problem of segmenting complex sentences into simple sentences in English. In different question types, some may involve keywords such as "most", "least", and "second". If these keywords appear in the first simple sentence of the query, it is necessary to perform a statistical analysis on the query results to obtain the answer that best matches this sentence question (intermediate result of the complex question). At this time, there is often only one result. It is only necessary to use this single result as the initial node for the next simple sentence query. Compared with a similar process in 3), this can greatly reduce the retrieval volume of the next step, hence it is called the pruning process.

[0013] 5) Image retrieval: First, input the simple sentence with query order 1 into the database for querying. Use the obtained query result as the initial node for the next simple sentence to be queried, and continue to query the database, and so on, until the answer of the final complex sentence is obtained.

[0014] 6) Evaluation results: The performance of the method is evaluated using accuracy, precision, recall, and F1 score.

[0015] Preferably, in step 1), segmenting the complex sentence includes the following process:

[0016] Step 101, Part-of-Speech Tagging Stage: Part-of-Speech tagging is performed using a Hidden Markov Model (HMM). The HMM calculates the probability distribution of possible tag sequences and selects the optimal tag sequence. For the HMM model, we first assume that Q is the set of all possible hidden states and V is the set of all possible observed states, i.e.:

[0017] Q={q1,q2,...,qN},V={v1,v2,...vM} (1)

[0018] Where N is the number of possible hidden states and M is the number of all possible observed states.

[0019] For a sequence of length T, I represents the state sequence, and O represents the observation sequence, i.e.:

[0020] I={i1,i2,...,iT},O={o1,o2,...oT} (2)

[0021] Wherein, for any hidden state it∈Q, and for any observed state ot∈V

[0022] The Hidden Markov Model (HMM) makes two very important assumptions as follows:

[0023] (1) Homogeneous Markov Chain Assumption. That is, the hidden state at any time depends only on its preceding hidden state. Of course, this assumption is a bit extreme, because often a certain hidden state depends not only on the preceding hidden state, but also on the preceding two or three. However, the advantage of this assumption is that the model is simple and easy to solve. If the hidden state at time t is it = qi, and the hidden state at time t+1 is it+1 = qj, then the HMM state transition probability aij from time t to time t+1 can be expressed as:

[0024] aij=P(it+1=qj|it=qi) (3)

[0025] Thus, aij can form the state transition matrix A of a Markov chain:

[0026] A=[aij]N×N (4)

[0027] (2) Observation Independence Assumption. This means that the observed state at any given time depends only on the hidden state at that time. This is also an assumption made to simplify the model. If the hidden state at time t is it = qj, and the corresponding observed state is ot = vk, then the probability of the observed state vk being generated under the hidden state qj at that time is bj(k), satisfying:

[0028] bj(k)=P(ot=vk|it=qj) (5)

[0029] Thus, bj(k) can form the probability matrix B for generating the observed states:

[0030] B=[bj(k)]N×M (6)

[0031] In addition, we need a set of hidden state probability distributions Π at time t=1:

[0032] Π=[π(i)]N (7)

[0033] Where π(i) = P(i1 = qi)

[0034] An Hidden Mirror Model (HMM) can be determined by the initial probability distribution Π of the hidden states, the state transition probability matrix A, and the observation state probability matrix B. Π and A determine the state sequence, and B determines the observation sequence. Therefore, an HMM model can be represented by a triple λ as follows:

[0035] λ=(A,B,Π) (8)

[0036] Step 102, Dependency Parsing Stage: The dependency relation representation is a labeled directed graph, where nodes are lexical items and labeled arcs represent dependencies, from the headword to the dependency.

[0037] The four axioms of dependency relations in dependency syntax:

[0038] (1) Only one component in a sentence is independent;

[0039] (2) Other components directly depend on a certain component;

[0040] (3) No single component can depend on two or more components;

[0041] (4) If component A directly depends on component B, and component C is located between A and B in the sentence, then C either directly depends on B, or directly depends on a component between A and B.

[0042] Preferably, in step 3), abstracting the sentence of the retrieved image includes the following process:

[0043] Step 301, Preparation stage: The query database used in this invention is a fusion of data abstracted from some images of ConceptNet and COCO, and the Neo4j high-performance graph engine is used.

[0044] Step 302, Query Phase: First, input the simple sentence with query order 1 into Neo4j for querying. Use the obtained query result as the initial node for the next simple sentence to be queried, and so on, until the answer of the final complex sentence is obtained.

[0045] Preferably, in step 5), the image retrieval process includes:

[0046] To describe the image retrieval process in more detail, we divide the complex sentence responses into two main categories. One category is the judgmental response format. Similarly, we first analyze the segmented simple sentences sequentially, query the database, and retrieve the results at each stage. The final answer is either "Yes" or "No" based on whether a result matching the question is found. The other category differs from the first in that it retrieves the corresponding image based on the final result matching the question.

[0047] Preferably, in step 6), the evaluation result process includes:

[0048] To quantitatively evaluate the performance of image retrieval tasks, accuracy (Acc) and precision (Prec) are used. i ), recall rate (Rc i F1 score and F1 score are used as the benchmarks:

[0049]

[0050]

[0051] Among them, TP i TN i FP i FN i These represent the true positive, true negative, false positive, and false negative values ​​for the i-th label in the label set, respectively.

[0052] The advantages and positive effects of this invention are:

[0053] This invention combines NLP with image retrieval, making full use of the advantages of NLP to perform detailed component syntax analysis and dependency syntax analysis on the complex sentences proposed by users. The analyzed and processed sentences are then used as query statements and input into the database to retrieve images. This not only greatly improves the image retrieval effect but also expands the application scope of NLP. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of an image retrieval method and application process based on NLP complex sentence segmentation provided by some embodiments of the present invention.

[0055] Figure 2This is a schematic diagram of the Part-of-Speech of the compound sentence of this invention.

[0056] Figure 3 This is a schematic diagram of the compound sentence Enhanced++Dependencies of this invention. Detailed Implementation

[0057] To make the technical solution and design concept of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0058] This paper presents an image retrieval method and application based on NLP-based complex sentence segmentation. The method analyzes a user-generated question using NLP, queries a database for relevant images, and a knowledge base containing the necessary information to answer the question. Finally, it searches the database for the most suitable answer. This task involves both Natural Language Processing and Computer Vision. Insufficient or inadequate consideration of information from either domain can lead to low information utilization and low accuracy in image retrieval. Therefore, fully utilizing information from various modalities is crucial.

[0059] The database used in this invention comes from two sources. One source is external knowledge, specifically ConcepNet. The other source is a portion of the MS-COCO dataset. The SGG model is used to abstract images from MS-COCO into a specific data structure, which is then fused with ConceptNet. The high-performance graph engine Neo4j is used for this fusion, and the fused data serves as the database for this invention.

[0060] The specific implementation process of this invention is as follows:

[0061] 1) Segmenting Complex Sentences: Based on the complex sentences provided by the user, Part-of-Speech is used to assign part-of-speech tags to each word in the input sentence. This method eliminates ambiguity in word part-of-speech and finds the correct tags for each word. After obtaining the part-of-speech tags, Enhanced++Dependencies is used to obtain the dependency relationships between words. Combined with grammar, the sentences are finally segmented into multiple simple sentences.

[0062] 2) Sort multiple simple sentences: Sort the multiple simple sentences obtained in 1). According to computational linguistics, a simple sentence consists of a subject, a predicate, and an object. Since the problem involved in this invention is a complex problem, there is continuity between the subject and the predicate. The simple sentences can be sorted according to their characteristics.

[0063] 3) Abstracting the sentences for retrieving images: In a knowledge graph, each entity in an image corresponds to a node. Therefore, we only need to traverse and iterate through the nodes involved in the above sentences to determine the first node for the query. First, we query the first simple sentence to obtain the query result (intermediate result of the compound question). At this point, there may be many results. To ensure the accuracy of the final query, we need to use each result as the first entity node of the next simple sentence for querying. This process is repeated until the final answer to the compound question is obtained.

[0064] 4) Query Pruning: This invention relates to the problem of segmenting complex sentences into simple sentences in English. In different question types, some may involve keywords such as "most", "least", and "second". If these keywords appear in the first simple sentence of the query, it is necessary to perform a statistical analysis on the query results to obtain the answer that best matches this sentence question (intermediate result of the complex question). At this time, there is often only one result. It is only necessary to use this single result as the initial node for the next simple sentence query. Compared with a similar process in 3), this can greatly reduce the retrieval volume of the next step, hence it is called the pruning process.

[0065] 5) Image retrieval: First, input the simple sentence with query order 1 into the database for querying. Use the obtained query result as the initial node for the next simple sentence to be queried, and continue to query the database, and so on, until the answer of the final complex sentence is obtained.

[0066] 6) Evaluation results: The performance of the method is evaluated using accuracy, precision, recall, and F1 score.

[0067] Step 1) includes the following process:

[0068] Step 101, Part-of-Speech Tagging Stage: Part-of-Speech tagging is performed using a Hidden Markov Model (HMM). The HMM calculates the probability distribution of possible tag sequences and selects the optimal tag sequence. For the HMM model, we first assume that Q is the set of all possible hidden states and V is the set of all possible observed states, i.e.:

[0069] Q={q1,q2,...,qN},V={v1,v2,...vM} (11)

[0070] Where N is the number of possible hidden states and M is the number of all possible observed states.

[0071] For a sequence of length T, I represents the state sequence, and O represents the observation sequence, i.e.:

[0072] I={i1,i2,...,iT},O={o1,o2,...oT} (12)

[0073] Wherein, for any hidden state it∈Q, and for any observed state ot∈V

[0074] The Hidden Markov Model (HMM) makes two very important assumptions as follows:

[0075] (1) Homogeneous Markov Chain Assumption. That is, the hidden state at any time depends only on its preceding hidden state. Of course, this assumption is a bit extreme, because often a certain hidden state depends not only on the preceding hidden state, but also on the preceding two or three. However, the advantage of this assumption is that the model is simple and easy to solve. If the hidden state at time t is it = qi, and the hidden state at time t+1 is it+1 = qj, then the HMM state transition probability aij from time t to time t+1 can be expressed as:

[0076] aij=P(it+1=qj|it=qi) (13)

[0077] Thus, aij can form the state transition matrix A of a Markov chain:

[0078] A=[aij]N×N (14)

[0079] (2) Observation Independence Assumption. This means that the observed state at any given time depends only on the hidden state at that time. This is also an assumption made to simplify the model. If the hidden state at time t is it = qj, and the corresponding observed state is ot = vk, then the probability of the observed state vk being generated under the hidden state qj at that time is bj(k), satisfying:

[0080] bj(k)=P(ot=vk|it=qj) (15)

[0081] Thus, bj(k) can form the probability matrix B for generating the observed states:

[0082] B = [bj(k)]N×M (16)

[0083] In addition, we need a set of hidden state probability distributions Π at time t=1:

[0084] Π=[π(i)]N (17)

[0085] Where π(i) = P(i1 = qi)

[0086] An Hidden Mirror Model (HMM) can be determined by the initial probability distribution Π of the hidden states, the state transition probability matrix A, and the observation state probability matrix B. Π and A determine the state sequence, and B determines the observation sequence. Therefore, an HMM model can be represented by a triple λ as follows:

[0087] λ=(A,B,Π) (18)

[0088] Step 102, Dependency Parsing Stage: The dependency relation representation is a labeled directed graph, where nodes are lexical items and labeled arcs represent dependencies, from the headword to the dependency.

[0089] The four axioms of dependency relations in dependency syntax:

[0090] (1) Only one component in a sentence is independent;

[0091] (2) Other components directly depend on a certain component;

[0092] (3) No single component can depend on two or more components;

[0093] (4) If component A directly depends on component B, and component C is located between A and B in the sentence, then C either directly depends on B, or directly depends on a component between A and B.

[0094] Step 3) includes:

[0095] Step 301, Preparation stage: The query database used in this invention is a fusion of data abstracted from some images of ConceptNet and COCO, and the Neo4j high-performance graph engine is used.

[0096] Step 302, Query Phase: First, input the simple sentence with query order 1 into Neo4j for querying. Use the obtained query result as the initial node for the next simple sentence to be queried, and so on, until the answer of the final complex sentence is obtained.

[0097] Step 5) specifically includes:

[0098] To describe the image retrieval process in more detail, we divide the complex sentence responses into two main categories. One category is the judgmental response format. Similarly, we first analyze the segmented simple sentences sequentially, query the database, and retrieve the results at each stage. The final answer is either "Yes" or "No" based on whether a result matching the question is found. The other category differs from the first in that it retrieves the corresponding image based on the final result matching the question.

[0099] Step 6) includes the following process:

[0100] Step 601, Result Evaluation: To quantitatively evaluate the performance of the image retrieval task, accuracy (Acc) and precision (Prec) are used. i ), recall rate (Rc i F1 score and F1 score are used as the benchmarks:

[0101]

[0102]

[0103] Among them, TP i TN i FP i FN i These represent the true positive, true negative, false positive, and false negative values ​​for the i-th label in the label set, respectively.

[0104] The above description is merely a preferred embodiment of the present invention, which will be readily understood by those skilled in the art and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still make other modifications or variations based on the foregoing embodiments or drawings. It is impossible to exhaustively describe all implementation methods or technical solutions here, and all modifications, substitutions, etc., within the principles of the present invention should be included within the scope of protection claimed by the present invention.

Claims

1. A method for image retrieval based on NLP complex sentence segmentation, characterized in that, Includes the following steps: 1) Segmenting Complex Sentences: Based on the complex sentences provided by the user, Part-of-Speech is used to assign part-of-speech tags to each word in the input sentence. This method eliminates ambiguity in word parts of speech and finds the correct tags for each word. After obtaining the part-of-speech tags, Enhanced++ Dependencies is used to obtain the dependency relationships between words. Combined with grammar, the sentences are finally segmented into multiple simple sentences. 2) Sort multiple simple sentences: Sort the multiple simple sentences obtained in step 1); According to computer linguistics, a simple sentence consists of a subject, a predicate, and an object. Since the problem involved is a complex problem, there is continuity between the subject and the predicate. Sort the simple sentences according to their characteristics. 3) Abstracting the sentences for retrieving images: In a knowledge graph, each entity in an image corresponds to a node. Therefore, we only need to traverse and iterate through the nodes involved in the above sentences to determine the first node to be queried. First, we query the first simple sentence to obtain the query result, which is the intermediate result of the compound question. At this time, there will be many results. In order to ensure the accuracy of the final query, we need to use each result as the first entity node of the next simple sentence for querying. This process is repeated until the final answer to the compound question is obtained. 4) Query pruning: English complex sentences are divided into simple sentences. In different question types, keywords such as "most", "least", and "second" may be involved. If these keywords appear in the first simple sentence of the query, the query results need to be statistically analyzed to obtain the answer that best matches the question of this sentence, which is the intermediate result of the complex question. At this time, there is only one result. Only this result needs to be used as the initial node for the next simple sentence to query. Compared with the similar process in 3), this greatly reduces the retrieval volume of the next step, so it is called the pruning process. 5) Image retrieval: First, input the simple sentence with query order 1 into the database for querying. Use the obtained query result as the initial node for the next simple sentence to be queried, and continue to query the database, and so on, until the answer of the final complex sentence is obtained. 6) Evaluation Results: Using accuracy rate Accuracy Recall rate and The `_score` parameter is used to evaluate the method's performance; the evaluation process includes: To quantitatively evaluate the performance of image retrieval tasks, accuracy is used. Accuracy Recall rate and _score as a metric: （9） （10） in, , , , They represent the first in the tag set. True positive, true negative, false positive, and false negative for each label.

2. The image retrieval method based on NLP complex sentence segmentation as described in claim 1, characterized in that, Step 1) involves the following process of segmenting the complex sentence: Step 101, Part-of-Speech Tagging Stage: Part-of-Speech tagging is performed using a Hidden Markov Model (HMM). The HMM calculates the probability distribution of possible tag sequences and selects the optimal tag sequence. For the HMM model, it is first assumed that Q is the set of all possible hidden states and V is the set of all possible observed states, i.e.: Q={q1,q2,...,qN},V={v1,v2,...vM} (1) Where N is the number of possible hidden states, and M is the number of all possible observed states; For a sequence of length T, I represents the state sequence, and O represents the observation sequence, i.e.: I={i1,i2,...,iT},O={o1,o2,...oT} (2) Wherein, any hidden state it∈Q, and any observed state ot∈V; The Hidden Markov Model (HMM) makes two very important assumptions as follows: (1) Homogeneous Markov Chain Assumption: That is, the hidden state at any time depends only on its previous hidden state. Of course, this assumption is a bit extreme, but the advantage of this assumption is that the model is simple and easy to solve. If the hidden state at time t is it=qi and the hidden state at time t+1 is it+1=qj, then the HMM state transition probability aij from time t to time t+1 is expressed as: aij=P(it+1=qj|it=qi) (3) Thus, aij forms the state transition matrix A of the Markov chain: A=[aij]N×N (4) (2) Observation independence assumption; that is, the observed state at any time depends only on the hidden state at the current time. This is also an assumption to simplify the model. If the hidden state at time t is it=qj, and the corresponding observed state is ot=vk, then the probability of the observed state vk being generated under the hidden state qj at that time is bj(k), satisfying: bj(k)=P(ot=vk|it=qj) (5) Thus, bj(k) forms the probability matrix B for generating the observed state: B=[bj(k)]N×M (6) In addition, a set of hidden state probability distributions Π at time t=1 is needed: Π=[π(i)]N (7) Where π(i) = P(i1 = qi) An Hidden Mirror Model (HMM) is determined by the initial probability distribution Π of the hidden states, the state transition probability matrix A, and the observation state probability matrix B; Π and A determine the state sequence, and B determines the observation sequence; therefore, the HMM model is represented by a triple λ as follows: λ=(A,B,Π) (8) Step 102, Dependency parsing stage: The dependency relation representation is a labeled directed graph, where nodes are lexical items and labeled arcs represent dependencies, from the headword to the dependency. The four axioms of dependency relations in dependency syntax: (1) Only one component in a sentence is independent; (2) Other components directly depend on a certain component; (3) No single component can depend on two or more components; (4) If component A directly depends on component B, and component C is located between A and B in the sentence, then C either directly depends on B, or directly depends on a component between A and B.

3. The image retrieval method based on NLP complex sentence segmentation as described in claim 1, characterized in that, Step 3) involves abstracting the sentences retrieved from the images, which includes the following process: Step 301, the preparation stage, uses a query database that is a fusion of data from ConceptNet and COCO image abstraction, and uses the Neo4j high-performance graph engine. Step 302, Query Phase: First, input the simple sentence with query order 1 into Neo4j for querying. Use the obtained query result as the initial node for the next simple sentence to be queried, and so on, until the answer of the final complex sentence is obtained.

4. The image retrieval method based on NLP complex sentence segmentation as described in claim 1, characterized in that, Step 5) includes the image retrieval process as follows: To describe the image retrieval process in more detail, the response forms of complex sentences are divided into two main categories. One category is the judgment-based response form, in which the segmented simple sentences are analyzed in sequence, the database is queried, and the results of each stage are retrieved in turn. The final answer is either Yes or No, depending on whether a result matching the question is found. The other category differs from the first in that the corresponding image is obtained based on the final result matching the question.

Images