Natural language based search method and system

Abstract

The embodiment of the present application relates to the field of artificial intelligence technology, and specifically discloses a search method and system based on natural language, after determining original search object keywords and original search intention keywords from search text based on natural language processing, target search object keywords and target search intention keywords are obtained by expansion, a first candidate image set is obtained by determining images matching the target search object keywords from an image library, a second candidate image set is obtained by determining images matching the target search intention keywords from the first candidate image set, scores of images in the second candidate image set are calculated using the original search object keywords, the original search intention keywords, object labels and object description labels, and the N images with the highest scores in the second candidate image set are returned to the user as search results, the search intention analysis of the user is realized, the image scores are calculated according to the search intention, and the search results are more in line with the search intention of the user.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence, and particularly relates to a search method and system based on natural language. BACKGROUND

[0002] With the development of multimedia and big data technologies, users share various images on a big data platform, and other users can search the images shared by the users in a network.

[0003] Currently, when searching for images, a user mainly inputs text in a search box, expresses the images to be searched for through the text, and a search engine receives the text input by the user, performs word segmentation on the input text, and then searches for images based on the word segmentation and returns the images to the user, so that there are many images that do not conform to the search intention of the user in the images returned by the search engine. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a search method and system based on natural language, aiming to solve the problem that when searching for images in the background art, the search engine cannot recognize the search intention of the user, resulting in many images that do not conform to the search intention of the user in the search results.

[0005] To achieve the above-mentioned purpose, the embodiments of the present application provide the following technical solutions:

[0006] The search method based on natural language is used for searching for images, and the method specifically comprises the following steps:

[0007] Receiving a search text input by a user on an image search page;

[0008] Determining an original search object keyword and at least one original search intention keyword from the search text based on natural language processing;

[0009] Augmenting the original search object keyword and the original search intention keyword to obtain at least one target search object keyword and at least one target search intention keyword;

[0010] Determining images in which an object label matches the target search object keyword from an image library to obtain a first candidate image set, each image in the image library being provided with an object label and at least one object description label, the object label and the object description label being pre-set labels of the image;

[0011] For each image in the first candidate image set, determining images in which the object description label matches the target search intention keyword to obtain a second candidate image set;

[0012] Calculate scores of images in the second candidate image set based on the original search object keyword, the original search intent keyword, the object label, and the object description label;

[0013] Return the N images with the highest scores in the second candidate image set to the user as search results.

[0014] As a further limitation of the technical solutions of the embodiments of the present application, before determining the original search object keyword and the at least one original search intent keyword from the search text based on natural language processing, the following steps are further included:

[0015] Preprocess the search text to obtain a preprocessed search text.

[0016] As a further limitation of the technical solutions of the embodiments of the present application, the determination of the original search object keyword and the at least one original search intent keyword from the search text based on natural language processing specifically includes the following steps:

[0017] Determine the part of speech of each word in the search text based on dependency syntax analysis;

[0018] Determine the words with the part of speech as nouns as candidate search object keywords;

[0019] Determine the words in the candidate search object keywords that are ranked in the specified position as the original search object keyword;

[0020] Determine the words with the part of speech as nouns, adjectives, adverbs, and verbs before the original search object keyword in the search text as the original search intent keyword.

[0021] As a further limitation of the technical solutions of the embodiments of the present application, the determination of the original search object keyword and the at least one original search intent keyword from the search text based on natural language processing specifically includes the following steps:

[0022] Determine the part of speech of each word in the search text based on dependency syntax analysis, and determine the position of each word in the search text;

[0023] Input the search text, the part of speech, and the position into a keyword classification model to obtain the original search object keyword and the at least one original search intent keyword.

[0024] As a further limitation of the technical solutions of the embodiments of the present application, the keyword classification model is trained through the following steps:

[0025] Obtain training sentences, which are sentences input by users when searching for pictures, and the part of speech, position, and label category of each word in the training sentences are labeled;

[0026] inputting the training sentence into a keyword classification model to obtain a predicted category of each word;

[0027] calculating a loss rate by using the labeled category and the predicted category;

[0028] stopping training of the keyword classification model when the loss rate is less than a preset threshold to obtain a keyword classification model;

[0029] updating parameters of the keyword classification model according to the loss rate and returning to the step of inputting the training sentence into the keyword classification model when the loss rate is greater than the preset threshold;

[0030] wherein, the loss rate calculation formula is as follows:

[0031] ;

[0032] LABEL i is a labeled category of the i th word, TYPE i is a predicted category of the i th word, w i is a weight of the i th word, wherein, when the part-of-speech of the i th word is a noun and the position of the i th word in the search text is n, w i = 0.5, when the position of the i th word is not n, w i = 0.2.

[0033] As a further limitation of the technical scheme of the embodiment of the present application, the expanding of the original search object keyword and the original search intention keyword to obtain at least one target search object keyword and at least one target search intention keyword specifically includes the following steps:

[0034] obtaining synonyms and near-synonyms of the original search object keyword, and determining the synonyms, the near-synonyms and the original search object keyword as target search object keywords;

[0035] obtaining synonyms and near-synonyms of the original search intention keyword, and determining the synonyms, the near-synonyms and the original search intention keyword as target search intention keywords.

[0036] As a further limitation of the technical scheme of the embodiment of the present application, the determining of the image in which the object label matches the target search object keyword from the image library to obtain a first candidate image set specifically includes the following steps:

[0037] obtaining an object label of each image in the image library, wherein the object label represents a main object contained in the image;

[0038] determine a first node of the object label, a second node of the target search object keyword, and an edge line from the first node to the second node in a preset search object keyword knowledge graph, wherein each node in the search object keyword knowledge graph is provided with a node value, and each edge line is provided with a weight;

[0039] calculate a product of the node value of the first node, the node value of the second node, and the weight of the edge line, to obtain a plurality of first similarities;

[0040] when any one of the first similarities is greater than a preset threshold, add the image to a first candidate image set, to obtain the first candidate image set.

[0041] As a further limitation of the technical scheme of the embodiment of the application, for each image in the first candidate image set, the object description label and the target search intent keyword matching image are determined, to obtain a second candidate image set, specifically including the following steps:

[0042] obtain an object description label of each image in the first candidate image set, wherein the object description label is used to describe the main object contained in the image;

[0043] determine a first node of the object description label, a second node of the target search intent keyword, and an edge line from the first node to the second node in a preset search intent knowledge graph, wherein each node in the search intent knowledge graph is provided with a node value, and each edge line is provided with a weight;

[0044] calculate a product of the node value of the first node, the node value of the second node, and the weight of the edge line, to obtain a plurality of second similarities;

[0045] when any one of the second similarities is greater than a preset threshold, add the image to a second candidate image set, to obtain the second candidate image set.

[0046] As a further limitation of the technical scheme of the embodiment of the application, the score of the image in the second candidate image set is calculated by using the original search object keyword, the original search intent keyword, the object label, and the object description label, specifically including the following steps:

[0047] for each image in the second candidate image set, determine a target search object keyword most similar to the object label of the image from the target search object keyword, and determine a first weight of an edge line between the most similar target search object keyword and the original search object keyword according to a preset search object keyword knowledge graph;

[0048] According to the preset search intention knowledge graph, the number of object description labels of the image whose similarity to the target search intention keyword is greater than a preset threshold is counted, and a second weight of a side line of the target search intention keyword and the original search intention keyword is determined, and an average value of the second weight is determined to obtain a third weight;

[0049] The product of the first weight, the third weight and the number is calculated to obtain a score of the image.

[0050] A natural language-based search system for searching images, the system comprising a search text receiving unit, an original keyword determining unit, a target keyword determining unit, a first candidate image set determining unit, a second candidate image set determining unit, a score calculating unit and a search result determining unit, wherein:

[0051] The search text receiving unit is configured to receive search text input by a user on an image search page.

[0052] The original keyword determining unit is configured to determine an original search object keyword and at least one original search intention keyword from the search text based on natural language processing.

[0053] The target keyword determining unit is configured to expand the original search object keyword and the original search intention keyword to obtain at least one target search object keyword and at least one target search intention keyword.

[0054] The first candidate image set determining unit is configured to determine images whose object labels match the target search object keyword from an image library to obtain a first candidate image set, wherein each image in the image library is provided with an object label and at least one object description label, and the object label and the object description label are pre-set labels of the image.

[0055] The second candidate image set determining unit is configured to determine images whose object description labels match the target search intention keyword for each image in the first candidate image set to obtain a second candidate image set.

[0056] The score calculating unit is configured to calculate scores of images in the second candidate image set using the original search object keyword, the original search intention keyword, the object label and the object description label.

[0057] The search result determining unit is configured to return the N images with the highest scores in the second candidate image set to the user as search results.

[0058] As a further limitation of the technical scheme of the embodiments of the present application, the following is further included:

[0059] A preprocessing unit is configured to preprocess the search text to obtain a preprocessed search text.

[0060] As a further limitation of the technical scheme of the embodiment of the present application, the original keyword determination unit specifically comprises:

[0061] A part-of-speech analysis module is configured to determine the part-of-speech of each word in the search text based on dependency syntax analysis;

[0062] A candidate search object keyword determination module is configured to determine the words with the part-of-speech of noun as candidate search object keywords;

[0063] An original search object keyword determination module is configured to determine the word ranked at a specified position in the candidate search object keywords as an original search object keyword;

[0064] An original search intent keyword determination module is configured to determine the words with the part-of-speech of noun, adjective, adverb or verb before the original search object keyword in the search text as original search intent keywords.

[0065] As a further limitation of the technical scheme of the embodiment of the present application, the original keyword determination unit specifically comprises:

[0066] A part-of-speech analysis module is configured to determine the part-of-speech of each word in the search text based on dependency syntax analysis, and determine the position of each word in the search text;

[0067] A keyword classification module is configured to input the search text, the part-of-speech and the position into a keyword classification model to obtain an original search object keyword and at least one original search intent keyword.

[0068] As a further limitation of the technical scheme of the embodiment of the present application, it further comprises:

[0069] A training prediction acquisition unit is configured to acquire a training sentence, wherein the training sentence is a sentence input by a user when searching for a picture, and the training sentence is labeled with the part-of-speech, position and label category of each word;

[0070] A training unit is configured to input the training sentence into a keyword classification model to obtain a predicted category of each word;

[0071] A loss rate calculation unit is configured to calculate a loss rate by using the label category and the predicted category;

[0072] A stop training unit is configured to stop training the keyword classification model when the loss rate is less than a preset threshold to obtain a keyword classification model;

[0073] a parameter updating unit, configured to update parameters of the keyword classification model according to the loss rate when the loss rate is greater than a preset threshold, and return to the training unit;

[0074] wherein the loss rate calculation formula is as follows:

[0075] ;

[0076] LABEL i is a label category of the ith word, TYPE i is a predicted category of the ith word, w i is a weight of the ith word, wherein w i = 0.5 when the part of speech of the ith word is a noun and the position of the ith word in the search text is n, and w i = 0.2 when the position of the ith word is not n.

[0077] As a further limitation of the technical scheme of the embodiment of the application, the target keyword determination unit specifically comprises:

[0078] a search object keyword expansion module, configured to obtain synonyms and near-synonyms of the original search object keyword, and determine the synonyms, near-synonyms and the original search object keyword as target search object keywords;

[0079] a search intent keyword expansion module, configured to obtain synonyms and near-synonyms of the original search intent keyword, and determine the synonyms, near-synonyms and the original search intent keyword as target search intent keywords.

[0080] As a further limitation of the technical scheme of the embodiment of the application, the first candidate image set determination unit specifically comprises:

[0081] an object label obtaining module, configured to obtain an object label of each image in the image library, the object label indicating a main object contained in the image;

[0082] a first node and edge value determination module, configured to determine a first node of the object label, a second node of the target search object keyword, and an edge line from the first node to the second node in a preset search object keyword knowledge graph, each node in the search object keyword knowledge graph being provided with a node value, and each edge line being provided with a weight;

[0083] a first similarity calculation module, configured to calculate a product of the node value of the first node, the node value of the second node and the weight of the edge line, to obtain a plurality of first similarities;

[0084] The first candidate image set generation module is configured to add the image to a first candidate image set when any one of the first similarities is greater than a preset threshold, to obtain the first candidate image set.

[0085] As a further limitation of the technical scheme of the embodiment of the application, the second candidate image set determination unit specifically comprises:

[0086] The object description label acquisition module is configured to acquire an object description label of each image in the first candidate image set, the object description label being used to describe a main object contained in the image.

[0087] The first node and edge value determination module is configured to determine a first node of the object description label, a second node of the target search intent keyword, and an edge from the first node to the second node in a preset search intent knowledge graph, each node in the search intent knowledge graph being provided with a node value, and each edge being provided with a weight.

[0088] The second similarity calculation module is configured to calculate a product of the node value of the first node, the node value of the second node, and the weight of the edge, to obtain a plurality of second similarities.

[0089] The second candidate image set generation module is configured to add the image to a second candidate image set when any one of the second similarities is greater than a preset threshold, to obtain the second candidate image set.

[0090] As a further limitation of the technical scheme of the embodiment of the application, the score calculation unit specifically comprises:

[0091] The object keyword weight determination module is configured to determine, for each image in the second candidate image set, a target search object keyword that is most similar to an object label of the image from the target search object keywords, and determine a first weight of an edge between the most similar target search object keyword and the original search object keyword according to a preset search object keyword knowledge graph.

[0092] The object description label data statistics module is configured to determine, according to a preset search intent knowledge graph, a number of object description labels whose similarity to the target search intent keyword is greater than a preset threshold, determine a second weight of an edge between the target search intent keyword and the original search intent keyword, and determine an average value of the second weight, to obtain a third weight.

[0093] The score calculation module is configured to calculate a product of the first weight, the third weight, and the number, to obtain a score of the image.

[0094] Compared with the prior art, the application has the following advantages:

[0095] The embodiment of the present application determines the target search object keyword and the target search intention keyword through the expansion processing based on the natural language processing after determining the original search object keyword and the at least one original search intention keyword from the search text, determines the first candidate image set from the image library, and determines the object description label matching the target search intention keyword for each image in the first candidate image set to obtain the second candidate image set. The original search object keyword, the original search intention keyword, the object label and the object description label are used to calculate the score of the image in the second candidate image set, and the N images with the highest scores are returned to the user, which realizes the natural language recognition of the search object keyword and the search intention keyword in the search text input by the user, the image screening through the search object keyword and the search intention keyword, the combination of the search object and the search intention of the user to search the image, and the accurate return of the image meeting the intention of the user to the user. BRIEF DESCRIPTION OF DRAWINGS

[0096] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application.

[0097] Figure 1 A flow chart of the natural language-based search method provided by the embodiment of the present application is shown.

[0098] Figure 2 An image in the embodiment of the present application is shown.

[0099] Figure 3 A search object keyword knowledge graph in the embodiment of the present application is shown.

[0100] Figure 4 A search intention keyword knowledge graph in the embodiment of the present application is shown.

[0101] Figure 5 An application architecture diagram of the natural language-based search system provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0102] In order to make the purpose, technical solutions and advantages of the present application more clear, the following will further describe the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0103] Figure 1A flowchart of a method for natural language based search is shown. Specifically, the method for natural language based search is used for searching images, and specifically includes the following steps:

[0104] In step S101, a search text input by a user on an image search page is received.

[0105] In this embodiment, the image search page can be a page specially used for searching images, such as a web page used for searching images, which is provided with an input box, and the user can input a search text in the input box, which is used to describe the image the user expects to search. In one example, the user can input the search text in the input box by using a keyboard or a handwriting pen as an input device. In another example, the user can input a voice through a microphone or other voice device, and the image search page converts the voice into a search text. The embodiment does not limit the input method of the search text.

[0106] In one optional embodiment, after the search text is received, the search text can be preprocessed to obtain a preprocessed search text, wherein the preprocessing can include removing useless characters and error correction, and the removing of useless characters can be removing punctuation marks, spaces and the like irrelevant to the search, and the error correction can be correcting errors in the search text. Through the preprocessing of the search text, the accuracy of the search text can be improved, and the noise content in the search text can be removed, thereby improving the accuracy of subsequent image search.

[0107] In step S102, an original search object keyword and at least one original search intention keyword are determined from the search text based on natural language processing.

[0108] In this embodiment, the search object keyword represents an object contained in the image the user expects to search, and the search intention keyword represents what kind of object the user expects to search.

[0109] In one embodiment, the part of speech of each word in the search text can be determined based on dependency syntax analysis, and the words with the part of speech of noun are determined as candidate search object keywords, and the word ranked at a specified position in the candidate search object keywords is determined as the original search object keyword, and the words with the part of speech of noun, adjective, adverb or verb before the original search object keyword in the search text are determined as the original search intention keyword.

[0110] Specifically, dependency syntax analysis is one of the core technologies of natural language processing, which aims to determine the syntactic structure of a sentence by analyzing the dependency relationship between words in the sentence. Dependency syntax analysis can determine the part of speech of words in a sentence, the dependency relationship between words, etc. In the image search scenario of the present embodiment, the user usually inputs “XXXXX image” into the input box, such as the user searching for “simple drawing of tree with fruit”. After dependency syntax analysis, “simple drawing of tree with fruit” is segmented into “hang / fruit / ’ / tree / ’ / simple drawing”, thereby determining the part of speech of each word, such as “hang” being a verb, “fruit” being a noun, “tree” being a noun, “simple drawing” being a noun, and “’” being an auxiliary word. Through the Chinese syntactic structure, it can be determined that “simple drawing” is the search object keyword, i.e., the user expects to search for a simple drawing, and each word before “simple drawing” (except the auxiliary word) is the original search intent keyword, i.e., the user’s intent is to search for a simple drawing of a tree with fruit. The present embodiment determines the search object keyword and the search intent keyword through dependency syntax analysis, and the algorithm is simple and can quickly determine the search object keyword and the search intent keyword in the search text.

[0111] In another embodiment, the keyword classification model can be pre-trained. Specifically, training sentences can be obtained, wherein the training sentences are sentences input by the user when searching for pictures, and each word in the training sentences is labeled with its part of speech, position, and label category. The training sentences are input into the keyword classification model to obtain the predicted category of each word. The loss rate is calculated using the label category and the predicted category. When the loss rate is less than a preset threshold, the training of the keyword classification model is stopped to obtain the keyword classification model. When the loss rate is greater than the preset threshold, the parameters of the keyword classification model are updated according to the loss rate, and the step of inputting the training sentences into the keyword classification model is returned. The loss rate calculation formula is as follows:

[0112] ;

[0113] LABEL i is the label category of the i-th word, TYPE i is the predicted category of the i-th word, w i is the weight of the i-th word, wherein when the part of speech of the i-th word is a noun and the position in the search text is n, w i = 0.5, and when the position of the i-th word is not n, w i = 0.2. For example, the training sentence has a total of n = 5 words, and the weight of the 5th word is 0.5 when the 5th word is a noun, and the weights of the other words are 0.2, so that the loss function can pay more attention to the search object keyword in the training process. The way to update the model parameters can refer to the stochastic gradient descent method in the existing model training, which is not limited in the present embodiment.

[0114] Taking the training sentence "a simple sketch of a tree with fruit" as an example, the word segmentation is "hang / fruit / ’s / tree / ’s / simple sketch", and the parts of speech of each word are labeled, such as "hang" as a verb, "fruit" as a noun, "tree" as a noun, "simple sketch" as a noun, and "’s" as a auxiliary word. The key word category of "simple sketch" is labeled as the search object key word through the Chinese syntax structure, and the key word categories of each word (except auxiliary words) before "simple sketch" are the original search intent key words. Through training the key word classification model, the key word classification model learns the ability to identify the object key word and the search intent key word in the sentence according to the parts of speech, positions, and labeled categories of each word in the sentence.

[0115] After determining the parts of speech of each word in the search text and the position of each word in the search text based on the dependency syntax analysis, the search text, the parts of speech, and the positions are input into the key word classification model to obtain the original search object key word and at least one original search intent key word. Through training the key word classification model, the search object key word and the search intent key word can be identified, and the generality is strong, and the search object key word and the search intent key word of the search text of various syntax structures can be identified.

[0116] In step S103, the original search object key word and the original search intent key word are expanded to obtain at least one target search object key word and at least one target search intent key word.

[0117] In an optional embodiment, for the original search object key word, the synonyms and near-synonyms of the original search object key word can be obtained, and the synonyms, near-synonyms, and the original search object key word are determined as the target search object key word. For the original search intent key word, the synonyms and near-synonyms of the original search intent key word can be obtained, and the synonyms, near-synonyms, and the original search intent key word are determined as the target search intent key word.

[0118] The original search object key word and the original search intent key word are expanded in this embodiment, which can expand the image search range, improve the probability of recalling images, and make the search results more comprehensive.

[0119] In step S104, the images with the object label matching the target search object key word are determined from the image library to obtain a first candidate image set. Each image in the image library is provided with an object label and at least one object description label, and the object label and the object description label are pre-set labels of the image.

[0120] Each image in the image library in the embodiment is provided with an object label and an object description label, the object label represents the main object contained in the image, and the object description label is used to describe the main object in the image, wherein the object label can be one, and the object description label can be multiple. As shown in one example of the image Figure 2 The object label can be "stick figure", and the object description label can include "tree", "fruit", "fruit", "plant", "forest", "fruit", "fruit", "small tree", and the like.

[0121] Based on the pre-established image library, the object label of each image in the image library can be obtained, the first node of the object label, the second node of the target search object keyword, and the edge line from the first node to the second node are determined in the preset search object keyword knowledge graph, each node in the search object keyword knowledge graph is provided with a node value, each edge line is provided with a weight, the product of the node value of the first node, the node value of the second node and the weight of the edge line is calculated, and a plurality of first similarities are obtained. When any one of the first similarities is greater than a preset threshold, the image is added to the first candidate image set to obtain the first candidate image set.

[0122] As shown in Figure 3 As shown in one example of the search object keyword knowledge graph, the object label of image A in the image library is "line drawing", the expanded target search object keyword of the search text "stick figure of fruit-bearing tree" can include "stick figure", "stick figure", "white drawing" and "black and white drawing", and the product of the value of each node and each edge line on the shortest path from each target search object keyword to the object label of the image can be calculated as the first similarity. The specific is as follows:

[0123] The shortest path of the target search object keyword "stick figure" to the object label "line drawing" is "stick figure"-"line drawing", and the first similarity is 1.0x0.5x0.7=0.35;

[0124] The shortest path of the target search object keyword "stick figure" to the object label "line drawing" is "stick figure"-"line drawing", and the first similarity is 1.0x0.5x0.7=0.35;

[0125] The shortest path of the target search object keyword "white drawing" to the object label "line drawing" is "white drawing"-"stick figure"-"line drawing", and the first similarity is 0.8x0.7x1.0x0.5x0.7=0.196;

[0126] The shortest path of the target search object keyword "black and white drawing" to the object label "line drawing" includes path 1:

[0127] “black and white picture”-“simple sketch”-“line drawing”, the first similarity is 0.5*0.3*1.0*0.6*0.7=0.063;

[0128] Path 2:

[0129] “black and white picture”-“simple sketch”-“line drawing”, the first similarity is 0.5*0.4*1.0*0.5*0.7=0.07.

[0130] Assuming the threshold is 0.4, the similarity between the object label “line drawing” of image A and the target search object keyword “simple sketch” is greater than 0.4, then image A is added to the first candidate image set, so as to realize the rough screening of the first candidate image set through the target search object keyword, and to exclude images not containing the original search object. In the above example, the user searches for simple sketches, so color images, black and white images that are not simple sketches, etc. can be excluded from the image library to reduce the number of images in the subsequent fine screening and improve search efficiency.

[0131] Step S105, for each image in the first candidate image set, determine the image whose object description label matches the target search intent keyword, and obtain the second candidate image set.

[0132] As Figure 4 shown is a schematic diagram of a search intent knowledge graph, after obtaining the object description label of each image in the first candidate image set, the first node of the object description label, the second node of the target search intent keyword, and the edge line from the first node to the second node are determined in the preset search intent knowledge graph. Each node in the search intent knowledge graph is provided with a node value, and each edge line is provided with a weight; the product of the node value of the first node, the node value of the second node, and the weight of the edge line is calculated to obtain a plurality of second similarities. When any one of the second similarities is greater than a preset threshold, the image is added to the second candidate image set to obtain the second candidate image set.

[0133] Exemplarily, one of the object description labels of image B in the first candidate image set is “fruit”, and one of the target search intent keywords of the search text “simple sketch of tree with fruit” is “fruit”. From Figure 4 the second similarity can be calculated: 0.8*0.7*1.0=0.56, if the threshold is 0.4, then image B is added to the second candidate image set, so as to screen out images that meet part of the user's search intent from the first candidate image set.

[0134] Step S106, the original search object keyword, the original search intent keyword, the object label, and the object description label are used to calculate the score of the image in the second candidate image set.

[0135] Specifically, for each image in the second candidate image set, a target search object keyword most similar to the object label of the image is determined from the target search object keywords, and a first weight of the most similar target search object keyword and the original search object keyword is determined according to a preset search object keyword knowledge graph. According to the preset search intent knowledge graph, the number of object description labels of the image whose similarity to the target search intent keyword is greater than a preset threshold is counted, and a second weight of the target search intent keyword and the original search intent keyword is determined, and an average value of the second weight is determined to obtain a third weight. The product of the first weight, the third weight and the number is calculated to obtain a score of the image.

[0136] For example, the object label of the image C in the second candidate image set is "line drawing", and the target search object keywords of the search text can include "simple drawing", "simple drawing", "white drawing" and "black and white drawing", wherein the original object keyword is "simple drawing". The object label of the image C is determined by S104 and S105. Figure 3 It can be seen that the target search object keyword "simple drawing" has the maximum similarity to the object label "line drawing", and the first weight of the target search object keyword "simple drawing" to the edge line of the object label "line drawing" is 0.6.

[0137] Then, according to the second similarity between the object description label of the image and the target search intent keyword calculated based on the value of the node and the edge line in the search intent knowledge graph in S105, the number of object description labels whose second similarity is greater than a preset threshold is counted, and the second weight of the corresponding edge line is determined. The average value of a plurality of second weights is calculated to obtain a third weight, and the product of the first weight, the number and the third weight is finally calculated as the score of the image C. For example, the image C has object description label 11, object description label 12, object description label 13 and object description label 14, and the score of the image C is calculated as follows: Figure 4As shown, the search text has target search intent keywords 21 (fruit), target search intent keywords 22 (fruit), target search intent keywords 23 (fruit), target search intent keywords 24 (fruit), target search intent keywords 25 (fruit), and target search intent keywords 26 (fruit), wherein the target search intent keyword 21 (fruit) is the original search intent keyword. After calculating the similarity of each target search intent keyword and each object description label of the image C, if the second similarity of the object description label 11 and the target search intent keyword 22, the second similarity of the object description label 13 and the target search intent keyword 26, and the second similarity of the object description label 14 and the target search intent keyword 24 are all greater than a threshold value, the number is 3, and then the average of the weight of the edge line between the target search intent keyword 22 and the original search intent keyword, the weight of the edge line between the target search intent keyword 26 and the original search intent keyword, and the weight of the edge line between the target search intent keyword 24 and the original search intent keyword is calculated as the third weight. Finally, the number, the first weight, and the third weight are calculated as the score of the image C, which takes into account the similarity of the search object keyword and the object label of the image, the similarity of the search intent keyword and the object description label of the image, and the number and weight of the similarity of the search intent keyword and the object description label of the image. The greater the number is, the greater the weight is, and the greater the score of the image is, and the more the image meets the search intent of the user.

[0138] In step S107, the N images with the highest scores in the second candidate image set are returned to the user as search results.

[0139] Specifically, the images in the second candidate image set can be sorted in descending order of scores, and the N images in the front of the sorting are returned to the user as search results.

[0140] In the embodiment of the application, after determining the original search object keyword and at least one original search intent keyword from the search text based on natural language processing, the target search object keyword and the target search intent keyword are obtained through expansion processing. The first candidate image set is obtained by determining the images with object labels matching the target search object keyword from the image library. Further, for each image in the first candidate image set, the second candidate image set is obtained by determining the images with object description labels matching the target search intent keyword. The score of the image in the second candidate image set is calculated using the original search object keyword, the original search intent keyword, the object label, and the object description label. The N images with the highest scores are returned to the user, which realizes the identification of the search object keyword and the search intent keyword in the search text input by the user through natural language recognition, filters the images through the search object keyword and the search intent keyword, combines the search object and the search intent of the user, and accurately returns the images meeting the intent of the user to the user.

[0141] Figure 5 An application architecture diagram of a system for natural language based search provided by an embodiment of the present application is shown. The system for natural language based search of the present embodiment is used for searching images and comprises:

[0142] A search text receiving unit 501 is configured to receive a search text input by a user on an image search page.

[0143] An original keyword determining unit 502 is configured to determine an original search object keyword and at least one original search intention keyword from the search text based on natural language processing.

[0144] A target keyword determining unit 503 is configured to expand the original search object keyword and the original search intention keyword to obtain at least one target search object keyword and at least one target search intention keyword.

[0145] A first candidate image set determining unit 504 is configured to determine images with object labels matching the target search object keyword from an image library to obtain a first candidate image set, wherein each image in the image library is provided with an object label and at least one object description label, and the object label and the object description label are pre-set labels of the image.

[0146] A second candidate image set determining unit 505 is configured to determine images with object description labels matching the target search intention keyword for each image in the first candidate image set to obtain a second candidate image set.

[0147] A score calculating unit 506 is configured to calculate scores of images in the second candidate image set using the original search object keyword, the original search intention keyword, the object label and the object description label.

[0148] A search result determining unit 507 is configured to return the N images with the highest scores in the second candidate image set as search results to the user.

[0149] As a further limitation of the technical scheme of the present embodiment, the present embodiment further comprises:

[0150] A preprocessing unit is configured to pre-process the search text to obtain a pre-processed search text.

[0151] As a further limitation of the technical scheme of the present embodiment, the original keyword determining unit 502 specifically comprises:

[0152] A part-of-speech analysis module is configured to determine the part-of-speech of each word in the search text based on dependency syntax analysis.

[0153] The candidate search object keyword determination module is configured to determine a word with a part of speech of a noun as a candidate search object keyword.

[0154] The original search object keyword determination module is configured to determine a word ranked at a specified position in the candidate search object keyword as an original search object keyword.

[0155] The original search intent keyword determination module is configured to determine a word with a part of speech of a noun, an adjective, an adverb or a verb before the original search object keyword in the search text as an original search intent keyword.

[0156] As a further limitation of the technical scheme of the embodiment of the application, the original keyword determination unit 502 specifically comprises:

[0157] The part of speech analysis module is configured to determine a part of speech of each word in the search text and a position of each word in the search text based on dependency syntax analysis;

[0158] The keyword classification module is configured to input the search text, the part of speech and the position into a keyword classification model to obtain an original search object keyword and at least one original search intent keyword.

[0159] As a further limitation of the technical scheme of the embodiment of the application, the application further comprises:

[0160] The training prediction acquisition unit is configured to acquire a training sentence, the training sentence being a sentence input by a user when searching for a picture, the training sentence being labeled with a part of speech, a position and a labeled category of each word;

[0161] The training unit is configured to input the training sentence into the keyword classification model to obtain a predicted category of each word;

[0162] The loss rate calculation unit is configured to calculate a loss rate by using the labeled category and the predicted category;

[0163] The stop training unit is configured to stop training of the keyword classification model when the loss rate is less than a preset threshold value to obtain a keyword classification model;

[0164] The parameter updating unit is configured to update parameters of the keyword classification model according to the loss rate when the loss rate is greater than the preset threshold value and return to the training unit;

[0165] The loss rate calculation formula is as follows:

[0166] ;

[0167] LABEL i is a labeled category of the ith word, TYPE iw is a weight of the i-th word, wherein w i = 0.5 when the part-of-speech of the i-th word is a noun and the position of the i-th word in the search text is n, w i = 0.2 when the position of the i-th word is not n. i

[0168] As a further limitation of the technical solutions of the embodiments of the application, the target keyword determination unit 503 specifically includes:

[0169] A search object keyword expansion module is configured to obtain synonyms and near-synonyms of the original search object keyword, and determine the synonyms, near-synonyms and the original search object keyword as target search object keywords.

[0170] A search intent keyword expansion module is configured to obtain synonyms and near-synonyms of the original search intent keyword, and determine the synonyms, near-synonyms and the original search intent keyword as target search intent keywords.

[0171] As a further limitation of the technical solutions of the embodiments of the application, the first candidate image set determination unit 504 specifically includes:

[0172] An object label obtaining module is configured to obtain an object label of each image in the image library, the object label indicating a main object contained in the image.

[0173] A first node and edge value determination module is configured to determine a first node of the object label, a second node of the target search object keyword, and an edge from the first node to the second node in a preset search object keyword knowledge graph, each node in the search object keyword knowledge graph being provided with a node value, and each edge being provided with a weight.

[0174] A first similarity calculation module is configured to calculate a product of the node value of the first node, the node value of the second node and the weight of the edge, to obtain a plurality of first similarities.

[0175] A first candidate image set generation module is configured to add the image to a first candidate image set when any one of the first similarities is greater than a preset threshold, to obtain the first candidate image set.

[0176] As a further limitation of the technical solutions of the embodiments of the application, the second candidate image set determination unit 505 specifically includes:

[0177] ​An object description label obtaining module is configured to obtain an object description label of each image in the first candidate image set, the object description label being used to describe a main object contained in the image;

[0178] A first node and edge value determining module is configured to determine a first node of the object description label, a second node of the target search intent keyword, and an edge from the first node to the second node in a preset search intent knowledge graph, each node in the search intent knowledge graph being provided with a node value, and each edge being provided with a weight;

[0179] A second similarity calculating module is configured to calculate a product of the node value of the first node, the node value of the second node, and the weight of the edge, to obtain a plurality of second similarities.

[0180] A second candidate image set generating module is configured to add the image to a second candidate image set when any one of the second similarities is greater than a preset threshold, to obtain the second candidate image set.

[0181] As a further limitation of the technical scheme of the embodiment of the application, the score calculating unit 506 specifically comprises:

[0182] An object keyword weight determining module is configured to determine, for each image in the second candidate image set, a target search object keyword that is most similar to an object label of the image from the target search object keywords, and determine a first weight of an edge between the most similar target search object keyword and the original search object keyword according to a preset search object keyword knowledge graph.

[0183] An object description label data statistical module is configured to count, according to a preset search intent knowledge graph, a number of object description labels whose similarity to the target search intent keyword is greater than a preset threshold, and determine a second weight of an edge between the target search intent keyword and the original search intent keyword, and determine an average value of the second weight, to obtain a third weight.

[0184] A score calculating module is configured to calculate a product of the first weight, the third weight, and the number, to obtain a score of the image.

[0185] It should be understood that, although the steps in the flowcharts of the embodiments of the present application are shown in a certain order according to the arrows, the steps are not necessarily executed in the order of the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in order, and the steps can be executed in other orders. Moreover, at least some of the steps in the embodiments can include a plurality of sub-steps or a plurality of stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the sub-steps or stages is not necessarily sequential, but can be round-robin or alternately executed with at least some of the other steps or sub-steps or stages of the other steps.

[0186] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a non-volatile computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of the methods. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0187] The technical features of the above-mentioned embodiments can be combined in any way. In order to make the description concise, all possible combinations of the technical features in the above-mentioned embodiments are not described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0188] The above embodiments only express several implementation manners of the present application, which are described in a more specific and detailed manner, but should not be understood as a limitation on the patent scope of the present application. It should be noted that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, which are all within the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

[0189] The above merely describes the preferred embodiments of the present application and should not be used to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method of natural language based search, characterized in that, The method for searching images specifically comprises the following steps: receiving search text input by a user on an image search page; determining original search object keywords and at least one original search intent keyword from the search text based on natural language processing; expanding the original search object keywords and the original search intent keywords to obtain at least one target search object keyword and at least one target search intent keyword; determining images with object labels matching the target search object keywords from an image library to obtain a first candidate image set, wherein each image in the image library is provided with an object label and at least one object description label, and the object label and the object description label are pre-set labels of the image; for each image in the first candidate image set, determining images with object description labels matching the target search intent keywords to obtain a second candidate image set; calculating scores of images in the second candidate image set using the original search object keywords, the original search intent keywords, the object label and the object description label; returning the N images with the highest scores in the second candidate image set to the user as search results; the method for determining original search object keywords and at least one original search intent keyword from the search text based on natural language processing specifically comprises the following steps: determining the part-of-speech of each word in the search text and the position of each word in the search text based on dependency syntax analysis; inputting the search text, the part-of-speech and the position into a keyword classification model to obtain original search object keywords and at least one original search intent keyword; wherein the keyword classification model is trained through the following steps: obtaining training sentences, wherein the training sentences are sentences input by users when searching for images, and the part-of-speech, position and labeled category of each word in the training sentences are labeled; inputting the training sentences into the keyword classification model to obtain the predicted category of each word; calculating the loss rate using the labeled category and the predicted category; when the loss rate is less than a preset threshold, stopping training the keyword classification model to obtain the keyword classification model; when the loss rate is greater than the preset threshold, updating the parameters of the keyword classification model according to the loss rate, and returning to the step of inputting the training sentences into the keyword classification model; wherein the loss rate is calculated according to the following formula: ； LABEL i is a label class for the i-th word, TYPE i is a predicted class for the i-th word, w i is a weight for the i-th word, where w i = 0.5 when the part-of-speech of the i-th word is a noun and the position of the i-th word in the search text is n, and w i = 0.2 when the position of the i-th word is not n. the method for calculating the scores of images in the second candidate image set using the original search object keywords, the original search intent keywords, the object label and the object description label specifically comprises the following steps: for each image in the second candidate image set, determining the target search object keyword most similar to the object label of the image from the target search object keywords, and determining the first weight of the edge line between the most similar target search object keyword and the original search object keyword according to a preset search object keyword knowledge graph; According to the preset search intention knowledge graph, the number of object description labels of the image whose similarity with the target search intention keyword is greater than a preset threshold is counted, and a second weight of an edge line between the target search intention keyword and the original search intention keyword is determined, and an average value of the second weight is determined to obtain a third weight; The product of the first weight, the third weight, and the number is calculated to obtain a score of the image.

2. The natural language based search method of claim 1, wherein, Before determining the original search object keyword and the at least one original search intention keyword from the search text based on natural language processing, the following steps are further included: The search text is preprocessed to obtain a preprocessed search text.

3. The natural language based search method of claim 1, wherein, The original search object keyword and the at least one original search intention keyword are determined from the search text based on natural language processing, specifically including the following steps: The part of speech of each word in the search text is determined based on dependency syntax analysis; The word with the part of speech of noun is determined as a candidate search object keyword; The word ranked in a specified position in the candidate search object keyword is determined as the original search object keyword; The words with the part of speech of noun, adjective, adverb, and verb before the original search object keyword in the search text are determined as the original search intention keyword.

4. The natural language based search method of claim 1, wherein, The original search object keyword and the at least one original search intention keyword are expanded to obtain the at least one target search object keyword and the at least one target search intention keyword, specifically including the following steps: Synonyms and near-synonyms of the original search object keyword are obtained, and the synonyms, the near-synonyms, and the original search object keyword are determined as the target search object keyword; Synonyms and near-synonyms of the original search intention keyword are obtained, and the synonyms, the near-synonyms, and the original search intention keyword are determined as the target search intention keyword.

5. The natural language based search method of claim 1, wherein, The images whose object labels match the target search object keyword are determined from the image library to obtain a first candidate image set, specifically including the following steps: The object label of each image in the image library is obtained, and the object label represents the main object contained in the image; A first node of the object label, a second node of the target search object keyword, and an edge line from the first node to the second node are determined in a preset search object keyword knowledge graph, and each node in the search object keyword knowledge graph is provided with a node value, and each edge line is provided with a weight; The product of the node value of the first node, the node value of the second node, and the weight of the edge line is calculated to obtain a plurality of first similarities; When any one of the first similarities is greater than a preset threshold, the image is added to the first candidate image set to obtain the first candidate image set.

6. The natural language based search method of claim 1, wherein, For each image in the first candidate image set, the image whose object description label matches the target search intention keyword is determined to obtain a second candidate image set, specifically including the following steps: obtain an object description label of each image in the first candidate image set, the object description label being used to describe a main object contained in the image; determine a first node of the object description label, a second node of the target search intent keyword, and a line between the first node and the second node in a preset search intent knowledge graph, each node in the search intent knowledge graph being provided with a node value, and each line being provided with a weight; calculate a product of the node value of the first node, the node value of the second node, and the weight of the line to obtain a plurality of second similarities; when any one of the second similarities is greater than a preset threshold, add the image to a second candidate image set to obtain the second candidate image set.

7. A natural language based search system characterized in that, The system for searching images comprises a search text receiving unit, an original keyword determining unit, a target keyword determining unit, a first candidate image set determining unit, a second candidate image set determining unit, a score calculating unit, and a search result determining unit, wherein: The search text receiving unit is configured to receive a search text input by a user on an image search page. The original keyword determining unit is configured to determine an original search object keyword and at least one original search intent keyword from the search text based on natural language processing. The target keyword determining unit is configured to expand the original search object keyword and the original search intent keyword to obtain at least one target search object keyword and at least one target search intent keyword. The first candidate image set determining unit is configured to determine images whose object labels match the target search object keyword from an image library to obtain a first candidate image set, each image in the image library being provided with an object label and at least one object description label, the object label and the object description label being pre-set labels of the image. The second candidate image set determining unit is configured to determine images whose object description labels match the target search intent keyword for each image in the first candidate image set to obtain a second candidate image set. The score calculating unit is configured to calculate scores of images in the second candidate image set using the original search object keyword, the original search intent keyword, the object label, and the object description label. The search result determining unit is configured to return N images with the highest scores in the second candidate image set to the user as search results. The original keyword determining unit specifically comprises: A part-of-speech analysis module configured to determine the part-of-speech of each word in the search text and the position of each word in the search text based on dependency syntax analysis. A keyword classification module configured to input the search text, the part-of-speech, and the position into a keyword classification model to obtain the original search object keyword and at least one original search intent keyword. Further comprising: A training prediction obtaining unit configured to obtain training sentences, the training sentences being sentences input by users when searching for pictures, the training sentences being labeled with the part-of-speech, the position, and the labeled category of each word. The training unit is configured to input the training sentence into a keyword classification model to obtain a predicted category of each word; The loss rate calculation unit is configured to calculate a loss rate based on the labeled category and the predicted category; The stop training unit is configured to stop training the keyword classification model when the loss rate is less than a preset threshold to obtain a keyword classification model; The parameter updating unit is configured to update parameters of the keyword classification model based on the loss rate when the loss rate is greater than the preset threshold, and return to the training unit; The loss rate calculation formula is as follows: ； LABEL i is a label category for the i-th word, TYPE i is a predicted category for the i-th word, w i is a weight for the i-th word, where w i = 0.5 when the part-of-speech of the i-th word is a noun and at position n in the search text, and w i = 0.2 when the position of the i-th word is not n. The score calculation unit specifically includes: The object keyword weight determination module is configured to determine, for each image in the second candidate image set, a target search object keyword that is most similar to an object label of the image from the target search object keywords, and determine a first weight of an edge line between the most similar target search object keyword and the original search object keyword based on a preset search object keyword knowledge graph. The object description label data statistics module is configured to count, based on a preset search intent knowledge graph, a number of object description labels whose similarity to the target search intent keyword is greater than a preset threshold, determine a second weight of an edge line between the target search intent keyword and the original search intent keyword, and determine an average value of the second weight to obtain a third weight. The score calculation module is configured to calculate a product of the first weight, the third weight, and the number to obtain a score of the image.

Images