A method and device for determining image keywords

By obtaining candidate words in the document and sorting them according to statistical information and graphic similarity, the problem of low accuracy of image keyword determination methods in the prior art is solved, and more accurate image keyword determination is achieved.

CN114139005BActive Publication Date: 2024-11-29BEIJING SOGOU TECHNOLOGY DEVELOPMENT CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202010915712.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-09-03
Publication Date
2024-11-29
Estimated Expiration
2040-09-03

AI Technical Summary

Technical Problem

The method of determining image keywords in the prior art has low accuracy and cannot correctly represent the image content.

Method used

By obtaining the document containing the query graph, extracting candidate words, and sorting them based on the statistical information of the candidate words in the document and the similarity to the graphic and text with the query graph, selecting the highest-sorted candidate words as the picture keyword.

Benefits of technology

It improves the accuracy of image keywords, makes it more relevant to the document topic and query pictures, and can more accurately reflect the image content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114139005B_ABST
    Figure CN114139005B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention discloses a method and device for determining image keywords. The method includes: first obtaining at least one document containing a query graph, extracting candidate words in the document, then determining the image-text similarity between the candidate words and the query graph, sorting the candidate words according to statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph, and finally, using the candidate words with the highest ranking as keywords corresponding to the query graph. In the method provided by the embodiment of the present invention, candidate words are extracted from multiple documents, and the candidate words are sorted according to statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph. The candidate words are sorted according to both statistical information and image-text similarity, and the candidate words with the highest ranking are used as keywords, which not only ensures a high relevance between the keywords and the documents, but also ensures a high degree of association between the keywords and the query graph, so that the determined keywords can accurately reflect the content of the query graph.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a method and device for determining picture keywords. Background Art

[0002] Image keywords can be used to represent key information corresponding to the image content. When the user is unsure of the image content or needs to obtain relevant information about the image content based on the image, the user can determine the information corresponding to the image content or obtain information related to the image content by determining the image keywords corresponding to the image.

[0003] The existing method for determining picture keywords is to select words as picture keywords from the document containing the picture according to the statistical information of the words. However, this method of determining picture keywords has low accuracy and cannot correctly represent the content of the picture. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method and device for determining picture keywords to solve the problem that the picture keywords determined in the prior art cannot accurately represent the picture content.

[0005] To solve the above problems, the technical solutions provided by the embodiments of the present invention are as follows:

[0006] A method for determining picture keywords, the method comprising:

[0007] Obtain at least one document containing the query graph;

[0008] Extracting candidate words from the document;

[0009] Determining the image-text similarity between the candidate word and the query graph;

[0010] Rank the candidate words using statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph;

[0011] The candidate word with the highest ranking is selected as the keyword corresponding to the query graph.

[0012] In a possible implementation, extracting candidate words from the document includes:

[0013] Segmenting the document, and merging the obtained segmented words to generate merged words;

[0014] If the merged word is in a pre-established candidate word list, the merged word is determined as a candidate word.

[0015] In a possible implementation, determining the image-text similarity between the candidate word and the query graph includes:

[0016] Inputting the query graph into a picture keyword feature extraction model to obtain keyword text features corresponding to the query graph output by the picture keyword feature model, wherein the picture keyword feature model is trained using the query graph to be trained and the keyword text features corresponding to the query graph to be trained;

[0017] Obtaining text features of the candidate words;

[0018] Calculating the correlation between the text features of the candidate word and the text features of the keywords corresponding to the query graph as the image-text similarity between the candidate word and the query graph;

[0019] or,

[0020] The candidate word and the query graph are input into a picture-text similarity determination model to obtain the picture-text similarity between the candidate word and the query graph output by the picture-text similarity determination model, wherein the picture-text similarity determination model is trained using the picture-text similarity between the query graph to be trained and the candidate word to be trained.

[0021] In a possible implementation, the step of ranking the candidate words by using the statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph includes:

[0022] Inputting statistical information of the candidate word in the document and the image-text similarity between the candidate word and the query graph into a first candidate word feature evaluation model to obtain an evaluation score of the candidate word output by the first candidate word feature evaluation model, wherein the first candidate word feature evaluation model is trained based on statistical information of the candidate word to be trained in the document to be trained, the image-text similarity between the candidate word to be trained and the query graph to be trained, and the evaluation score of the candidate word to be trained;

[0023] The candidate words are sorted according to their evaluation scores.

[0024] In a possible implementation, the method further includes:

[0025] Obtain additional features corresponding to the candidate word, wherein the additional features of the candidate word are features of the website to which the document corresponding to the candidate word belongs and / or category information corresponding to the candidate word; the features of the website to which the document corresponding to the candidate word belongs are determined based on the number of visits to the website to which the document corresponding to the candidate word belongs.

[0026] In a possible implementation, the step of ranking the candidate words by using the statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph includes:

[0027] The candidate words are sorted by using statistical information of the candidate words in the document, the image-text similarity between the candidate words and the query graph, and additional features of the candidate words.

[0028] In a possible implementation, the ranking of the candidate words by using statistical information of the candidate words in the document, the image-text similarity between the candidate words and the query graph, and additional features of the candidate words includes:

[0029] Inputting statistical information of the candidate word in the document, the image-text similarity between the candidate word and the query graph, and the additional features of the candidate word into a second candidate word feature evaluation model to obtain an evaluation score of the candidate word output by the second candidate word feature evaluation model, wherein the second candidate word feature evaluation model is trained based on statistical information of the candidate word to be trained in the document to be trained, the image-text similarity between the candidate word to be trained and the query graph to be trained, the additional features of the candidate word to be trained, and the evaluation score of the candidate word to be trained, wherein the additional features of the candidate word to be trained are features of the website to which the document corresponding to the candidate word to be trained belongs and / or category information corresponding to the candidate word to be trained;

[0030] The candidate words are sorted according to their evaluation scores.

[0031] A device for determining picture keywords, the device comprising:

[0032] A document acquisition unit, configured to acquire at least one document containing the query graph;

[0033] A candidate word extraction unit, used to extract candidate words from the document;

[0034] A picture-text similarity determination unit, used to determine the picture-text similarity between the candidate word and the query graph;

[0035] A sorting unit, configured to sort each of the candidate words by using statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph;

[0036] The keyword determination unit is used to select the candidate word with the highest ranking as the keyword corresponding to the query graph.

[0037] In a possible implementation, the candidate word extraction unit includes:

[0038] A merge word generation subunit, used for segmenting the document and merging the obtained segmented words to generate a merge word;

[0039] The candidate word determination subunit is used to determine the merged word as a candidate word if the merged word is in a pre-established candidate word list.

[0040] In a possible implementation, the image-text similarity determination unit includes:

[0041] A keyword text feature acquisition subunit is used to input the query graph into a picture keyword feature extraction model to obtain keyword text features corresponding to the query graph output by the picture keyword feature model, wherein the picture keyword feature model is trained using the query graph to be trained and the keyword text features corresponding to the query graph to be trained;

[0042] A candidate word text feature acquisition subunit, used to acquire text features of the candidate word;

[0043] A correlation calculation subunit, used to calculate the correlation between the text features of the candidate word and the text features of the keywords corresponding to the query graph as the image-text similarity between the candidate word and the query graph;

[0044] or,

[0045] The image-text similarity determination unit is specifically used to input the candidate word and the query graph into an image-text similarity determination model to obtain the image-text similarity between the candidate word and the query graph output by the image-text similarity determination model. The image-text similarity determination model is trained using the image-text similarity between the query graph to be trained and the candidate word to be trained.

[0046] In a possible implementation, the sorting unit includes:

[0047] A first evaluation score acquisition subunit is used to input the statistical information of the candidate word in the document and the image-text similarity between the candidate word and the query graph into a first candidate word feature evaluation model to obtain the evaluation score of the candidate word output by the first candidate word feature evaluation model, wherein the first candidate word feature evaluation model is trained based on the statistical information of the candidate word to be trained in the document to be trained, the image-text similarity between the candidate word to be trained and the query graph to be trained, and the evaluation score of the candidate word to be trained;

[0048] The first sorting subunit is used to sort the candidate words according to the evaluation scores of the candidate words.

[0049] In a possible implementation manner, the device further includes:

[0050] The additional feature acquisition unit is used to obtain additional features corresponding to the candidate word, wherein the additional features of the candidate word are the features of the website to which the document corresponding to the candidate word belongs and / or the category information corresponding to the candidate word; the features of the website to which the document corresponding to the candidate word belongs are determined based on the number of visits to the website to which the document corresponding to the candidate word belongs.

[0051] In a possible implementation, the ranking unit is specifically configured to rank each of the candidate words by utilizing statistical information of the candidate words in the document, the image-text similarity between the candidate words and the query graph, and additional features of the candidate words.

[0052] In a possible implementation, the sorting unit includes:

[0053] A second evaluation score acquisition subunit is used to input statistical information of the candidate word in the document, the image-text similarity between the candidate word and the query graph, and additional features of the candidate word into a second candidate word feature evaluation model to obtain an evaluation score of the candidate word output by the second candidate word feature evaluation model, wherein the second candidate word feature evaluation model is trained based on statistical information of the candidate word to be trained in the document to be trained, the image-text similarity between the candidate word to be trained and the query graph to be trained, the additional features of the candidate word to be trained, and the evaluation score of the candidate word to be trained, wherein the additional features of the candidate word to be trained are features of the website to which the document corresponding to the candidate word to be trained belongs and / or category information corresponding to the candidate word to be trained;

[0054] The second sorting subunit is used to sort the candidate words according to the evaluation scores of the candidate words.

[0055] A device for determining image keywords includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations:

[0056] Obtain at least one document containing the query graph;

[0057] Extracting candidate words from the document;

[0058] Determining the image-text similarity between the candidate word and the query graph;

[0059] Rank the candidate words using statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph;

[0060] The candidate word with the highest ranking is selected as the keyword corresponding to the query graph.

[0061] A computer-readable medium stores instructions which, when executed by one or more processors, enable a device to execute the above-mentioned method for determining picture keywords.

[0062] It can be seen that the embodiments of the present invention have the following beneficial effects:

[0063] The method for determining image keywords provided by the embodiment of the present invention, after obtaining the query graph, first obtains at least one document containing the query graph, extracts candidate words in the document, then determines the image-text similarity between the candidate words and the query graph, sorts the candidate words according to the statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph, and finally uses the candidate words with the highest ranking as the keywords corresponding to the query graph. In the method provided by the embodiment of the present invention, candidate words are extracted from multiple documents, and the candidate words are sorted according to the statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph, so that the obtained candidate words with the highest ranking have a high correlation with the subject of the document and the query graph. Using the candidate words with the highest ranking as keywords not only ensures a high correlation between the keywords and the subject of the document, but also ensures a high degree of association between the keywords and the query graph, so that the determined keywords can more accurately reflect the content of the query graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A schematic diagram of a query graph provided for an embodiment of the present invention;

[0065] Figure 2 A schematic diagram of a framework of an exemplary application scenario provided by an embodiment of the present invention;

[0066] Figure 3 A flow chart of a method for determining image keywords provided by an embodiment of the present invention;

[0067] Figure 4 A method for determining the image-text similarity between a candidate word and a query graph provided by an embodiment of the present invention;

[0068] Figure 5 A schematic diagram of the structure of a device for determining image keywords provided by an embodiment of the present invention;

[0069] Figure 6 A schematic diagram of the structure of a client provided by an embodiment of the present invention;

[0070] Figure 7 A schematic diagram of the structure of a server provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0071] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0072] To facilitate understanding of the technical solution provided by the embodiment of the present invention, the background technology of the present invention will be described below.

[0073] The inventors discovered in their research on traditional methods for determining image keywords that traditional methods for determining image keywords extract vocabulary and determine keywords based on the statistical information of vocabulary in a document containing a query graph. Traditional methods for determining image keywords first obtain a document containing a query graph, and then use keyword extraction technology, that is, sort the vocabulary in the document based on the statistical information of the vocabulary, and use the best vocabulary as the keyword for the image. The method for determining keywords based on statistical information only considers the relevance of the vocabulary to the subject of the document. The subject of the document may not correspond to the content of the image contained in the document, resulting in the image keywords determined based on the statistical information not being consistent with the content of the image, and the image keywords are not accurate enough. For example, see Figure 1 , the figure is a schematic diagram of a query graph provided by an embodiment of the present invention. The content of the query graph is "J-10 fighter jet". The corresponding document containing the query graph may be a related report about a pilot flying a J-10 fighter jet for flight training. After keyword extraction is performed on the words in the obtained document according to the statistical information of the words, since the theme of the related document report may be a pilot conducting flight training, the keyword obtained according to the statistical information may be "pilot", which does not match the content of "J-10 fighter jet" in the query graph, and the keyword of the obtained picture is not accurate enough.

[0074] Based on this, an embodiment of the present invention provides a method for determining image keywords, first obtaining at least one document containing a query graph, extracting candidate words in the document, further determining the image-text similarity between the candidate words and the query graph, and then ranking the candidate words using the statistical information of the candidate words in the document and the image-text similarity with the query graph. The highest ranked candidate words thus obtained have a higher relevance to the document and a higher degree of association with the query graph, and the highest ranked candidate words are selected as keywords corresponding to the query graph, and keywords can be determined from two aspects of relevance to the document and relevance to the query graph, so that the determined keywords are more accurate and can correctly represent the content of the image.

[0075] To facilitate understanding of the method for determining image keywords provided by the embodiment of the present invention, Figure 2 The application scenario shown is explained and illustrated. Figure 2 A schematic diagram of a framework of an exemplary application scenario provided for an embodiment of the present invention.

[0076] like Figure 2As shown, the method for determining image keywords provided by the embodiment of the present invention can be applied to the server 20. In practical applications, the server 20 first obtains the query graph sent by the client, obtains the document containing the query graph according to the query graph, extracts the candidate words in the document, sorts the candidate words according to the statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph, takes the candidate words with the highest ranking as keywords, and sends the keywords of the determined query graph to the client 10, so that the user can obtain the relevant information of the query graph.

[0077] Those skilled in the art will understand that Figure 2 The framework diagram shown is only an example in which the embodiments of the present invention can be implemented. The scope of application of the embodiments of the present invention is not limited by any aspect of the framework.

[0078] It should be noted that the client 10 can be carried on a terminal, which can be any user device that is currently available, under development, or developed in the future, and can interact with each other through any form of wired and / or wireless connection (for example, Wi-Fi, LAN, cellular, coaxial cable, etc.), including but not limited to: smart wearable devices, smart phones, non-smart phones, tablet computers, laptop personal computers, desktop personal computers, small computers, medium-sized computers, mainframe computers, etc. that are currently available, under development, or developed in the future. The embodiments of the present invention are not limited in this regard. It should also be noted that the server 20 in the embodiment of the present invention can be an example of a device that is currently available, under development, or developed in the future, and can provide image keyword queries. The embodiments of the present invention are not limited in this regard.

[0079] To facilitate understanding of the technical solution provided by the embodiment of the present invention, the input method provided by the embodiment of the present invention will be described below with reference to the accompanying drawings.

[0080] See also Figure 3 , which is a flow chart of a method for determining image keywords provided by an embodiment of the present invention. Figure 3 As shown, the method may include S301-S305:

[0081] S301: Obtain at least one document containing a query graph.

[0082] When a user only has a picture and needs to obtain relevant information about the picture, the picture will be used as a query graph to search for relevant information. In an embodiment of the present invention, the query graph can be a picture for which keywords of the picture need to be determined and sent by the user through the client. For example, when a user obtains a scenic spot picture without keywords and other relevant information from the Internet, the scenic spot picture can be used as a query graph to perform a keyword search, obtain the keywords corresponding to the scenic spot picture, and then determine the location or position corresponding to the scenic spot picture.

[0083] After obtaining the query graph, at least one document containing the query graph can be obtained according to the query graph, so as to determine the keywords of the query graph according to the document. It is understandable that the document generally contains pictures related to the content of the document, and correspondingly, the keywords of the pictures can be extracted from the document. In order to improve the accuracy of the keywords of the query graph obtained, at least one document can be obtained according to the query graph. The keywords obtained from multiple documents are more universal, which can avoid the possible errors of the keywords obtained from one document and improve the accuracy of the determined keywords.

[0084] The method of obtaining the document containing the query graph is not limited in the embodiment of the present invention. The image features of the query graph can be obtained first, and then the document containing the image having the image features is obtained from the network according to the image features of the query graph. The type of document is not limited in the embodiment of the present invention, and can be news reports, encyclopedia materials, etc. containing the query graph.

[0085] S302: Extract candidate words from the document.

[0086] It is understandable that a document is composed of many words, but not all words can be used as keywords of the query graph. For example, some prepositions used to connect sentences should not be used as keywords. Therefore, words that can be used as keywords of the query graph can be used as candidate words, and before determining the keywords of the query graph, the candidate words in the document are first extracted to narrow the scope of the keywords of the query graph to be determined later and improve the accuracy of the determined keywords.

[0087] The embodiment of the present invention does not limit the method of extracting candidate words. In one possible implementation, candidate words can be extracted according to the part of speech of the vocabulary, and the vocabulary with noun attributes can be extracted to obtain candidate words. In another possible implementation, a candidate word list can be established first, and the words in the document that belong to the candidate word list are determined as candidate words. The embodiment of the present invention provides an implementation method for extracting candidate words in a document, please refer to the specific implementation method below.

[0088] S303: Determine the image-text similarity between the candidate word and the query graph.

[0089] The candidate word has a graphic similarity with the query graph, and the graphic similarity can be used to indicate the degree of association between the candidate word and the query graph. When the degree of association between the candidate word and the query graph is high, the corresponding graphic similarity between the candidate word and the query graph is high; when the degree of association between the candidate word and the query graph is low, the corresponding graphic similarity is low. For example, when the query graph is a rose, if the candidate word is "rose" or "flower", the corresponding graphic similarity is high; if the candidate word is "soil" or "air" and other words with a low degree of association with the image content, the corresponding graphic similarity is low.

[0090] The method for determining the image-text similarity between the candidate word and the query graph is not limited in the embodiment of the present invention. The characteristics of the keyword corresponding to the query graph can be first determined based on the query graph, and then compared with the characteristics of the candidate word to obtain the image-text similarity. In another possible implementation, the image-text similarity between the candidate word and the query graph can be directly determined based on the candidate word and the query graph. Corresponding to different methods for determining image-text similarity, the embodiment of the present invention provides two implementation methods for determining the image-text similarity between the candidate word and the query graph, which will be described in detail below.

[0091] In an embodiment of the present invention, the graphic and text similarity between the candidate words and the query graph is determined, so that when determining the keywords of the query graph, the graphic and text similarity between the candidate words and the query graph is used as one of the reference bases for determining the keywords, so that the determined keywords have a higher graphic and text similarity with the query graph, and the candidate words with lower graphic and text similarity with the query graph can be excluded, thereby improving the correlation between the keywords and the query graph.

[0092] S304: Sort the candidate words by using the statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph.

[0093] The statistical information of the candidate word in the document may indicate the relevant information of the candidate word appearing in the document, and may specifically include the word frequency, part of speech, inverse document frequency, word length, and distribution of the candidate word. It is understandable that the statistical information of the candidate word in the document may reflect the relevance between the candidate word and the topic of the document. When the relevance between the candidate word and the topic of the document is high, the candidate word may be a keyword corresponding to the query graph.

[0094] The image-text similarity between the candidate word and the query graph can represent the degree of association between the candidate word and the query graph. By using the image-text similarity between the candidate word as one of the bases for ranking the candidate words, keywords can be selected based on the degree of association between the candidate word and the query graph to ensure that the determined keywords have a high degree of association with the query graph.

[0095] The statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph are used as the basis for ranking the candidate words. The candidate words can be ranked from two aspects: the relevance between the candidate words and the document topic and the relevance between the candidate words and the query graph. This makes the keywords obtained according to the ranking results have a higher relevance between the document topic and the query graph, which is more accurate. For example, when the query graph is Figure 1 When the J-10 fighter jet shown in the figure is selected, the candidate words in the document are sorted according to the statistical information and the image-text similarity between the candidate words and the query graph. The highest ranked candidate word is "J-10 fighter jet", rather than "pilot" which has the best statistical information.

[0096] It should be noted that the embodiments of the present invention are not limited to the sorting method for sorting candidate words using the statistical information of the candidate words in the document and the graphic similarity between the candidate words and the query graph. In a possible implementation, the statistical information of the candidate words in the document and the graphic similarity can be scored separately, and the candidate words can be sorted according to the obtained scores of the candidate words.

[0097] In an embodiment of the present invention, candidate words are sorted using statistical information of the candidate words in the document and the graphic similarity between the candidate words and the query graph. The candidate words can be sorted from two aspects: the relevance of the candidate words to the document topic and the degree of association with the query graph. This makes it possible for keywords determined based on the sorting results to have both a high relevance to the document topic and a high degree of association with the query graph. This can avoid the problem of low association between keywords determined based only on the statistical information of the candidate words in the document and the query graph, thereby improving the accuracy of the determined keywords.

[0098] In addition, the embodiment of the present invention provides an implementation of S304, please refer to the specific implementation below.

[0099] S305: Select the candidate word with the highest ranking as the keyword corresponding to the query graph.

[0100] After the candidate words are sorted using statistical information and image-text similarity, the highest-ranked candidate word is used as the keyword corresponding to the query graph. It can be understood that the highest-ranked candidate word is the candidate word with the highest comprehensive ranking of statistical information and image-text similarity among the candidate words, and is the candidate word closest to the query graph content among the candidate words.

[0101] Based on the relevant contents of S301-S305 above, it can be known that in an embodiment of the present invention, by obtaining at least one document containing a query graph, candidate words in the document are extracted, and keywords are selected from the candidate words; the image-text similarity between the candidate words and the query graph is determined, and the candidate words are sorted using the statistical information and image-text similarity of the candidate words in the document, and the candidate words with the highest ranking are used as keywords of the query graph. In this way, the keywords of the query graph can be determined from two aspects: the relevance of the document topic and the association degree of the query graph, so that the keywords of the query graph obtained are more accurate, and the problem of low association degree between the keywords determined only by the relevance to the document topic and the query graph can be avoided.

[0102] It can be understood that in one possible implementation, in order to improve the accuracy of the determined keywords, a candidate word list can be pre-established based on the characteristics of the candidate words. In this way, when extracting candidate words, the candidate words in the document can be determined more accurately, and it can be avoided that words that do not belong to the candidate words are extracted as candidate words, which makes the final determined keywords inaccurate.

[0103] Specifically, extracting candidate words from a document can include the following two steps:

[0104] A1: Segment the document and merge the segmented words to generate merged words.

[0105] It is understandable that there are multiple words in a document. In order to obtain candidate words, the document can be segmented first to obtain the segmented words that make up the document, and then the segmented words obtained can be merged to obtain the merged words, and then the candidate words can be determined. It should be noted that the segmented words obtained by segmenting the document are basic words. By merging the segmented words, a merged word with a complete meaning can be obtained. By merging the segmented words, it can be ensured that the meaning of the merged word is correct, and the meaning of the words in the document is prevented from changing due to the segmented words. For example, the document contains the word "National Museum". Segmenting the document can obtain two segmented words, "National" and "Museum". Obviously, these two segmented words cannot fully express the meaning of the word "National Museum" in the document, so the two segmented words "National" and "Museum" should be merged to obtain the word "National Museum" with a complete meaning.

[0106] The method of merging participles to generate a merged word is not limited in the embodiment of the present invention. In a possible implementation, the participles can be merged with the preceding and following participles in the order of the participles in the document. If a word with a complete meaning is obtained, the word with a complete meaning is used as the generated merged word.

[0107] A2: If the merged word is in the pre-established candidate word list, the merged word is determined as a candidate word.

[0108] In order to more accurately determine the candidate words, a candidate word list may be pre-established, and the candidate word list may be established based on common words. After obtaining the merged word, if the merged word is in the pre-established candidate word list, the merged word is determined as the candidate word.

[0109] Based on the above content, it can be known that by segmenting the document and merging the segmented words, a merged word with a complete meaning can be obtained. If the merged word is in a pre-established candidate word list, the merged word is determined as a candidate word. By segmenting and merging the document, the document can be correctly divided into words to obtain a merged word with a complete meaning. Then, through a pre-established candidate word list, the merged word in the candidate word list can be determined as a candidate word. In this way, the candidate word can be determined more accurately, avoiding errors in determining keywords due to inaccurate candidate words.

[0110] Based on the content of S303 above, it can be known that, in order to determine the image-text similarity between the candidate word and the query graph, in a possible implementation, the image-text similarity can be obtained by first determining the features of the keyword corresponding to the query graph, and then comparing them with the features of the candidate word; the image-text similarity between the candidate word and the query graph can also be determined directly based on the candidate word and the query graph. Correspondingly, the embodiment of the present invention also provides two implementation methods for determining the image-text similarity between the candidate word and the query graph, which are introduced below respectively.

[0111] As a first implementation of S303, see Figure 4 As shown, Figure 4 A method for determining the image-text similarity between a candidate word and a query graph provided by an embodiment of the present invention includes S401-S403:

[0112] S401: Input the query graph into the image keyword feature extraction model to obtain the keyword text features corresponding to the query graph output by the image keyword feature model. The image keyword feature model is trained using the query graph to be trained and the keyword text features corresponding to the query graph to be trained.

[0113] By inputting the query graph into the image keyword model, the keyword text features corresponding to the query graph output by the image keyword model can be obtained. The keyword text features corresponding to the query graph refer to the text features of the keywords of the query graph, which are used to represent the relevant information of the keywords corresponding to the query graph. Based on the obtained keyword text features corresponding to the query graph and the text features of the candidate words, the image-text similarity between the query graph and the candidate words can be further obtained.

[0114] The image keyword feature model can be obtained by training based on the query graph to be trained and the keyword text features corresponding to the query graph to be trained. Among them, the query graph to be trained is a picture with specific picture content. According to the picture content of the query graph to be trained, the keywords corresponding to the picture content are first determined, and then the text features of the keywords are extracted. The types of keyword text features are not limited in the embodiments of the present invention. For example, in a possible implementation method, the word vector of the keyword can be used as the keyword text feature. In actual applications, the keyword corresponding to the query graph to be trained can be input into the word vector model to obtain the word vector of the keyword, and the obtained word vector of the keyword is used as the keyword text feature.

[0115] The query graph to be trained is labeled according to the keyword text features corresponding to the query graph to be trained, and the query graph to be trained with the keyword text feature labels is used as training data. A large number of query graphs to be trained with the keyword text feature labels are used to obtain an image keyword feature model. The trained image keyword feature model can output the keyword text features of the query graph according to the input query graph.

[0116] S402: Obtain text features of candidate words.

[0117] After obtaining the keyword text features of the query graph, the corresponding text features of the candidate words are obtained, which are used to obtain the image-text similarity based on the text features of the candidate words and the keyword text features of the query graph. The text features of the candidate words are features used to represent the relevant information of the candidate words. It should be noted that in order to facilitate the subsequent calculation of the correlation between the text features of the candidate words and the keyword text features corresponding to the query graph, the text features of the candidate words can be consistent with the type of the keyword text features corresponding to the query graph. For example, when the keyword text feature corresponding to the query graph is the word vector of the keyword, the word vector of the candidate word can be obtained through the word vector model as the text feature of the candidate word.

[0118] S403: Calculate the correlation between the text features of the candidate word and the text features of the keyword corresponding to the query graph as the image-text similarity between the candidate word and the query graph.

[0119] The correlation between the text feature of the candidate word and the keyword text feature corresponding to the query graph may refer to the degree of similarity between the text feature of the candidate word and the keyword text feature corresponding to the query graph. Specifically, the degree of similarity may be determined by calculating the distance between the text feature of the candidate word and the keyword text feature corresponding to the query graph. The method for calculating the correlation is not limited in the embodiment of the present invention. For example, when the text feature of the candidate word is the word vector of the candidate word and the keyword text feature corresponding to the query graph is the word vector of the keyword, the degree of similarity may be determined by calculating the cosine similarity between the word vector of the candidate word and the word vector of the keyword.

[0120] By calculating the correlation between the text features of the candidate words and the text features of the keywords corresponding to the query graph, we can get the degree of association between the candidate words and the keywords corresponding to the query graph, that is, the image-text similarity between the candidate words and the query graph.

[0121] Based on the contents of S401-S403 above, it can be known that by obtaining the keyword text features corresponding to the query graph, the correlation between the keyword text features and the text features of the candidate words can be calculated, thereby obtaining the image-text similarity between the candidate words and the query graph, which is used for subsequent sorting of the candidate words according to the image-text similarity.

[0122] As a second implementation of S303, the image-text similarity between the candidate word and the query graph may be obtained through an image-text similarity model. The specific method includes:

[0123] The candidate words and the query graph are input into the image-text similarity determination model to obtain the image-text similarity between the candidate words and the query graph output by the image-text similarity determination model. The image-text similarity determination model is trained using the image-text similarity between the query graph to be trained and the candidate words to be trained.

[0124] The image-text similarity determination model is obtained by training the image-text similarity between the query graph to be trained and the candidate words to be trained, and can be used to output the image-text similarity between the candidate words and the query graph according to the input query graph and the candidate words. The query graph to be trained is a picture with specific image content, and the candidate words to be trained corresponding to different image-text similarities can be determined according to the query graph to be trained, and the candidate words to be trained are marked according to the corresponding image-text similarities to obtain the corresponding image-text similarity labels. The query graph to be trained, the corresponding candidate words to be trained, and the labels of the image-text similarity corresponding to the candidate words to be trained are used as training data, and the image-text similarity determination model is obtained through training with a large amount of training data. Based on the above content, it can be known that by inputting the candidate words and the query graph into the image-text similarity determination model, the image-text similarity between the candidate words and the query graph output by the image-text similarity determination model is obtained, so that the candidate words can be sorted according to the image-text similarity between the query graph and the candidate words in the subsequent.

[0125] In a possible implementation of the embodiment of the present invention, the embodiment of the present invention further provides a specific implementation of S304, including the following two steps:

[0126] B1: The statistical information of the candidate word in the document and the image-text similarity between the candidate word and the query graph are input into the first candidate word feature evaluation model to obtain the evaluation score of the candidate word output by the first candidate word feature evaluation model.

[0127] The statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph are used as the input of the first candidate word feature evaluation model to obtain the evaluation score of the candidate words output by the first candidate word feature evaluation model. Among them, the training data of the first candidate word feature evaluation model can be composed of the statistical information of the candidate words to be trained in the document to be trained, the image-text similarity between the candidate words to be trained and the query graph, and the evaluation score label of the candidate words to be trained, and the evaluation score of the candidate words to be trained can be marked in advance. For example, when the statistical information of the candidate words to be trained in the document to be trained is better and the image-text similarity between the candidate words to be trained and the query word is higher, the corresponding evaluation score of the candidate words to be trained is higher. The first candidate word feature evaluation model trained with a large amount of training data can be used to output the evaluation score corresponding to the candidate word according to the statistical information of the input candidate words in the document and the image-text similarity between the candidate words and the query graph.

[0128] B2: Sort the candidate words according to their evaluation scores.

[0129] The candidate words are sorted according to their evaluation scores so that keywords can be selected according to the sorted positions of the candidate words.

[0130] Based on the above content, it can be known that by inputting the statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph into the first candidate word feature evaluation model, the evaluation score of the candidate words can be obtained, and the candidate words can be sorted according to the obtained evaluation score. Sorting the candidate words by the obtained candidate word evaluation score can make the ranking of the candidate words more accurate, so that the candidate word with the highest ranking position is the most accurate candidate word.

[0131] By using the statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph as the basis for ranking the candidate words, it can be ensured that the obtained keywords have a high degree of relevance to the subject of the document and a high degree of association with the query graph. However, when the credibility or quality of the obtained document itself is not high, for example, the query graph in the document is irrelevant to the text in the document or the quality of the text content in the document is poor, even if the candidate words with a high degree of relevance to the subject of the document and a high degree of association with the query graph are determined as keywords, due to the problems of the document itself, the keywords finally obtained cannot accurately represent the content of the query graph.

[0132] In addition, in general, keywords determined based on the query graph have a greater probability of being common words and only a smaller probability of being unusual words. When a candidate word is an unusual word, it has a greater possibility of belonging to a vocabulary that cannot accurately represent the content of the query graph.

[0133] In order to solve the above problem, an embodiment of the present invention further provides a method for determining a picture keyword. In this embodiment, in addition to the above S301-S305, it also includes:

[0134] The additional features corresponding to the candidate word are obtained. The additional features corresponding to the candidate word are the features of the website to which the document belongs and / or the category information corresponding to the candidate word. The features of the website to which the document belongs corresponding to the candidate word are determined according to the number of visits to the website to which the document belongs corresponding to the candidate word.

[0135] After obtaining at least one document containing the query graph and extracting the candidate words in the document, the characteristics of the website to which the document corresponding to the candidate words belongs are obtained. The quality of the document can be judged by the characteristics of the website to which the document belongs, which can then affect the sorting order of the candidate words. The characteristics of the website to which the document corresponding to the candidate words belongs can be determined based on the number of visits to the website to which the document corresponding to the candidate words belongs. When the number of visits to the website is high, the credibility of the documents in the corresponding website is high, and the candidate words extracted from the document are better; when the number of visits to the website is low, it can be considered that the quality of the documents in the website is low, and correspondingly, the candidate words extracted from the document are poor.

[0136] Considering the usage habits of words, a candidate word list can be pre-established, wherein the candidate word list has category information obtained according to the classification of common words. For example, the words in the candidate word list can be divided into 11 categories, including characters, movies and TV series, books, electronics, attractions, cars, flowers, animals, vegetables, ingredients and others. When the candidate word does not belong to the first 10 categories with specific category names, the candidate word can be classified as "others".

[0137] After determining the candidate words, the category information corresponding to the candidate words can be obtained. If the candidate words belong to a predetermined specific category, the candidate words belong to a relatively common vocabulary; if the candidate words do not belong to the predetermined specific category or belong to the "other" category, the candidate words may not belong to a common vocabulary. Further, the category information corresponding to the candidate words can be used as one of the bases for sorting the candidate words, and the candidate words are sorted according to the category information corresponding to the candidate words.

[0138] In an embodiment of the present invention, by obtaining the characteristics of the website to which the document of the candidate word belongs and / or the category information corresponding to the candidate word, relevant information of the candidate word is obtained from the document of the candidate word and / or the category of the candidate word, so that when the candidate words are subsequently sorted, the candidate words are sorted according to the characteristics of the website to which the document of the candidate word belongs and / or the category information of the candidate words, ensuring that the document quality of the candidate word finally selected is high and / or the candidate word is a more commonly used word, thereby further improving the accuracy of the determined keywords.

[0139] Furthermore, after obtaining the characteristics of the website to which the document corresponding to the candidate word belongs and / or the category information corresponding to the candidate word, when sorting the candidate words, the characteristics of the website to which the document corresponding to the candidate word belongs and / or the category information corresponding to the candidate word can be used as one of the basis for sorting. Specifically, the statistical information of the candidate word in the document and the image-text similarity between the candidate word and the query graph are used to sort each candidate word, including:

[0140] The candidate words are ranked using the statistical information of the candidate words in the document, the image-text similarity between the candidate words and the query graph, and the additional features of the candidate words.

[0141] After obtaining the features of the website to which the document corresponding to the candidate word belongs and / or the category information corresponding to the candidate word, the features of the website to which the document corresponding to the candidate word belongs and / or the category information corresponding to the candidate word can be used as additional features. The candidate words are sorted using the statistical information of the candidate words in the document, the image-text similarity between the candidate words and the query graph, and the additional features. According to the relevance of the candidate words to the subject of the document, the degree of association with the query graph, the quality of the candidate words' documents, and / or the commonness of the candidate words, the keywords obtained after sorting the candidate words have high relevance to the subject of the document, the degree of association with the query graph, and the quality and / or commonness of the document. The keywords obtained thereby have high accuracy and can better represent the content of the query graph.

[0142] In an embodiment of the present invention, additional features are added when sorting candidate words. The additional features are features of the website to which the document corresponding to the candidate word belongs and / or category information corresponding to the candidate word. On the basis of sorting according to the relevance of the candidate word to the document and the degree of association with the query graph, the document quality of the candidate word and / or the common use of the candidate word are added, so that the ranking of the candidate words is more accurate and the keywords obtained according to the ranking of the candidate words are more associated with the query graph.

[0143] Based on the above-mentioned content of sorting candidate words by using the statistical information of candidate words in the document, the image-text similarity between the candidate words and the query graph, and the additional features of the candidate words, the embodiment of the present invention further provides an implementation method of sorting candidate words by using the statistical information of candidate words in the document, the image-text similarity with the query graph, and the additional features of the candidate words, which specifically includes:

[0144] Inputting statistical information of the candidate word in the document, the image-text similarity between the candidate word and the query graph, and the additional features of the candidate word into a second candidate word feature evaluation model to obtain an evaluation score of the candidate word output by the second candidate word feature evaluation model, wherein the second candidate word feature evaluation model is trained based on statistical information of the candidate word to be trained in the document to be trained, the image-text similarity between the candidate word to be trained and the query graph to be trained, the additional features of the candidate word to be trained, and the evaluation score of the candidate word to be trained, wherein the additional features of the candidate word to be trained are features of the website to which the document corresponding to the candidate word to be trained belongs and / or category information corresponding to the candidate word to be trained;

[0145] The candidate words are ranked according to their evaluation scores.

[0146] It should be noted that the additional features of the candidate word are the features of the website to which the document corresponding to the candidate word belongs and / or the category information corresponding to the candidate word. Correspondingly, when the additional features of the candidate word contain different types of features, the corresponding second candidate word feature evaluation model is different.

[0147] When the additional features of the candidate word are the features of the website to which the document corresponding to the candidate word belongs, the corresponding second candidate word feature evaluation model is trained by the statistical information of the candidate word to be trained in the document to be trained, the image-text similarity between the candidate word to be trained and the query graph to be trained, the features of the website to which the document corresponding to the candidate word to be trained belongs, and the evaluation score of the candidate word to be trained. Among them, the evaluation score of the candidate word to be trained can be marked according to the statistical information of the candidate word to be trained in the document to be trained, the image-text relevance between the candidate word to be trained and the query graph to be trained, and the features of the website to which the document corresponding to the candidate word to be trained belongs. The trained second candidate word feature evaluation model can output the evaluation score of the candidate word according to the input candidate word statistical information, the image-text similarity between the candidate word and the query graph, and the features of the website to which the document corresponding to the candidate word belongs.

[0148] When the additional feature of the candidate word is the category information corresponding to the candidate word, the corresponding second candidate word feature evaluation model is trained by the statistical information of the candidate word to be trained in the document to be trained, the image-text similarity between the candidate word to be trained and the query graph to be trained, the category information corresponding to the candidate word to be trained, and the evaluation score of the candidate word to be trained. The evaluation score of the candidate word to be trained can be marked according to the statistical information of the candidate word to be trained in the document to be trained, the image-text relevance between the candidate word to be trained and the query graph to be trained, and the category information corresponding to the candidate word to be trained. The trained second candidate word feature evaluation model can output the evaluation score of the candidate word according to the input candidate word statistical information, the image-text similarity between the candidate word and the query graph, and the category information corresponding to the candidate word.

[0149] When the additional features of the candidate word are the features of the website to which the document corresponding to the candidate word belongs and the category information corresponding to the candidate word, the corresponding additional features of the candidate word to be trained are the features of the website to which the document corresponding to the candidate word to be trained belongs and the category information corresponding to the candidate word to be trained. The second candidate word feature evaluation model is trained based on the statistical information of the candidate word to be trained in the document to be trained, the image-text similarity between the candidate word to be trained and the query graph to be trained, the features of the website to which the document corresponding to the candidate word to be trained belongs, the category information corresponding to the candidate word to be trained, and the evaluation score label of the candidate word to be trained as training data. Among them, the evaluation score of the candidate word to be trained can be obtained by marking the evaluation score of the candidate word to be trained based on the statistical information of the candidate word to be trained in the document to be trained, the image-text relevance between the candidate word to be trained and the query graph to be trained, the features of the website to which the document corresponding to the candidate word to be trained, and the category information corresponding to the candidate word to be trained. The trained second candidate word feature evaluation model can output the evaluation score of the candidate word based on the input candidate word statistical information, the image-text similarity between the candidate word and the query graph, the features of the website to which the document corresponding to the candidate word belongs, and the category information corresponding to the candidate word.

[0150] Thus, the second candidate word feature evaluation model can output the evaluation score of the candidate word according to the statistical information of the input candidate word in the document, the image-text similarity between the candidate word and the query graph to be trained, and the additional features of the candidate word. According to the evaluation scores of each candidate word, the candidate words can be sorted so that the candidate word with the highest ranking is determined as a keyword. In an embodiment of the present invention, the evaluation score of the candidate word can be obtained by the second candidate word feature evaluation model, and the candidate words can be sorted according to the evaluation score of the candidate word, so that the ranking result of the candidate word is more accurate, and then the keyword finally obtained according to the sorting position is the candidate word with the highest degree of association with the query graph among the candidate words, which can more accurately represent the content of the query graph.

[0151] Based on the picture keyword determination method provided by the above method embodiment, the embodiment of the present invention further provides a picture keyword determination device, which is introduced below in conjunction with the accompanying drawings.

[0152] See also Figure 5 , which is a structural schematic diagram of a device for determining image keywords provided by an embodiment of the present invention.

[0153] A document acquisition unit 501, configured to acquire at least one document containing the query graph;

[0154] A candidate word extraction unit 502, used to extract candidate words from the document;

[0155] A picture-text similarity determination unit 503, used to determine the picture-text similarity between the candidate word and the query graph;

[0156] A sorting unit 504, configured to sort each of the candidate words by using statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph;

[0157] The keyword determination unit 505 is used to select the candidate word with the highest ranking as the keyword corresponding to the query graph.

[0158] Optionally, the candidate word extraction unit 502 includes:

[0159] A merge word generation subunit, used for segmenting the document and merging the obtained segmented words to generate a merge word;

[0160] The candidate word determination subunit is used to determine the merged word as a candidate word if the merged word is in a pre-established candidate word list.

[0161] Optionally, the image-text similarity determination unit 503 includes:

[0162] A keyword text feature acquisition subunit is used to input the query graph into a picture keyword feature extraction model to obtain keyword text features corresponding to the query graph output by the picture keyword feature model, wherein the picture keyword feature model is trained using the query graph to be trained and the keyword text features corresponding to the query graph to be trained;

[0163] A candidate word text feature acquisition subunit, used to acquire text features of the candidate word;

[0164] The correlation calculation subunit is used to calculate the correlation between the text features of the candidate word and the text features of the keywords corresponding to the query graph as the image-text similarity between the candidate word and the query graph.

[0165] Optionally, the image-text similarity determination unit 503 is specifically used to input the candidate word and the query graph into an image-text similarity determination model to obtain the image-text similarity between the candidate word and the query graph output by the image-text similarity determination model, and the image-text similarity determination model is trained using the image-text similarity between the query graph to be trained and the candidate word to be trained.

[0166] Optionally, the sorting unit 504 includes:

[0167] A first evaluation score acquisition subunit is used to input the statistical information of the candidate word in the document and the image-text similarity between the candidate word and the query graph into a first candidate word feature evaluation model to obtain the evaluation score of the candidate word output by the first candidate word feature evaluation model, wherein the first candidate word feature evaluation model is trained based on the statistical information of the candidate word to be trained in the document to be trained, the image-text similarity between the candidate word to be trained and the query graph to be trained, and the evaluation score of the candidate word to be trained;

[0168] The first sorting subunit is used to sort the candidate words according to the evaluation scores of the candidate words.

[0169] Optionally, the device further comprises:

[0170] The additional feature acquisition unit is used to obtain additional features corresponding to the candidate word, wherein the additional features of the candidate word are the features of the website to which the document corresponding to the candidate word belongs and / or the category information corresponding to the candidate word; the features of the website to which the document corresponding to the candidate word belongs are determined based on the number of visits to the website to which the document corresponding to the candidate word belongs.

[0171] Optionally, the ranking unit 504 is specifically configured to rank each of the candidate words by utilizing statistical information of the candidate words in the document, the image-text similarity between the candidate words and the query graph, and additional features of the candidate words.

[0172] Optionally, the sorting unit 504 includes:

[0173] A second evaluation score acquisition subunit is used to input statistical information of the candidate word in the document, the image-text similarity between the candidate word and the query graph, and additional features of the candidate word into a second candidate word feature evaluation model to obtain an evaluation score of the candidate word output by the second candidate word feature evaluation model, wherein the second candidate word feature evaluation model is trained based on statistical information of the candidate word to be trained in the document to be trained, the image-text similarity between the candidate word to be trained and the query graph to be trained, the additional features of the candidate word to be trained, and the evaluation score of the candidate word to be trained, wherein the additional features of the candidate word to be trained are features of the website to which the document corresponding to the candidate word to be trained belongs and / or category information corresponding to the candidate word to be trained;

[0174] The second sorting subunit is used to sort the candidate words according to the evaluation scores of the candidate words.

[0175] Figure 61 shows a block diagram of a client 1200. For example, the client 1200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0176] Reference Figure 6 The client 1200 may include one or more of the following components: a processing component 1202 , a memory 1204 , a power component 1206 , a multimedia component 1208 , an audio component 1210 , an input / output (I / O) interface 1212 , a sensor component 1214 , and a communication component 1216 .

[0177] The processing component 1202 generally controls the overall operation of the client 1200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1202 may include one or more processors 1220 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 1202 may include one or more modules to facilitate the interaction between the processing component 1202 and other components. For example, the processing component 1202 may include a multimedia module to facilitate the interaction between the multimedia component 1208 and the processing component 1202.

[0178] The memory 1204 is configured to store various types of data to support operations on the client 1200. Examples of such data include instructions for any application or method operating on the client 1200, contact data, phone book data, messages, pictures, videos, etc. The memory 1204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0179] The power supply component 1206 provides power to the various components of the client 1200. The power supply component 1206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the client 1200.

[0180] The multimedia component 1208 includes a screen that provides an output interface between the client 1200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1208 includes a front camera and / or a rear camera. When the client 1200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0181] The audio component 1210 is configured to output and / or input audio signals. For example, the audio component 1210 includes a microphone (MIC), and when the client 1200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1204 or sent via the communication component 1216. In some embodiments, the audio component 1210 also includes a speaker for outputting audio signals.

[0182] The I / O interface provides an interface between the processing component 1202 and the peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0183] The sensor assembly 1214 includes one or more sensors for providing various aspects of status assessment for the client 1200. For example, the sensor assembly 1214 can detect the open / closed state of the device 1200, the relative positioning of components, such as the display and keypad of the client 1200, the sensor assembly 1214 can also detect the position change of the client 1200 or a component of the client 1200, the presence or absence of user contact with the client 1200, the orientation or acceleration / deceleration of the client 1200, and the temperature change of the client 1200. The sensor assembly 1214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 1214 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1214 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0184] The communication component 1216 is configured to facilitate wired or wireless communication between the client 1200 and other devices. The client 1200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0185] In an exemplary embodiment, the client 1200 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the following methods:

[0186] Obtain at least one document containing the query graph;

[0187] Extracting candidate words from the document;

[0188] Determining the image-text similarity between the candidate word and the query graph;

[0189] Rank the candidate words using statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph;

[0190] The candidate word with the highest ranking is selected as the keyword corresponding to the query graph.

[0191] Optionally, extracting candidate words from the document includes:

[0192] Segmenting the document, and merging the obtained segmented words to generate merged words;

[0193] If the merged word is in a pre-established candidate word list, the merged word is determined as a candidate word.

[0194] Optionally, determining the image-text similarity between the candidate word and the query graph includes:

[0195] Inputting the query graph into a picture keyword feature extraction model to obtain keyword text features corresponding to the query graph output by the picture keyword feature model, wherein the picture keyword feature model is trained using the query graph to be trained and the keyword text features corresponding to the query graph to be trained;

[0196] Obtaining text features of the candidate words;

[0197] The correlation between the text features of the candidate word and the text features of the keywords corresponding to the query graph is calculated as the image-text similarity between the candidate word and the query graph.

[0198] Optionally, determining the image-text similarity between the candidate word and the query graph includes:

[0199] The candidate word and the query graph are input into a picture-text similarity determination model to obtain the picture-text similarity between the candidate word and the query graph output by the picture-text similarity determination model, wherein the picture-text similarity determination model is trained using the picture-text similarity between the query graph to be trained and the candidate word to be trained.

[0200] Optionally, the step of ranking the candidate words by using statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph includes:

[0201] Inputting statistical information of the candidate word in the document and the image-text similarity between the candidate word and the query graph into a first candidate word feature evaluation model to obtain an evaluation score of the candidate word output by the first candidate word feature evaluation model, wherein the first candidate word feature evaluation model is trained based on statistical information of the candidate word to be trained in the document to be trained, the image-text similarity between the candidate word to be trained and the query graph to be trained, and the evaluation score of the candidate word to be trained;

[0202] The candidate words are sorted according to their evaluation scores.

[0203] Optionally, the method further includes:

[0204] Obtain additional features corresponding to the candidate word, wherein the additional features of the candidate word are features of the website to which the document corresponding to the candidate word belongs and / or category information corresponding to the candidate word; the features of the website to which the document corresponding to the candidate word belongs are determined based on the number of visits to the website to which the document corresponding to the candidate word belongs.

[0205] Optionally, the step of ranking the candidate words by using statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph includes:

[0206] The candidate words are sorted by using statistical information of the candidate words in the document, the image-text similarity between the candidate words and the query graph, and additional features of the candidate words.

[0207] Optionally, the ranking of the candidate words by using statistical information of the candidate words in the document, the image-text similarity between the candidate words and the query graph, and additional features of the candidate words includes:

[0208] Inputting statistical information of the candidate word in the document, the image-text similarity between the candidate word and the query graph, and the additional features of the candidate word into a second candidate word feature evaluation model to obtain an evaluation score of the candidate word output by the second candidate word feature evaluation model, wherein the second candidate word feature evaluation model is trained based on statistical information of the candidate word to be trained in the document to be trained, the image-text similarity between the candidate word to be trained and the query graph to be trained, the additional features of the candidate word to be trained, and the evaluation score of the candidate word to be trained, wherein the additional features of the candidate word to be trained are features of the website to which the document corresponding to the candidate word to be trained belongs and / or category information corresponding to the candidate word to be trained;

[0209] The candidate words are sorted according to their evaluation scores.

[0210] Figure 7 1 is a schematic diagram of the structure of the server in an embodiment of the present invention. The server 1300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1322 (for example, one or more processors) and memory 1332, and one or more storage media 1330 (for example, one or more mass storage devices) storing application programs 1342 or data 1344. Among them, the memory 1332 and the storage medium 1330 can be short-term storage or permanent storage. The program stored in the storage medium 1330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 1322 can be configured to communicate with the storage medium 1330 to execute a series of instruction operations in the storage medium 1330 on the server 1300.

[0211] The server 1300 may also include one or more power supplies 1326, one or more wired or wireless network interfaces 1350, one or more input and output interfaces 1356, one or more keyboards 1356, and / or, one or more operating systems 1341, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0212] In addition, an embodiment of the present invention further provides a computer-readable medium having instructions stored thereon, which, when executed by one or more processors, enables a device to execute the above-mentioned method for determining picture keywords.

[0213] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0214] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0215] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0216] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0217] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for determining image keywords, characterized in that: The method comprises: Obtain at least one document containing the query graph; Extracting candidate words from the document; Determining the image-text similarity between the candidate word and the query graph; Acquire additional features corresponding to the candidate word, wherein the additional features of the candidate word are features of a website to which the document corresponding to the candidate word belongs and category information corresponding to the candidate word, wherein the features of the website to which the document corresponding to the candidate word belongs are determined according to the number of visits to the website to which the document corresponding to the candidate word belongs; Using statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph, the candidate words are sorted, including: using statistical information of the candidate words in the document, the image-text similarity between the candidate words and the query graph, and additional features of the candidate words, the candidate words are sorted, the statistical information including the word frequency, part of speech, reverse document frequency, word length, and distribution of the candidate words; The candidate word with the highest ranking is selected as the keyword corresponding to the query graph.

2. The method according to claim 1, characterized in that The extracting candidate words from the document comprises: Segmenting the document, and merging the obtained segmented words to generate merged words; If the merged word is in a pre-established candidate word list, the merged word is determined as a candidate word.

3. The method according to claim 1, characterized in that The determining of the image-text similarity between the candidate word and the query graph includes: Inputting the query graph into a picture keyword feature extraction model to obtain keyword text features corresponding to the query graph output by the picture keyword feature model, wherein the picture keyword feature model is trained using the query graph to be trained and the keyword text features corresponding to the query graph to be trained; Obtaining text features of the candidate words; Calculating the correlation between the text features of the candidate word and the text features of the keywords corresponding to the query graph as the image-text similarity between the candidate word and the query graph; or, The candidate word and the query graph are input into a picture-text similarity determination model to obtain the picture-text similarity between the candidate word and the query graph output by the picture-text similarity determination model, wherein the picture-text similarity determination model is trained using the picture-text similarity between the query graph to be trained and the candidate word to be trained.

4. The method according to claim 1, characterized in that: The step of ranking the candidate words by using the statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph includes: Inputting statistical information of the candidate word in the document and the image-text similarity between the candidate word and the query graph into a first candidate word feature evaluation model to obtain an evaluation score of the candidate word output by the first candidate word feature evaluation model, wherein the first candidate word feature evaluation model is trained based on statistical information of the candidate word to be trained in the document to be trained, the image-text similarity between the candidate word to be trained and the query graph to be trained, and the evaluation score of the candidate word to be trained; The candidate words are sorted according to their evaluation scores.

5. The method according to claim 1, characterized in that The method of ranking each of the candidate words by using the statistical information of the candidate words in the document, the image-text similarity between the candidate words and the query graph, and the additional features of the candidate words comprises: Inputting statistical information of the candidate word in the document, the image-text similarity between the candidate word and the query graph, and the additional features of the candidate word into a second candidate word feature evaluation model to obtain an evaluation score of the candidate word output by the second candidate word feature evaluation model, wherein the second candidate word feature evaluation model is trained based on statistical information of the candidate word to be trained in the document to be trained, the image-text similarity between the candidate word to be trained and the query graph to be trained, the additional features of the candidate word to be trained, and the evaluation score of the candidate word to be trained, wherein the additional features of the candidate word to be trained are features of the website to which the document corresponding to the candidate word to be trained belongs and / or category information corresponding to the candidate word to be trained; The candidate words are sorted according to their evaluation scores.

6. A device for determining image keywords, characterized in that: The device comprises: A document acquisition unit, configured to acquire at least one document containing the query graph; A candidate word extraction unit, used to extract candidate words from the document; A graphic-text similarity determination unit is used to determine the graphic-text similarity between the candidate word and the query graph; an additional feature acquisition unit is used to acquire additional features corresponding to the candidate word, wherein the additional features of the candidate word are features of the website to which the document corresponding to the candidate word belongs and category information corresponding to the candidate word, wherein the features of the website to which the document corresponding to the candidate word belongs are determined based on the number of visits to the website to which the document corresponding to the candidate word belongs; A sorting unit is used to sort each of the candidate words by using the statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph, including: sorting each of the candidate words by using the statistical information of the candidate words in the document, the image-text similarity between the candidate words and the query graph, and additional features of the candidate words, wherein the statistical information includes the word frequency, part of speech, reverse document frequency, word length, and distribution of the candidate words; The keyword determination unit is used to select the candidate word with the highest ranking as the keyword corresponding to the query graph.

7. The device according to claim 6, characterized in that The candidate word extraction unit comprises: A merge word generation subunit, used for segmenting the document and merging the obtained segmented words to generate a merge word; The candidate word determination subunit is used to determine the merged word as a candidate word if the merged word is in a pre-established candidate word list.

8. The device according to claim 6, characterized in that The image-text similarity determination unit comprises: A keyword text feature acquisition subunit is used to input the query graph into a picture keyword feature extraction model to obtain keyword text features corresponding to the query graph output by the picture keyword feature model, wherein the picture keyword feature model is trained using the query graph to be trained and the keyword text features corresponding to the query graph to be trained; A candidate word text feature acquisition subunit, used to acquire text features of the candidate word; A correlation calculation subunit, used to calculate the correlation between the text features of the candidate word and the text features of the keywords corresponding to the query graph as the image-text similarity between the candidate word and the query graph; or, The image-text similarity determination unit is specifically used to input the candidate word and the query graph into an image-text similarity determination model to obtain the image-text similarity between the candidate word and the query graph output by the image-text similarity determination model. The image-text similarity determination model is trained using the image-text similarity between the query graph to be trained and the candidate word to be trained.

9. The device according to claim 6, characterized in that The sorting unit comprises: A first evaluation score acquisition subunit is used to input the statistical information of the candidate word in the document and the image-text similarity between the candidate word and the query graph into a first candidate word feature evaluation model to obtain the evaluation score of the candidate word output by the first candidate word feature evaluation model, wherein the first candidate word feature evaluation model is trained based on the statistical information of the candidate word to be trained in the document to be trained, the image-text similarity between the candidate word to be trained and the query graph to be trained, and the evaluation score of the candidate word to be trained; The first sorting subunit is used to sort the candidate words according to the evaluation scores of the candidate words.

10. The device according to claim 6, characterized in that The sorting unit comprises: A second evaluation score acquisition subunit is used to input statistical information of the candidate word in the document, the image-text similarity between the candidate word and the query graph, and additional features of the candidate word into a second candidate word feature evaluation model to obtain an evaluation score of the candidate word output by the second candidate word feature evaluation model, wherein the second candidate word feature evaluation model is trained based on statistical information of the candidate word to be trained in the document to be trained, the image-text similarity between the candidate word to be trained and the query graph to be trained, the additional features of the candidate word to be trained, and the evaluation score of the candidate word to be trained, wherein the additional features of the candidate word to be trained are features of the website to which the document corresponding to the candidate word to be trained belongs and / or category information corresponding to the candidate word to be trained; The second sorting subunit is used to sort the candidate words according to the evaluation scores of the candidate words.

11. A device for determining image keywords, characterized in that: The system comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the following operations: Obtain at least one document containing the query graph; Extracting candidate words from the document; Determining the image-text similarity between the candidate word and the query graph; Acquire additional features corresponding to the candidate word, wherein the additional features of the candidate word are features of a website to which the document corresponding to the candidate word belongs and category information corresponding to the candidate word, wherein the features of the website to which the document corresponding to the candidate word belongs are determined according to the number of visits to the website to which the document corresponding to the candidate word belongs; Using statistical information of the candidate words in the document and the image-text similarity between the candidate words and the query graph, the candidate words are sorted, including: using statistical information of the candidate words in the document, the image-text similarity between the candidate words and the query graph, and additional features of the candidate words, the candidate words are sorted, the statistical information including the word frequency, part of speech, reverse document frequency, word length, and distribution of the candidate words; The candidate word with the highest ranking is selected as the keyword corresponding to the query graph.

12. A computer-readable medium having instructions stored thereon, which, when executed by one or more processors, causes a device to perform the method for determining picture keywords as described in one or more of claims 1 to 5.

Citation Information

Patent Citations

  • Keyword extraction method and device

    CN110019675A