Data content indexing method, device and electronic equipment

Abstract

The embodiment of the present invention provides a data content indexing method, device and electronic device, which are applied to the field of data processing technology. The method includes: based on the inverted index method, determining multiple initial data contents containing index segmentation words of the index statement in the text domain; for each initial data content, using a preset forward index, obtaining each text domain of the initial data content, wherein the forward index is an index mapped from the content identifier of the data content to the text domain; for each initial data content, based on the matching segmentation words in each text domain of the initial data content, determining the matching degree between the initial data content and the index statement, wherein the matching segmentation words of any text domain are the segmentation words in the text domain that belong to the index statement; sorting each initial data content according to the matching degree between each initial data content and the index statement, and obtaining the index result corresponding to the index statement. Through this scheme, the accuracy of the index result can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data content indexing method, device and electronic equipment. Background Art

[0002] Indexing service is a basic service function provided by data content providers. Through indexing service, users can quickly and efficiently find the data content they want, such as videos, news, products, etc.

[0003] Related technology, a data content indexing method includes: based on an inverted index method, determining multiple initial data contents containing index segmentation words of an index statement in a text domain; for each initial data content, using an index relationship corresponding to the initial data content, determining the matching degree between the initial data content and the index statement, and then, using the matching degree between each initial data content and the index statement, sorting each initial data content to obtain an index result corresponding to the index statement. Among them, the index relationship corresponding to any initial data content is: the corresponding relationship between the index segmentation of the data content and the initial data content is found; and a text domain of the data content is the text corresponding to an attribute field of the data content.

[0004] However, in the process of determining multiple initial data contents, there will be a problem of index truncation, that is, when too much data content is found through a certain index participle, part of the data content found through the index participle will be discarded.

[0005] Then, the index relationship corresponding to the initial data content may be an index relationship for some index participles. In this way, when determining the matching degree between the initial data content and the index statement, only the correlation with some index participles is considered, resulting in inaccurate matching and low accuracy of the final index result. Summary of the invention

[0006] The purpose of the embodiments of the present invention is to provide a data content indexing method, device and electronic device to provide accuracy of indexing results. The specific technical solution is as follows:

[0007] In a first aspect, an embodiment of the present invention provides a data content indexing method, the method comprising:

[0008] Based on the inverted index method, determine multiple initial data contents of index words containing index sentences in the text domain;

[0009] For each initial data content, using a preset forward index, each text field of the initial data content is obtained, wherein the forward index is an index mapped from a content identifier of the data content to a text field;

[0010] For each initial data content, based on the matching segmented words in each text field of the initial data content, determining the matching degree between the initial data content and the index statement, wherein the matching segmented words of any text field are the segmented words in the text field that belong to the index statement;

[0011] According to the matching degree between each initial data content and the index statement, each initial data content is sorted to obtain the index result corresponding to the index statement.

[0012] In one embodiment, the determining the matching degree between the initial data content and the index statement based on the matching segmentation of each text field of the initial data content includes:

[0013] For each text domain of the initial data content, based on the matching segmentation of the text domain and the index statement, determining an evaluation value corresponding to the text domain, wherein the evaluation value is used to represent the degree of similarity between the text domain and the index statement;

[0014] The matching degree between the initial data content and the index statement is determined according to the evaluation values ​​corresponding to the various text fields of the initial data content.

[0015] In one embodiment, the evaluation value includes:

[0016] One or more of a coverage value representing that the index statement is covered by the text domain, a coverage value representing that the text domain is covered by the index statement, and a target distance value;

[0017] The target distance value is the distance information between two matching words that are adjacent in the index statement and included in the text domain in the text domain.

[0018] In one embodiment, the evaluation value includes: a coverage value representing the coverage of the index statement by the text domain;

[0019] The step of determining the evaluation value corresponding to the text domain based on the matching segmentation words of the text domain and the index statement includes:

[0020] Obtaining the index weight of each matching word in the text domain in the index statement, and summing the index weights of each matching word in the text domain to obtain the index weight sum value;

[0021] Calculating a ratio of the number of characters in the index statement to the number of characters in the text field as a character ratio;

[0022] The character ratio and the index weight sum are summed to obtain an evaluation value corresponding to the text field.

[0023] In one embodiment, the evaluation value includes: a coverage value representing the degree of coverage of the text domain by the index statement;

[0024] The step of determining the evaluation value corresponding to the text domain based on the matching segmentation words of the text domain and the index statement includes:

[0025] Obtaining the text weight of the matching word of the text domain in the text domain, and summing the text weights of each matching word in the text domain to obtain a text weight sum value;

[0026] Calculating a ratio of the number of characters in the index statement to the number of characters in the text field as a character ratio;

[0027] The character ratio and the text weight sum are summed to obtain an evaluation value corresponding to the text field.

[0028] In one embodiment, the evaluation value includes a target distance value;

[0029] The step of determining the evaluation value corresponding to the text domain based on the matching segmentation words of the text domain and the index statement includes:

[0030] Determine two matching participles that are adjacent to the index statement and included in the text domain;

[0031] Obtaining a proximity probability between the two matching participles, wherein the proximity probability indicates a proximity probability of the two matching participles being adjacent when they appear in the same sentence;

[0032] Calculating the sum of the text weights of the two matching segmentations in the text domain, and determining the character distance between the two matching segmentations in the text domain and the total number of characters in the text domain;

[0033] Based on the adjacent probability, the sum of the text weights, the character distance and the total number of characters, target distance information of the two matching word segments in the text domain is determined.

[0034] In one embodiment, determining the target distance information of the two matching segmented words in the text domain based on the adjacent probability, the sum of the text weights, the character distance, and the total number of characters includes:

[0035] According to the following formula, the target distance information of the two matching words in the text domain is determined:

[0036]

[0037] Wherein, term_proximity[i][i+1] is the proximity probability between the i-th index participle and the i+1-th index participle in the index statement, w[i][i+1] is the sum of the text weights of the i-th index participle and the i+1-th index participle, dist is the character distance between the i-th index participle and the i+1-th index participle in the text domain, and len_field is the total number of characters in the text domain.

[0038] In one embodiment, determining the degree of matching between the initial data content and the index statement according to the evaluation values ​​corresponding to the various text fields of the initial data content includes:

[0039] The evaluation values ​​corresponding to the text fields of the initial data content are input into a pre-trained neural network model for determining the degree of matching, and the result output by the neural network model is used as the degree of matching between the initial data content and the index statement; wherein the neural network model is a model trained using sample evaluation values ​​and the annotation results of the sample evaluation values.

[0040] In a second aspect, an embodiment of the present invention provides a data content indexing device, the device comprising:

[0041] A data content determination module, used to determine a plurality of initial data contents of index words containing index sentences in a text domain based on an inverted index method;

[0042] A text domain acquisition module, for acquiring, for each initial data content, each text domain of the initial data content using a preset positive index, wherein the positive index is an index mapped from a content identifier of the data content to a text domain;

[0043] A matching degree determination module is used to determine the matching degree between each initial data content and the index statement based on the matching segmented words in each text field of the initial data content, wherein the matching segmented words of any text field are the segmented words in the text field that belong to the index statement;

[0044] The index result determination module is used to sort the initial data contents according to the matching degree between the initial data contents and the index statement, so as to obtain the index result corresponding to the index statement.

[0045] In one embodiment, the matching degree determination module is specifically used to determine, for each text domain of the initial data content, an evaluation value corresponding to the text domain based on the matching word segmentation of the text domain and the index statement, wherein the evaluation value is used to characterize the degree of similarity between the text domain and the index statement; and determine the matching degree of the initial data content and the index statement based on the evaluation values ​​corresponding to each text domain of the initial data content.

[0046] In one embodiment, the evaluation value includes: one or more of: a coverage value representing the coverage of the index statement by the text domain, a coverage value representing the coverage of the text domain by the index statement, and a target distance value; wherein the target distance value is the distance information in the text domain between two matching word segments that are adjacent in the index statement and contained in the text domain.

[0047] In one embodiment, the evaluation value includes: a coverage value representing the coverage of the index statement by the text domain;

[0048] The matching degree determination module is specifically used to obtain the index weight of each matching word in the text domain in the index statement, and sum the index weights of each matching word in the text domain to obtain the index weight sum value; calculate the ratio of the number of characters in the index statement and the text domain as the character ratio; sum the character ratio and the index weight sum value to obtain the evaluation value corresponding to the text domain.

[0049] In one embodiment, the evaluation value includes: a coverage value representing the degree of coverage of the text domain by the index statement;

[0050] The matching degree determination module is specifically used to obtain the text weight of the matching segmentation of the text domain in the text domain, and sum the text weights of each matching segmentation in the text domain to obtain a text weight sum value; calculate the ratio of the number of characters of the index statement and the text domain as a character ratio; sum the character ratio and the text weight sum value to obtain an evaluation value corresponding to the text domain.

[0051] In one embodiment, the evaluation value includes a target distance value;

[0052] The matching degree determination module is specifically used to determine two matching segmentations that are adjacent to the index sentence and included in the text domain; obtain the proximity probability between the two matching segmentations, wherein the proximity probability represents the proximity probability that the two matching segmentations are adjacent when they appear in the same sentence; calculate the sum of the text weights of the two matching segmentations in the text domain, and determine the character distance between the two matching segmentations in the text domain and the total number of characters in the text domain; and determine the target distance information of the two matching segmentations in the text domain based on the proximity probability, the sum of the text weights, the character distance and the total number of characters.

[0053] In one embodiment, the matching degree determination module is specifically used to determine the target distance information of the two matching segmented words in the text domain according to the following formula:

[0054]

[0055] Wherein, term_proximity[i][i+1] is the proximity probability between the i-th index participle and the i+1-th index participle in the index statement, w[i][i+1] is the sum of the text weights of the i-th index participle and the i+1-th index participle, dist is the character distance between the i-th index participle and the i+1-th index participle in the text domain, and len_field is the total number of characters in the text domain.

[0056] In one embodiment, the matching degree determination module is specifically used to input the evaluation values ​​corresponding to each text field of the initial data content into a pre-trained neural network model for determining the matching degree, and use the result output by the neural network model as the matching degree between the initial data content and the index statement; wherein the neural network model is a model trained by sample evaluation values ​​and the annotation results of the sample evaluation values.

[0057] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0058] Memory, used to store computer programs;

[0059] The processor is used to implement any method step described in the first aspect when executing a program stored in the memory.

[0060] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of the first aspects are implemented.

[0061] Beneficial effects of the embodiments of the present invention:

[0062] In the data content indexing method provided by the embodiment of the present invention, after determining multiple initial data contents, the preset forward index can be used to obtain each text domain of the initial data content, and then based on the matching segmentation words in each text domain, the matching degree between the initial data content and the index statement is determined. Since the forward index can be used to obtain each text domain of the initial data content, even if part of the data content found by the index segmentation is discarded when performing the inverted index, the relevance of the complete index segmentation words can be determined from each text domain in the initial data content, thereby improving the accuracy of the matching degree and improving the accuracy of the final index result.

[0063] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying creative work.

[0065] Figure 1 A first flow chart of a data content indexing method provided by an embodiment of the present invention;

[0066] Figure 2 A second flow chart of the data content indexing method provided by an embodiment of the present invention;

[0067] Figure 3 A third flow chart of the data content indexing method provided by an embodiment of the present invention;

[0068] Figure 4 A fourth flow chart of the data content indexing method provided by an embodiment of the present invention;

[0069] Figure 5 A fifth flow chart of the data content indexing method provided by an embodiment of the present invention;

[0070] Figure 6 A schematic diagram of the structure of a data content indexing device provided by an embodiment of the present invention;

[0071] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] In order to improve the accuracy of indexing results, embodiments of the present invention provide a data content indexing method, device, and electronic device.

[0074] The embodiments of the present invention can be applied to various electronic devices, such as personal computers, servers, mobile phones and other devices with data processing capabilities. In addition, the data content indexing method provided by the embodiments of the present invention can be implemented by software, hardware or a combination of software and hardware.

[0075] like Figure 1As shown, a data content indexing method provided by an embodiment of the present invention may include the following steps:

[0076] S101, determining a plurality of initial data contents of index words containing index sentences in a text domain based on an inverted index method;

[0077] It should be noted that, before executing S101, an index statement may be obtained first, and the index statement is a statement for indexing input by the user for the data content to be obtained. For example, if the data content is a video, the index statement may be "Shen X's comedy", "Variety shows related to songs", "Jin Gang River", etc., if the data content is news, the index statement may be "US XXX", "XX incident", etc., if the data content is a commodity, the index statement may be "blue electric fan", "round water cup", etc.

[0078] The index participle of the index sentence is obtained by performing a participle processing on the index sentence. For example, the index sentence "Shen X's comedy" in the above example can obtain two index participles, namely "Shen X" and "comedy", after the participle processing.

[0079] It should be noted that each data content has at least one text field, and a text field of the data content is the text corresponding to an attribute field of the data content. The relationship between the data content and the text field can be expressed as: doc[field1,field2], that is, the data content with the content identifier doc contains the text fields field1 and field2.

[0080] For example, for videos, the text field of each video may include titles, aliases, actors, etc., such as video 1: {Charlotte's Troubles, Charlotte, Shen Teng}, which means that the title of video 1 is "Charlotte's Troubles", the alias is "Charlotte", and the actor is "Shen Teng".

[0081] The above inverted index method is a method of indexing by index words in the index statement, that is, for each index word in the index statement, the data content containing the index word in the text domain is searched as the initial data content.

[0082] For example, the data content is video content, and the index sentence "Charlotte's Troubles" has two index participles "Charlotte" and "troubles". By indexing "Charlotte", we get "Charlotte's Troubles" and "Charlotte's Day", and by indexing "troubles", we get "Charlotte's Troubles" and "Growing Troubles". Then "Charlotte's Troubles", "Charlotte's Day" and "Growing Troubles" are the initial content data.

[0083] S102, for each initial data content, using a preset forward index, obtaining each text field of the initial data content, wherein the forward index is an index mapped from a content identifier of the data content to a text field;

[0084] The preset positive index is an index mapped from the content identifier of the data content to the text domain, that is, the text domain of the data content can be found through the content identifier of the data content.

[0085] After the initial data content is determined, the text domain mapped by the content identifier can be searched based on the content identifier of the initial data content by using the forward index.

[0086] Exemplarily, the pre-built forward indexes are doc1[field1,field2] and doc2[field3,field4], that is, the data content with the content identifier doc1 includes the text fields field1 and field2, and the data content with the content identifier doc2 includes the text fields field3 and field4. When doc2 is the content identifier of the initial data content, the text fields field3 and field4 of the data content with the content identifier doc2 can be obtained.

[0087] S103, for each initial data content, determining the matching degree between the initial data content and the index statement based on the matching segmented words in each text field of the initial data content, wherein the matching segmented words of any text field are the segmented words in the text field that belong to the index statement;

[0088] The matching segmentation of any text domain is the segmentation in the text domain that belongs to the index statement. For example, text domain 1 contains segmentation A, B, C, and D, and the index statement contains index segmentation B, D, and E, then B and D are the matching segmentation in text domain 1.

[0089] Generally speaking, the more matching segmentations there are in the text domain with the index statement, the higher the matching degree between the initial data content and the index statement. The higher the weight of the matching segmentation in the text domain and the index statement, the higher the matching degree between the initial data content and the index statement. Therefore, after determining the matching segmentations of each text domain of the initial data content for each initial data content, the matching degree between the initial data content and the index statement can be determined based on the matching segmentations in each text domain of the initial data content.

[0090] The matching degree may indicate the degree of matching between the initial data content and the index statement. The matching degree is highest when the initial data content and the index statement are the same.

[0091] S104, sorting each initial data content according to the matching degree between each initial data content and the index statement, and obtaining the index result corresponding to the index statement.

[0092] After determining the matching degree between each initial data content and the index statement, the initial data content with a higher matching degree may be placed at the front.

[0093] Exemplarily, the initial data content includes data content 1, data content 2, and data content 3, wherein the matching degree between data content 1 and the index statement is matching degree 1, the matching degree between data content 2 and the index statement is matching degree 2, and the matching degree between data content and the index statement is matching degree 3, matching degree 2> matching degree 1> matching degree 3, then the sorting sequence number of data content 1 is 2, the sorting sequence number of data content 2 is 1, and the sorting sequence number of data content 3 is 3, then the index result corresponding to the index statement is as shown in Table 1:

[0094] Table 1

[0095]

[0096]

[0097] In the solution provided by the present embodiment, after determining a plurality of initial data contents, the preset forward index can be used to obtain each text domain of the initial data contents, and then based on the matching word segments in each text domain, the matching degree between the initial data contents and the index statement can be determined. Since the forward index can be used to obtain each text domain of the initial data contents, even if part of the data contents found by the index word segmentation is discarded during the inverted indexing, the relevance of the complete index word segmentation can be determined from each text domain in the initial data contents, thereby improving the accuracy of the matching degree and improving the accuracy of the final index result.

[0098] based on Figure 1 Examples of Figure 2 As shown, in another embodiment of the present invention, the data content indexing method provided in the above S103 may include:

[0099] S1031, for each text domain of the initial data content, based on the matching segmentation words and index statements of the text domain, determining an evaluation value corresponding to the text domain, where the evaluation value is used to represent the similarity between the text domain and the index statement;

[0100] Optionally, in one implementation, the evaluation value includes: one or more of a coverage value representing that the index statement is covered by the text domain, a coverage value representing that the text domain is covered by the index statement, and a target distance value; wherein the target distance value is the distance information in the text domain between two matching words that are adjacent in the index statement and contained in the text domain.

[0101] S1032: Determine the matching degree between the initial data content and the index statement according to the evaluation values ​​corresponding to the respective text fields of the initial data content.

[0102] In one implementation, the evaluation values ​​corresponding to the various text fields of the initial data content may be summed up, and the sum result may be used as the matching degree of the initial data content.

[0103] For example, the initial data content includes three text domains, namely text domain 1, text domain 2 and text domain 3, where the evaluation value of text domain 1 and the index statement is evaluation value 1, the evaluation value of text domain 2 and the index statement is evaluation value 2, and the evaluation value of text domain 3 and the index statement is evaluation value 3. Then the matching degree of the initial data content and the index statement = evaluation value 1 + evaluation value 2 + evaluation value 3.

[0104] Optionally, in another implementation, the evaluation values ​​corresponding to the various text fields of the initial data content may be weightedly summed, and the sum result may be used as the matching degree of the initial data content.

[0105] At this time, for each data content, a weight value is assigned to each text field contained in the data content according to the type and number of characters of each text field. After the evaluation value of each text field is determined, the evaluation values ​​can be weighted and summed according to the preset weight value.

[0106] In the solution provided by the present embodiment, after determining a plurality of initial data contents, the preset forward index can be used to obtain each text domain of the initial data contents, and then based on the matching word segments in each text domain, the matching degree between the initial data contents and the index statement can be determined. Since the forward index can be used to obtain each text domain of the initial data contents, even if part of the data contents found by the index word segmentation is discarded during the inverted indexing, the relevance of the complete index word segmentation can be determined from each text domain in the initial data contents, thereby improving the accuracy of the matching degree and improving the accuracy of the final index result.

[0107] Optionally, when the evaluation value includes: a coverage value representing the index statement being covered by the text domain, based on Figure 2 Examples of Figure 3 As shown, another embodiment of the present invention provides a data content indexing method, wherein the above S1031 may include:

[0108] S103A1: Obtain the index weight of each matching word in the text domain in the index statement, and sum the index weights of each matching word in the text domain to obtain the index weight sum value;

[0109] After the index sentence is segmented, the index weights of the index sentences are obtained while the index sentences are segmented.

[0110] Exemplarily, the text domain contains matching word 1 and matching word 2, wherein the index weight of matching word 1 in the index statement is 0.2, and the index weight of matching word 2 in the index statement is 0.4, then the sum of the index weights = 0.2 + 0.4 = 0.6.

[0111] S103A2: Calculate the ratio of the number of characters in the index statement to the number of characters in the text field as the character ratio;

[0112] Exemplarily, the index statement contains 4 characters and the text field contains 5 characters, then the character ratio = 4 / 5.

[0113] S103A3: Sum the character ratio and the index weight and value to obtain an evaluation value corresponding to the text domain.

[0114] Among them, the character ratio and the index weight and value can be directly added together, and the added result is used as the evaluation value corresponding to the text field.

[0115] Optionally, in one implementation, the character ratio and the index weight and value may be weighted and summed to obtain an evaluation value corresponding to the text domain.

[0116] Optionally, in one implementation, the evaluation value corresponding to the text field may also be calculated according to the following formula:

[0117] unigram_cqr=0.8*sum[hit(i)*q_imp(i)]+0.2*LenPunish

[0118] Among them, unigram_cqr is the evaluation value. When the i-th index word is a matching word, hit(i) = 1. Otherwise, when the i-th index word is not a matching word, hit(i) = 0. q_imp(i) is the index weight of the i-th matching word in the index statement. LenPunish represents the relevance of the index statement to the text domain, and the calculation formula is:

[0119] LenPunish=pow(len_query / len_field,0.5)

[0120] Among them, pow is an open function, len_query is the number of characters in the index statement, len_field is the number of characters in the text field, and len_query / len_field is the character ratio.

[0121] In the solution provided by the present embodiment, after determining a plurality of initial data contents, the preset forward index can be used to obtain each text domain of the initial data contents, and then based on the matching word segments in each text domain, the matching degree between the initial data contents and the index statement can be determined. Since the forward index can be used to obtain each text domain of the initial data contents, even if part of the data contents found by the index word segmentation is discarded during the inverted indexing, the relevance of the complete index word segmentation can be determined from each text domain in the initial data contents, thereby improving the accuracy of the matching degree and improving the accuracy of the final index result.

[0122] Optionally, when the evaluation value includes: a coverage value representing the coverage of the text domain by the index statement, based on Figure 2 Examples of Figure 4 As shown, another embodiment of the present invention provides a data content indexing method, wherein the above S1031 may include:

[0123] S103B1, obtaining the text weight of the matching word of the text domain in the text domain, and summing the text weights of each matching word in the text domain to obtain a text weight sum value;

[0124] The text weight of the word segmentation in the text domain may be obtained when the text domain is segmented.

[0125] Exemplarily, the text domain contains matching word 1 and matching word 2, wherein the text weight of matching word 1 in the text domain is 0.5, and the text weight of matching word 2 in the text domain is 0.1, then the text weight sum value = 0.5 + 0.1 = 0.6.

[0126] S103B2, calculating the ratio of the number of characters in the index statement to the number of characters in the text field as the character ratio;

[0127] This step is the same or similar to S103A2 and will not be repeated here.

[0128] S103B3, summing up the character ratio and the text weight and value to obtain an evaluation value corresponding to the text domain.

[0129] Optionally, in one implementation, the character ratio and the text weight and value may be weighted and summed to obtain an evaluation value corresponding to the text domain.

[0130] Optionally, in one implementation, the evaluation value corresponding to the text field may also be calculated according to the following formula:

[0131] unigram_cfr=0.8*sum[hit(i)*f_imp(i))]+0.2*LenPunish

[0132] Among them, unigram_cfr is the evaluation value. When the i-th index word is a matching word, hit(i) = 1. Otherwise, when the i-th index word is not a matching word, hit(i) = 0. f_imp(i)) is the index weight of the i-th matching word in the text domain. LenPunish represents the relevance of the index statement to the text domain, and the calculation formula is:

[0133] LenPunish=pow(len_query / len_field,0.5)

[0134] Among them, pow is an open function, len_query is the number of characters in the index statement, len_field is the number of characters in the text field, and len_query / len_field is the character ratio.

[0135] In the solution provided by the present embodiment, after determining a plurality of initial data contents, the preset forward index can be used to obtain each text domain of the initial data contents, and then based on the matching word segments in each text domain, the matching degree between the initial data contents and the index statement can be determined. Since the forward index can be used to obtain each text domain of the initial data contents, even if part of the data contents found by the index word segmentation is discarded during the inverted indexing, the relevance of the complete index word segmentation can be determined from each text domain in the initial data contents, thereby improving the accuracy of the matching degree and improving the accuracy of the final index result.

[0136] Optionally, when the evaluation value includes a target distance value, based on Figure 2 Examples of Figure 5 As shown, another embodiment of the present invention provides a data content indexing method, wherein the above S1031 may include:

[0137] S103C1, determining two matching segmented words that are adjacent in the index statement and included in the text domain;

[0138] Exemplarily, the text domain contains the index participles "Charlotte" and "trouble", and the index sentence is "Charlotte trouble", which contains the index participles "Charlotte" and "trouble". Then "Charlotte" and "trouble" are two matching participles that are adjacent in the index sentence and contained in the text domain.

[0139] S103C2, obtaining the proximity probability between two matching participles, where the proximity probability indicates the proximity probability of two matching participles being adjacent when they appear in the same sentence;

[0140] The neighbor probability represents the neighbor probability of two matching words appearing in the same sentence, which can be obtained in advance based on the existing technology.

[0141] Exemplarily, the proximity probability of the two matching words "Charlotte" and "trouble" is 0.1, which means that when "Charlotte" and "trouble" appear in a sentence at the same time, the probability that "Charlotte" and "trouble" are adjacent is 10%.

[0142] S103C3, calculating the sum of the text weights of the two matching segmented words in the text domain, and determining the character distance between the two matching segmented words in the text domain and the total number of characters in the text domain;

[0143] Among them, the sum of the text weights is the same or similar to the above content and will not be repeated here.

[0144] For example, the text field is “Charlotte’s troubled day”, where the character distance between “Charlotte” and “troubled” is 1. The total number of characters in the text field is 9.

[0145] S103C4, based on the adjacent probability, the sum of the text weights, the character distance and the total number of characters, determine the target distance information of the two matching word segments in the text domain.

[0146] Optionally, in one implementation, the target distance information between two matching segmented words in the text domain may be determined according to the following formula:

[0147]

[0148] Where term_proximity[i][i+1] is the proximity probability between the i-th index term and the i+1-th index term in the index statement, w[i][i+1] is the sum of the text weights of the i-th index term and the i+1-th index term, dist is the character distance between the i-th index term and the i+1-th index term in the text field, and len_field is the total number of characters in the text field.

[0149] In the solution provided by the present embodiment, after determining a plurality of initial data contents, the preset forward index can be used to obtain each text domain of the initial data contents, and then based on the matching word segments in each text domain, the matching degree between the initial data contents and the index statement can be determined. Since the forward index can be used to obtain each text domain of the initial data contents, even if part of the data contents found by the index word segmentation is discarded during the inverted indexing, the relevance of the complete index word segmentation can be determined from each text domain in the initial data contents, thereby improving the accuracy of the matching degree and improving the accuracy of the final index result.

[0150] Optionally, in one embodiment, the above S1032 may include:

[0151] The evaluation values ​​corresponding to the text fields of the initial data content are input into a pre-trained neural network model for determining the degree of matching, and the result output by the neural network model is used as the degree of matching between the initial data content and the index statement; wherein the neural network model is a model trained using sample evaluation values ​​and the annotation results of the sample evaluation values.

[0152] Among them, the neural network model can be a classification model, such as a logistic regression model, GBDT, DNN and other classification models.

[0153] In the solution provided by the present embodiment, after determining a plurality of initial data contents, the preset forward index can be used to obtain each text domain of the initial data contents, and then based on the matching word segments in each text domain, the matching degree between the initial data contents and the index statement can be determined. Since the forward index can be used to obtain each text domain of the initial data contents, even if part of the data contents found by the index word segmentation is discarded during the inverted indexing, the relevance of the complete index word segmentation can be determined from each text domain in the initial data contents, thereby improving the accuracy of the matching degree and improving the accuracy of the final index result.

[0154] Corresponding to the above method, Figure 6 As shown, an embodiment of the present invention further provides a data content indexing device, the device comprising:

[0155] The data content determination module 601 is used to determine a plurality of initial data contents of index words containing index sentences in the text domain based on the inverted index method;

[0156] The text domain acquisition module 602 is used to acquire each text domain of each initial data content by using a preset positive index, wherein the positive index is an index mapped from a content identifier of the data content to a text domain;

[0157] A matching degree determination module 603 is used to determine the matching degree between each initial data content and the index statement based on the matching segmented words in each text field of the initial data content, wherein the matching segmented words in any text field are the segmented words in the text field that belong to the index statement;

[0158] The index result determination module 604 is used to sort the initial data contents according to the matching degree between the initial data contents and the index statement, and obtain the index result corresponding to the index statement.

[0159] In one embodiment, the matching degree determination module is specifically used to determine, for each text domain of the initial data content, an evaluation value corresponding to the text domain based on the matching word segmentation and index statement of the text domain, wherein the evaluation value is used to characterize the degree of similarity between the text domain and the index statement; and determine the matching degree of the initial data content and the index statement based on the evaluation values ​​corresponding to each text domain of the initial data content.

[0160] In one embodiment, the evaluation value includes:

[0161] One or more of a coverage value representing that the index statement is covered by the text domain, a coverage value representing that the text domain is covered by the index statement, and a target distance value;

[0162] The target distance value is the distance information between two matching words that are adjacent in the index statement and included in the text domain in the text domain.

[0163] In one embodiment, the evaluation value includes: a coverage value representing the coverage of the index statement by the text domain;

[0164] The matching degree determination module is specifically used to obtain the index weight of each matching word in the text domain in the index statement, and sum the index weights of each matching word in the text domain to obtain the index weight sum value; calculate the ratio of the number of characters in the index statement and the text domain as the character ratio; sum the character ratio and the index weight sum value to obtain the evaluation value corresponding to the text domain.

[0165] In one embodiment, the evaluation value includes: a coverage value representing the coverage of the text domain by the index statement;

[0166] The matching degree determination module is specifically used to obtain the text weight of the matching word of the text domain in the text domain, and sum the text weights of each matching word in the text domain to obtain the text weight sum value; calculate the ratio of the number of characters in the index statement and the text domain as the character ratio; sum the character ratio and the text weight sum value to obtain the evaluation value corresponding to the text domain.

[0167] In one embodiment, the evaluation value includes a target distance value;

[0168] The matching degree determination module is specifically used to determine two matching segmentations that are adjacent in the index sentence and included in the text domain; obtain the proximity probability between the two matching segmentations, the proximity probability indicating the proximity probability of the two matching segmentations being adjacent when they appear in the same sentence; calculate the sum of the text weights of the two matching segmentations in the text domain, and determine the character distance between the two matching segmentations in the text domain and the total number of characters in the text domain; determine the target distance information of the two matching segmentations in the text domain based on the proximity probability, the sum of the text weights, the character distance and the total number of characters.

[0169] In one embodiment, the matching degree determination module is specifically used to determine the target distance information of two matching segmented words in the text domain according to the following formula:

[0170]

[0171] Where term_proximity[i][i+1] is the proximity probability between the i-th index term and the i+1-th index term in the index statement, w[i][+1] is the sum of the text weights of the i-th index term and the i+1-th index term, dist is the character distance between the i-th index term and the i+1-th index term in the text field, and len_field is the total number of characters in the text field.

[0172] In one embodiment, the matching degree determination module is specifically used to input the evaluation values ​​corresponding to each text field of the initial data content into a pre-trained neural network model for determining the matching degree, and use the result output by the neural network model as the matching degree between the initial data content and the index statement; wherein the neural network model is a model trained by sample evaluation values ​​and the annotation results of the sample evaluation values.

[0173] In the solution provided by the present embodiment, after determining a plurality of initial data contents, the preset forward index can be used to obtain each text domain of the initial data contents, and then based on the matching word segments in each text domain, the matching degree between the initial data contents and the index statement can be determined. Since the forward index can be used to obtain each text domain of the initial data contents, even if part of the data contents found by the index word segmentation is discarded during the inverted indexing, the relevance of the complete index word segmentation can be determined from each text domain in the initial data contents, thereby improving the accuracy of the matching degree and improving the accuracy of the final index result.

[0174] The embodiment of the present invention further provides an electronic device, such as Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0175] Memory 703, used for storing computer programs;

[0176] The processor 701 is used to implement the above method steps when executing the program stored in the memory 703.

[0177] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0178] The communication interface is used for communication between the above electronic device and other devices.

[0179] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0180] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0181] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above data content indexing methods are implemented.

[0182] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes any one of the data content indexing methods in the above embodiments.

[0183] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0184] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0185] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0186] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A data content indexing method, characterized in that: The method comprises: Based on the inverted index method, determine multiple initial data contents of index words containing index sentences in the text domain; For each initial data content, using a preset forward index, each text field of the initial data content is obtained, wherein the forward index is an index mapped from a content identifier of the data content to a text field; a text field of the data content is a text corresponding to an attribute field of the data content; For each initial data content, based on the matching segmented words in each text field of the initial data content, determining the matching degree between the initial data content and the index statement, wherein the matching segmented words of any text field are the segmented words in the text field that belong to the index statement; Sort each initial data content according to the matching degree between each initial data content and the index statement, and obtain the index result corresponding to the index statement; The determining the matching degree between the initial data content and the index statement based on the matching segmentation of each text field of the initial data content includes: For each text domain of the initial data content, based on the matching segmentation of the text domain and the index statement, determining an evaluation value corresponding to the text domain, wherein the evaluation value is used to represent the degree of similarity between the text domain and the index statement; Determining the matching degree between the initial data content and the index statement according to the evaluation values ​​corresponding to the various text fields of the initial data content; The evaluation values ​​include: a coverage value representing the index statement being covered by the text domain, a coverage value representing the text domain being covered by the index statement, and a target distance value; The target distance value is the distance information between two matching words in the text domain that are adjacent to each other in the index statement and included in the text domain; When the evaluation value includes a target distance value; The step of determining the evaluation value corresponding to the text domain based on the matching segmentation words of the text domain and the index statement includes: Determine two matching participles that are adjacent to the index sentence and included in the text domain; Obtaining a proximity probability between the two matching participles, wherein the proximity probability indicates a proximity probability of the two matching participles being adjacent when they appear in the same sentence; Calculating the sum of the text weights of the two matching segmentations in the text domain, and determining the character distance between the two matching segmentations in the text domain and the total number of characters in the text domain; Determine target distance information of the two matching segmented words in the text domain based on the adjacent probability, the sum of the text weights, the character distance and the total number of characters; Wherein, determining the target distance information of the two matching segmented words in the text domain based on the adjacent probability, the sum of the text weights, the character distance and the total number of characters includes: According to the following formula, the target distance information of the two matching segmented words in the text domain is determined: Wherein, term_proximity[i][i+1] is the proximity probability between the ith index word and the i+1th index word in the index statement, w[i][i+1] is the sum of the text weights of the ith index word and the i+1th index word, dist is the character distance between the ith index word and the i+1th index word in the text field, and len_field is the total number of characters in the text field; When the evaluation value includes: a coverage value representing the coverage of the index statement by the text domain; The step of determining the evaluation value corresponding to the text domain based on the matching segmentation words of the text domain and the index statement includes: Obtaining the index weight of each matching word in the text domain in the index statement, and summing the index weights of each matching word in the text domain to obtain the index weight sum value; Calculating a ratio of the number of characters in the index statement to the number of characters in the text field as a character ratio; Summing the character ratio and the index weight sum value to obtain an evaluation value corresponding to the text domain; When the evaluation value includes: a coverage value representing that the text domain is covered by the index statement; The step of determining the evaluation value corresponding to the text domain based on the matching segmentation words of the text domain and the index statement includes: Obtaining the text weight of the matching word of the text domain in the text domain, and summing the text weights of each matching word in the text domain to obtain a text weight sum value; Calculating a ratio of the number of characters in the index statement to the number of characters in the text field as a character ratio; The character ratio and the text weight sum are summed to obtain an evaluation value corresponding to the text field.

2. The method according to claim 1, characterized in that Determining the matching degree between the initial data content and the index statement according to the evaluation values ​​corresponding to the respective text fields of the initial data content includes: The evaluation values ​​corresponding to the text fields of the initial data content are input into a pre-trained neural network model for determining the degree of matching, and the result output by the neural network model is used as the degree of matching between the initial data content and the index statement; wherein the neural network model is a model trained using sample evaluation values ​​and the annotation results of the sample evaluation values.

3. A data content indexing device, characterized in that: The device comprises: A data content determination module, used to determine a plurality of initial data contents of index words containing index sentences in a text domain based on an inverted index method; A text domain acquisition module is used to acquire, for each initial data content, each text domain of the initial data content using a preset positive index, wherein the positive index is an index mapped from a content identifier of the data content to a text domain; a text domain of the data content is a text corresponding to an attribute field of the data content; A matching degree determination module is used to determine the matching degree between each initial data content and the index statement based on the matching segmented words in each text field of the initial data content, wherein the matching segmented words of any text field are the segmented words in the text field that belong to the index statement; An index result determination module, used to sort each initial data content according to the matching degree between each initial data content and the index statement, and obtain the index result corresponding to the index statement; The matching degree determination module is specifically used to determine, for each text domain of the initial data content, an evaluation value corresponding to the text domain based on the matching segmentation of the text domain and the index statement, wherein the evaluation value is used to represent the similarity between the text domain and the index statement; and determine the matching degree of the initial data content and the index statement according to the evaluation values ​​corresponding to each text domain of the initial data content; The evaluation value includes: a coverage value representing the coverage of the index statement by the text domain, a coverage value representing the coverage of the text domain by the index statement, and a target distance value; wherein the target distance value is the distance information in the text domain between two matching segmentations that are adjacent in the index statement and included in the text domain; When the evaluation value includes a target distance value, the matching degree determination module is specifically used to determine two matching segmentations that are adjacent to the index sentence and included in the text domain; obtain a proximity probability between the two matching segmentations, wherein the proximity probability represents a proximity probability that the two matching segmentations are adjacent when they appear in the same sentence; calculate the sum of the text weights of the two matching segmentations in the text domain, and determine the character distance between the two matching segmentations in the text domain and the total number of characters in the text domain; determine the target distance information of the two matching segmentations in the text domain based on the proximity probability, the sum of the text weights, the character distance, and the total number of characters; wherein the determining the target distance information of the two matching segmentations in the text domain based on the proximity probability, the sum of the text weights, the character distance, and the total number of characters includes: According to the following formula, the target distance information of the two matching segmented words in the text domain is determined: Wherein, term_proximity[i][i+1] is the proximity probability between the ith index word and the i+1th index word in the index statement, w[i][i+1] is the sum of the text weights of the ith index word and the i+1th index word, dist is the character distance between the ith index word and the i+1th index word in the text field, and len_field is the total number of characters in the text field; When the evaluation value includes: a coverage value representing the coverage of the index statement by the text domain, the matching degree determination module is specifically used to obtain the index weight of each matching word in the text domain in the index statement, and sum the index weights of the matching words in the text domain to obtain the index weight sum value; calculate the ratio of the number of characters in the index statement and the text domain as the character ratio; sum the character ratio and the index weight sum value to obtain the evaluation value corresponding to the text domain; When the evaluation value includes: a coverage value representing the degree of coverage of the text domain by the index statement, the matching degree determination module is specifically used to obtain the text weight of the matching word of the text domain in the text domain, and sum the text weights of each matching word in the text domain to obtain a text weight sum value; calculate the ratio of the number of characters of the index statement and the text domain as the character ratio; sum the character ratio and the text weight sum value to obtain the evaluation value corresponding to the text domain.

4. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1-2 when executing a program stored in a memory.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1-2 are implemented.

Images