A method, device, equipment and medium for determining text similarity

Abstract

The disclosed embodiments relate to a method, device, equipment and medium for determining text similarity, wherein the method comprises: obtaining two target texts, the target texts being text information of multimedia to be processed; determining the structured tags of each target text; and determining the similarity between the two target texts according to the structured tags of each target text. The above technical solution is adopted to add structured tags to unstructured texts, and then the similarity of the two unstructured texts can be determined according to the structured tags, and the similarity is determined by combining structured features of different granularities, thereby improving the confidence of the results, and thus facilitating subsequent data analysis and avoiding the problem of unconfident analysis.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a method, device, equipment and medium for determining text similarity. Background Art

[0002] The titles in the videos are unstructured and high-noise texts. To find similar titles from a large number of titles, we usually determine whether two texts are similar and find similar matching titles.

[0003] In the related art, the basic calculation method for determining whether two texts are similar is to use each text as the granularity, and to determine by calculating various distances between texts (such as edit distance, cosine distance, etc.) or by using a classification model. However, this method is limited to the granularity of a single text, and the confidence of the result is low, which leads to the problem of unreliable subsequent analysis. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure provides a method, device, equipment and medium for determining text similarity.

[0005] The present disclosure provides a method for determining text similarity, the method comprising:

[0006] Obtain two target texts, wherein the target texts are text information of multimedia to be processed;

[0007] Determining a structured tag for each of the target texts;

[0008] The similarity between two target texts is determined according to the structured tag of each target text.

[0009] The present disclosure also provides a device for determining text similarity, the device comprising:

[0010] An acquisition module, used for acquiring two target texts, wherein the target texts are text information of multimedia to be processed;

[0011] A label module, used for determining a structured label of each target text;

[0012] The similarity module is used to determine the similarity between two target texts according to the structured tag of each target text.

[0013] An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the text similarity determination method provided in the embodiment of the present disclosure.

[0014] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the text similarity determination method provided by the embodiment of the present disclosure.

[0015] Compared with the prior art, the technical solution provided by the disclosed embodiment has the following advantages: the text similarity determination solution provided by the disclosed embodiment obtains two target texts, where the target texts are text information of the multimedia to be processed; determines the structured label of each target text; and determines the similarity between the two target texts based on the structured label of each target text. By adopting the above technical solution, by adding structured labels to unstructured texts, the similarity of two unstructured texts can be determined based on the structured labels, and the similarity is determined by combining structured features of different granularities, thereby improving the confidence of the results, which is beneficial to subsequent data analysis and avoids the problem of unconfident analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0017] Figure 1 A flowchart of a method for determining text similarity provided by an embodiment of the present disclosure;

[0018] Figure 2 A flowchart of another method for determining text similarity provided by an embodiment of the present disclosure;

[0019] Figure 3 A schematic diagram of determining text similarity provided by an embodiment of the present disclosure;

[0020] Figure 4 A schematic diagram of the structure of a device for determining text similarity provided by an embodiment of the present disclosure;

[0021] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0023] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0024] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0025] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0026] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0027] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0028] In related technologies, the basic calculation method for determining whether two texts are similar is to use each text as the granularity, and to determine by calculating various distances between texts (such as edit distance, cosine distance, etc.) or by using a classification model. This method is widely used in video title similarity matching, which can incubate many product strategies, and then conduct subsequent data analysis on the effect of the algorithm to find the direction of algorithm iteration optimization.

[0029] However, the above text similarity calculation method is limited to the granularity of a single text, and the confidence of the result is low. In the subsequent analysis, it can only be analyzed from the perspective of a single text (case), and then the problem is manually summarized. The analysis of a single granularity will have sparse distribution, limited recall, random problems, and excessive magnitude, which may lead to unconfidence problems in the later analysis. In order to solve the above problems, the embodiments of the present disclosure provide a method for determining text similarity, which is introduced in conjunction with specific embodiments below.

[0030] Figure 1The present invention provides a flowchart of a method for determining text similarity, which can be performed by a text similarity determination device, wherein the device can be implemented by software and / or hardware, and can generally be integrated in an electronic device. Figure 1 As shown, the method includes:

[0031] Step 101: Obtain two target texts, where the target texts are text information of multimedia to be processed.

[0032] The multimedia to be processed may be any multimedia data that needs to be paired. The embodiments of the present disclosure do not limit the specific format of the multimedia to be processed. The multimedia to be processed may include video, audio, and text. The target text may be the text corpus information of the multimedia to be processed. For example, when the multimedia to be processed is a video, the target text may include at least one of the title information of the video, the text content recognized from the image frames in the video, and the high-frequency search terms (queries) searched for the video; when the multimedia to be processed is text, the target text may be the text and the title information of the text. The embodiments of the present disclosure take the example of a game video that needs to be paired as the multimedia to be processed.

[0033] Specifically, the text similarity determination device can obtain target texts of two multimedia to be processed, and the specific source is not limited. For example, when the video to be processed is a game video, the target texts of two game videos whose number of daily playback times in two applications is greater than a preset number can be selected respectively. The preset number can be set according to actual conditions, for example, the preset number can be 10,000.

[0034] Step 102: Determine the structured tag of each target text.

[0035] Among them, the structured tag can be understood as a tag with hierarchical division, which can be similar to a tree-like hierarchical tag and can be represented by a unified structure. In the embodiment of the present disclosure, the structured tag can include a topic tag and a content tag. The topic tag can be understood as a tag obtained by topic clustering, and the content tag can be a pre-summarized structured tag related to the content of the multimedia to be processed.

[0036] In an embodiment of the present disclosure, determining the structured tag of each target text may include: determining the topic tag in the structured tag of the target text based on a pre-built topic clustering model; determining the content tag in the structured tag of the target text based on the target text, a preset name set, and a preset type set.

[0037] The topic clustering model can be understood as a pre-trained deep learning model or algorithm for topic clustering. The embodiment of the present disclosure is not limited to this model. For example, the topic clustering model can be a biterm topic model (Biterm Topic Model, BTM) or a latent Dirichlet allocation model (Latent Dirichlet Allocation, LDA).

[0038] Specifically, for the topic label in the structured label of each target text, each target text can be input into the topic clustering model to obtain the topic label. The topic label can include a topic identifier, a topic probability score, and a topic meaning word. The topic identifier can be a topic ID, which can be represented by numbers, letters, etc. The topic probability score can represent the probability of clustering under the current topic, and the topic meaning word can be understood as a term that expresses the meaning of the topic.

[0039] The content tag in the structured tag of each target text can be determined according to the target text, the preset name set and the preset type set. The preset name set and the preset type set can be name sets that are pre-set according to the attributes of the multimedia to be processed, and are set according to the actual situation. For example, when the multimedia to be processed is a game video, the preset name set can be a game name set, and the preset type set can be a video type set, and the video type can include processes, explanations, strategies, etc.; when the multimedia to be processed is a constellation video, the preset name set can be a constellation name set, and the preset type set can include a constellation fortune, constellation emotion, constellation personality, etc.

[0040] The content tag may include a name, a keyword tag, and a type. Preferably, it may be a three-level structured tag composed of the above three. In the above-mentioned subject clustering model training process, the clustering dimensions extracted according to multiple subject tags can be summarized to obtain the content tag. When the multimedia to be processed is different, the above-mentioned content tags may be different. For example, when the multimedia to be processed is a game video, the content tag may be a three-level structured tag defined for the game category, including the game name, keyword tag, and video type.

[0041] Optionally, determining the content tag in the structured tag of the target text based on the target text, a preset name set and a preset type set may include: extracting keywords of the target text to obtain a first keyword set; determining multiple subject meaning words corresponding to the subject tag of the target text, and combining the multiple subject meaning words to obtain a second keyword set; determining the content tag of the target text based on the first keyword set, the second keyword set, the preset name set and the preset type set.

[0042] When determining the content label of each target text, the text similarity determination device can perform word segmentation and keyword extraction on the target text to obtain multiple keywords of the multimedia to be processed that can express the target text, and combine them to obtain a first keyword set; combine multiple subject meaning words in the subject label of the target text to obtain a second keyword set; and then determine the content label of each target text based on the above-mentioned first keyword set, second keyword set, preset name set and preset type set.

[0043] Optionally, determining the content tag of the target text based on the first keyword set, the second keyword set, the preset name set and the preset type set may include: matching the preset name set in the second keyword set to obtain the name in the content tag of the target text; determining the same words in the first keyword set and the second keyword set as keyword tags in the content tag of the target text; matching the preset type set in the first keyword set to obtain the type in the content tag of the target text.

[0044] Since the content tag may include a name, a keyword tag and a type, for the name, a preset name set may be matched with multiple subject meaning words in the second keyword set. If a name in the preset name set successfully matches multiple subject meaning words, then the name is the name in the content tag; for the keyword tag, the same words in the first keyword set and the second keyword set may be extracted as the keyword tag of the target text; for the type, a preset type set may be matched with multiple keywords in the first keyword set. If a type in the preset type set successfully matches multiple keywords, then the type is the type of the target text.

[0045] In summary, for each target text, its structured tag can be determined, which may specifically include a topic tag plus a content tag.

[0046] Since the target text is unstructured text and the data structure is irregular or incomplete, it is not conducive to determining the similarity. Therefore, in the embodiment of the present disclosure, a structured tag system can be defined and structured tags can be added to each target text, which is conducive to the subsequent similarity determination. The subject tags in the structured tags can be automatically discovered by clustering, and the content tags can be summarized in the aggregated data. The advantage of this is that it can fully reflect the distribution of the data and is more in line with the actual situation.

[0047] Step 103: Determine the similarity between two target texts according to the structured tag of each target text.

[0048] The similarity in the embodiments of the present disclosure may be understood as a result of determining whether the text is similar based on structural features of the text.

[0049] In an embodiment of the present disclosure, determining the similarity between two target texts based on the structured tags of each target text may include: judging whether the subject identifiers in the subject tags and the names in the content tags of the two target texts are the same; when it is determined that the subject tags and the names of the two target texts are the same, determining the similarity scores between the two target texts, and determining whether the two target texts are similar based on the similarity scores.

[0050] If any one of the subject tags and names of the two target texts is different, it can be determined that the two target texts are not similar; when the subject tags and names of the two target texts are the same, the similarity score between the two target texts can be further calculated to determine whether the two target texts are similar based on the similarity score.

[0051] Since the type information in the content tag in the structured tag is relatively sparse, whether the types of the two target texts are the same or not has little impact on the structure, so no judgment is required here; when the similarity requirement is strict, a step of judging whether the types of the two target texts are the same can be added. When the types are the same, the similarity score between the two target texts can be determined.

[0052] For example, Figure 2 A flowchart of another text similarity determination method provided by an embodiment of the present disclosure is shown as follows: Figure 2 As shown, in a feasible implementation, determining the similarity score between two target texts may include the following steps:

[0053] Step 201: determine the same keyword tags in the content tags of two target texts as common keyword tags.

[0054] The common keyword tag may be the intersection of two keyword tags corresponding to the two target texts, that is, the same keyword tag.

[0055] Step 202: Determine the common keyword scores of the common keyword tags on the two target texts respectively, and determine the number of the common keyword tags.

[0056] The common keyword score can be understood as the degree of matching of the common keyword tag on the target text. Since the target text may include the title information of the multimedia to be processed, and the title information usually includes the most critical information, the common keyword score can preferably represent the degree of matching of the common keyword tag on the title information of the target text.

[0057] In an optional implementation, determining the common keyword scores of the common keyword tags on two target texts may include: combining multiple keywords in the common keyword tags in pairs to obtain multiple keyword groups; for each keyword group, determining its keyword score on each target text; for each target text, determining the sum of the keyword scores of the multiple keyword groups in the target text as the common keyword score of the target text.

[0058] After determining the common keyword tags of two target texts, the text similarity determination device can combine multiple keywords in the common keyword tags in pairs, and combine every two keywords into a keyword group to obtain multiple keyword groups, and then determine the keyword score of each keyword group on each target text, that is, each keyword group for each target text obtains a keyword score; then, for each target text, the multiple keyword scores of the multiple keyword groups in the target text are summed to obtain the common keyword score of each target text.

[0059] Optionally, for each keyword group, determining its keyword score on each target text may include: determining each target text as a text to be processed; determining its weight based on the inclusion relationship between two keywords in each keyword group and the text to be processed; determining the probability score of the two keywords in each keyword group as the subject meaning words of the text to be processed; for each keyword group, determining the keyword score of the keyword group on the text to be processed based on the weight of the keyword group and the probability score of the two keywords in the keyword group as the subject meaning words of the text to be processed.

[0060] Among them, the text to be processed can be understood as the target text currently being processed. The text similarity determination device can take each target text as the text to be processed, and then determine the weight of the keyword group according to the inclusion relationship between the two keywords in each keyword group and the text to be processed, and according to the subject meaning words in the subject tag of the text to be processed, determine the probability score of the two keywords in each keyword group as the subject meaning words by searching; then, for each keyword group, the weight of the keyword group and the probability scores of the two keywords therein are input into the keyword score formula, and the keyword score of the keyword group on the text to be processed can be obtained.

[0061] The above keyword score formula can be expressed as g=w*0.1 / (A+B), where A represents the keyword score of a keyword group on the text to be processed, w represents the weight, A represents the probability score of a keyword in the keyword group for the text to be processed, and B represents the probability score of another keyword in the keyword group for the text to be processed. Using the sum of the two probability scores as the denominator can improve the score of the niche keyword tag. By matching two-word combinations, the semantic integrity can be guaranteed to the greatest extent.

[0062] Optionally, the weight of each keyword group is determined based on the inclusion relationship between the two keywords in the text to be processed, including: for each keyword group, when two keywords are located in the text to be processed, the weight of the keyword group is determined to be the first weight; when only one keyword is located in the text to be processed, the weight of the keyword group is determined to be the second weight, wherein the text to be processed is title information, and wherein the first weight is greater than the second weight.

[0063] The first weight and the second weight can be set according to actual conditions, for example, the first weight can be set to 1.5, and the second weight can be set to 1. For example, assuming that a keyword group includes keyword A and keyword B, when keyword A and keyword B are both in the text to be processed, the weight of the keyword group is 1.5, and when keyword A or keyword B is in the text to be processed, the weight of the keyword group is 1.

[0064] Step 203: For each target text, the product of the topic probability score of the target text, the common keyword score of the common keyword tag under the target text, and the number of common keyword tags is determined as the single similarity score of the target text.

[0065] The above-mentioned single similarity score can be obtained by the formula S=T*G*N, wherein S represents the single similarity score of a target text, T represents the topic probability score of the target text, G represents the common keyword score of the common keyword tag under the target text, that is, the sum of the keyword scores g of the above-mentioned multiple keyword groups under the target file, and N represents the number of common keyword tags.

[0066] Step 204: Determine the sum of the two single similarity scores of the two target texts as the similarity score.

[0067] After respectively determining the two single similarity scores of the two target texts, the text similarity determination device can calculate the sum of the two single similarity scores to obtain a final similarity score, and can determine whether the two target texts are similar based on the similarity score.

[0068] Optionally, when the similarity score is greater than a preset threshold, the two target texts can be determined to be similar; otherwise, the two target texts can be determined to be dissimilar; or, for a target text, the similarity scores of the target text and multiple other target texts can be determined, and then the target text can be determined to be similar to the other target texts with the largest similarity scores. After determining that the two target texts are similar, the two to-be-processed multimedia corresponding to the two target texts can be bound and paired for subsequent use.

[0069] The text similarity determination scheme provided by the embodiment of the present disclosure obtains two target texts, where the target texts are text information of the multimedia to be processed; determines the structured tags of each target text; and determines the similarity between the two target texts based on the structured tags of each target text. The above technical scheme is adopted to add structured tags to unstructured texts, and then the similarity of two unstructured texts can be determined based on the structured tags. The similarity determination is performed in combination with structured features of different granularities, which improves the confidence of the results and is beneficial to subsequent data analysis, avoiding the problem of unconfident analysis.

[0070] Next, the text similarity determination method provided by the embodiment of the present disclosure is further illustrated by a specific example. Figure 3 A schematic diagram of determining text similarity provided by an embodiment of the present disclosure is shown in FIG. Figure 3 As shown, taking the game videos of two different applications as an example, the target text is the unstructured text composed of the title information of the game video, the text content recognized from the image frames in the game video, and the high-frequency search terms searched for the game video, and a preset game name set and a preset video type set are set in advance. The text similarity determination process may include:

[0071] 1. Define structured tags. This solution can define structured tags including topic tags and content tags. Topic tags can automatically discover topic categories through clustering, and content tags can be obtained by summarizing categories in aggregated data. The advantage of this is that it can fully reflect the distribution of data and is more in line with the actual situation than manually defining categories. The clustering method mainly adopts the topic clustering model (such as BTM or LDA). This solution can input the target text of multiple game videos into the topic clustering model to obtain the aggregation results. Through the analysis of the aggregation results, it can be found that the video aggregation form under the game category is basically aggregated according to the game name. If there are multiple aggregated themes for the same game, then each theme will have some emphasis on content. For example, for game A, some themes are "A+Survival 100 Days+Explanation", and some themes are "A+Blocks+Strategy", etc. Therefore, the content tags in the structured tags defined for the text of game videos in this solution can have three levels: game name->keyword tag->video type (explanation, process, strategy, etc.).

[0072] 2. Add structured tags to the two target texts. Select the first video and the second video to be processed from the video collection, which can be the first video and the second video whose number of daily playbacks is greater than 10,000 in two applications, and extract the two target texts corresponding to the first video and the second video respectively. The topic tags in the structured tags of the two target texts can be determined by the above topic clustering model. The topic tags can include topic identifiers, topic probability scores, and topic meaning words. See Table 1 for details.

[0073] Table 1 Theme tag table

[0074]

[0075] As shown in Table 1, the table shows exemplary topic tags of two target texts. The topic identifiers of these two topic tags are 172 and 142 respectively, and the topic probability scores are 0.012122 and 0.01122 respectively. The number of topic meaning words can be multiple, and the probability score of each topic meaning word can also be obtained.

[0076] For each target text, by segmenting the target text and extracting keywords, a keyword set A that can express the game video content corresponding to the target text is generated; at the same time, the theme meaning words of the target text are combined to obtain a keyword set B that expresses the theme meaning. Then, according to the existing preset game name set, the theme meaning words are matched in the keyword set to obtain the game name of the game video corresponding to each target text; then the words that appear in both keyword set A and keyword set B are extracted as keyword tags in the content tag; finally, according to the existing preset video type set, the video type in the content tag can be determined from keyword set A.

[0077] In summary, for each target text, its structured tag can be determined, which may specifically include a topic tag plus a content tag, as shown in Table 2.

[0078] Table 2 Structured tag table

[0079]

[0080] As shown in Table 2, the table shows two exemplary structured tags, game videos, including Video 1 and Video 2 in the table, and the target text is the video title information of the game video. The structured tag of each target text includes the topic identifier + game name + keyword tag + video type in Table 2. The topic tag in Table 2 only shows the topic identifier, and the topic probability score and topic meaning vocabulary are not shown in the table.

[0081] 3. Determine the similarity score between the two target texts.

[0082] The basic rule for determining the similarity of the target texts of two game videos is to first determine whether the subject identifiers in the subject tags of the two target texts, the names in the content tags, and the types are the same. If the names are the same, it means that the games corresponding to the two target texts are the same game. If the subject identifiers are the same, they have the same predicted topics, indicating that they have similar content emphases. Then match the same video types, such as both are commentary types. It is understandable that since the information on video types is relatively sparse, it does not necessarily need to be matched. When it is determined that the subject identifiers and names of the two target texts are the same, the similarity scores between the two target texts are determined.

[0083] The formula for the similarity score is as follows:

[0084] Single similarity score of the target text of the first video = topic probability score of the target text of the first video * common keyword score of the target text of the first video * number of common keyword tags;

[0085] Single similarity score of the target text of the second video = topic probability score of the target text of the second video * common keyword score of the target text of the second video * number of common keyword tags;

[0086] Similarity score = single similarity score of the target text of the first video + single similarity score of the target text of the second video.

[0087] The topic probability score of the target text of the first video or the topic probability score of the target text of the second video are the topic probability scores in the topic tags of the two target texts, respectively.

[0088] The common keyword tag may be the intersection of the keyword tag of the target text of the first video and the keyword tag of the target text of the first video. The common keyword score of the target text of the first video may mean the matching degree of the common keyword tag on the target text of the first video. The common keyword score of the target text of the second video has a similar meaning. If the video author wants the video to be distributed more, the title information usually contains the most critical information. Therefore, the target text may be the title information at this time.

[0089] Referring to the idea of ​​clustering, two-word matching is used at this time to maximize the integrity of semantics. The common keyword score of the target text of the first video = w*0.1 / (the keyword score of the common keyword label A in the target text of the first video + the keyword score of the common keyword label B in the target text of the first video), where w can represent the weight. If the common keyword label A and the common keyword label B are both in the target text of the first video, then w = 1.5. If the common keyword label A or the common keyword label B is in the target text of the first video, w = 1; the keyword score of the common keyword label in the target text of the first video can be the probability score of the common keyword label as a topic meaning word. The similarity scores of the two target texts of the first video and the second video are finally obtained as shown in Table 3.

[0090] Table 3 Similarity score table

[0091]

[0092] As shown in Table 3, the subject identification of the two target texts of the first video and the second video is 142, the game name is Game B, the common keyword tags include survival and 100 days, the video types are both commentary, the single similarity score of the first video, that is, the single similarity score of the target text of the first video is 0.177, the single similarity score of the second video, that is, the single similarity score of the target text of the second video is 0.007, and the similarity score between the two target texts is 0.187. The similarity scores of the two target texts can represent the similarity between the first video and the second video. After determining that the two target texts are similar according to the similarity scores, the first video and the second video corresponding to the two target texts can be bound and paired for subsequent analysis.

[0093] After the two videos are bound and paired, one video can be used to attract traffic and new users to the other video. For example, for the game business, after the game is launched, the new user index of each video pair needs to be evaluated in the subsequent analysis. For video pairs matched by traditional text similarity methods, the new user index is analyzed from each paired video, which has problems such as sparse distribution, limited recall, random problems, and excessive magnitude. This solution can calculate the new user index of the video pair through the different levels of label granularity in the structured labels of the target texts of the two videos in the video pair, for example, through the granularity of the topic or the topic plus keywords, and it is easy to compare the index gap between the topics and find the optimization direction. At the same time, the aggregated data is relatively confident in the statistical magnitude, that is, the confidence level is high.

[0094] In this solution, the structured information of the unstructured text of the multimedia to be processed can be generated, and the similarity between different unstructured texts can be determined based on the structured information to improve the confidence of the results. In the later analysis, the statistical calculation of indicators of the data can be performed according to the category labels of different levels in the structured labels. The changes in indicators can be intuitively seen from the category, and the optimization point of the problem can be found. It is equivalent to aggregating single data on categories of different granularities, which is convenient for calculation and improves the problems of sparse distribution, limited recall, randomness of problems, and excessive magnitude in subsequent analysis, avoiding the problem of unconfident analysis.

[0095] Figure 4 This is a schematic diagram of the structure of a text similarity determination device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 4 As shown, the device comprises:

[0096] An acquisition module 401 is used to acquire two target texts, where the target texts are text information of multimedia to be processed;

[0097] A tag module 402, used to determine a structured tag for each of the target texts;

[0098] The similarity module 403 is used to determine the similarity between two target texts according to the structured tag of each target text.

[0099] Optionally, the structured tag includes a subject tag and a content tag, and the tag module 402 includes:

[0100] A topic unit, used to determine a topic tag in the structured tag of the target text according to a pre-built topic clustering model;

[0101] The content unit is used to determine the content tag in the structured tag of the target text according to the target text, the preset name set and the preset type set.

[0102] Optionally, the topic tag includes a topic identifier, a topic probability score, and a topic meaning word.

[0103] Optionally, the content unit is used to:

[0104] Extracting keywords from the target text to obtain a first keyword set;

[0105] Determine a plurality of subject meaning words corresponding to the subject tag of the target text, and combine the plurality of subject meaning words to obtain a second keyword set;

[0106] A content tag of the target text is determined according to the first keyword set, the second keyword set, the preset name set, and the preset type set.

[0107] Optionally, the content tag includes a name, a keyword tag and a type, and the content unit is used to:

[0108] Matching the second keyword set according to the preset name set to obtain the name in the content tag of the target text;

[0109] Determine the same words in the first keyword set and the second keyword set as keyword tags in the content tag of the target text;

[0110] Matching is performed in the first keyword set according to the preset type set to obtain the type in the content tag of the target text.

[0111] Optionally, the similarity module 403 includes:

[0112] A judging unit, used for judging whether the subject identifiers in the subject tags and the names in the content tags of the two target texts are the same;

[0113] The score unit is used to determine the similarity score between the two target texts when it is determined that the subject tags and names of the two target texts are the same, and determine whether the two target texts are similar according to the similarity score.

[0114] Optionally, the score unit includes:

[0115] A first subunit is used to determine the same keyword tags in the content tags of the two target texts as common keyword tags;

[0116] A second subunit is used to determine the common keyword scores of the common keyword tags on the two target texts respectively, and determine the number of the common keyword tags;

[0117] A third subunit is used to determine, for each target text, a product of a topic probability score of the target text, a common keyword score of the common keyword tag under the target text, and the number of the common keyword tags as a single similarity score of the target text;

[0118] The fourth subunit is used to determine the sum of the two single similarity scores of the two target texts as the similarity score.

[0119] Optionally, the second subunit is used for:

[0120] Combining multiple keywords in the common keyword tag in pairs to obtain multiple keyword groups;

[0121] For each of the keyword groups, determining its keyword score on each of the target texts;

[0122] For each of the target texts, the sum of the keyword scores of the plurality of keyword groups in the target text is determined as the common keyword score of the target text.

[0123] Optionally, the second subunit is used for:

[0124] Determine each of the target texts as a text to be processed;

[0125] Determine the weight of each keyword group according to the inclusion relationship between the two keywords in the keyword group and the text to be processed;

[0126] Determining the probability scores of two keywords in each of the keyword groups as the subject meaning words of the text to be processed;

[0127] For each of the keyword groups, the keyword score of the keyword group on the text to be processed is determined according to the weight of the keyword group and the probability scores of the two keywords in the keyword group being the subject meaning words of the text to be processed.

[0128] Optionally, the second subunit is used for:

[0129] For each of the keyword groups, when two of the keywords are located in the text to be processed, the weight of the keyword group is determined to be the first weight; when only one keyword is located in the text to be processed, the weight of the keyword group is determined to be the second weight, wherein the first weight is greater than the second weight.

[0130] The text similarity determination device provided in the embodiments of the present disclosure can execute the text similarity determination method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0131] The embodiments of the present disclosure also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the text similarity determination method provided by any embodiment of the present disclosure.

[0132] Figure 5 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 5, which shows a schematic diagram of the structure of an electronic device 500 suitable for implementing the embodiment of the present disclosure. The electronic device 500 in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0133] like Figure 5 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0134] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0135] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the text similarity determination method of the embodiment of the present disclosure are executed.

[0136] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0137] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0138] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0139] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains two target texts, where the target texts are text information of the multimedia to be processed; determines the structured tag of each of the target texts; and determines the similarity between the two target texts based on the structured tag of each of the target texts.

[0140] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0141] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0143] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0144] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0145] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0146] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0147] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for determining text similarity, characterized in that: include: Obtain two target texts, wherein the target texts are text information of multimedia to be processed; Determine a structured tag for each of the target texts; the structured tag includes a topic tag and a content tag; the topic tag includes a topic identifier, a topic probability score, and a topic meaning word; the content tag includes a keyword extracted from the target text, a set obtained by combining a plurality of topic meaning words corresponding to the topic tag of the target text, a preset name set, and a preset type set; The similarity between two target texts is determined according to the structured tag of each target text.

2. The method according to claim 1, characterized in that Determining the structured tag of each target text includes: Determining a topic tag in the structured tag of the target text according to a pre-built topic clustering model; According to the target text, the preset name set and the preset type set, a content tag in the structured tag of the target text is determined.

3. The method according to claim 2, characterized in that Determining a content tag in a structured tag of the target text according to the target text, a preset name set, and a preset type set includes: Extracting keywords from the target text to obtain a first keyword set; Determine a plurality of subject meaning words corresponding to the subject tag of the target text, and combine the plurality of subject meaning words to obtain a second keyword set; A content tag of the target text is determined according to the first keyword set, the second keyword set, the preset name set, and the preset type set.

4. The method according to claim 3, characterized in that The content tag includes a name, a keyword tag and a type. Determining the content tag of the target text according to the first keyword set, the second keyword set, the preset name set and the preset type set includes: Matching the second keyword set according to the preset name set to obtain the name in the content tag of the target text; Determine the same words in the first keyword set and the second keyword set as keyword tags in the content tag of the target text; Matching is performed in the first keyword set according to the preset type set to obtain the type in the content tag of the target text.

5. The method according to claim 1, characterized in that Determining the similarity between two target texts according to the structured tag of each target text includes: Determining whether the subject identifiers in the subject tags and the names in the content tags of the two target texts are the same; When it is determined that the subject tags and names of the two target texts are the same, a similarity score between the two target texts is determined, and whether the two target texts are similar is determined based on the similarity score.

6. The method according to claim 5, characterized in that Determining a similarity score between the two target texts, including: Determine the same keyword tags in the content tags of the two target texts as common keyword tags; Determine the common keyword scores of the common keyword tags on the two target texts respectively, and determine the number of the common keyword tags; For each of the target texts, the product of the topic probability score of the target text, the common keyword score of the common keyword tag under the target text, and the number of the common keyword tags is determined as a single similarity score of the target text; The sum of the two single similarity scores of the two target texts is determined as the similarity score.

7. The method according to claim 6, characterized in that Determining the common keyword scores of the common keyword tags on the two target texts respectively includes: Combining multiple keywords in the common keyword tag in pairs to obtain multiple keyword groups; For each of the keyword groups, determining its keyword score on each of the target texts; For each of the target texts, the sum of the keyword scores of the plurality of keyword groups in the target text is determined as the common keyword score of the target text.

8. The method according to claim 7, characterized in that For each of the keyword groups, determining its keyword score on each of the target texts includes: Determine each of the target texts as a text to be processed; Determine the weight of each keyword group according to the inclusion relationship between the two keywords in the keyword group and the text to be processed; Determining the probability scores of two keywords in each of the keyword groups as the subject meaning words of the text to be processed; For each of the keyword groups, the keyword score of the keyword group on the text to be processed is determined according to the weight of the keyword group and the probability scores of the two keywords in the keyword group being the subject meaning words of the text to be processed.

9. The method according to claim 8, characterized in that The weights of the two keywords in each keyword group are determined according to the inclusion relationship between the two keywords and the text to be processed, including: For each of the keyword groups, when two of the keywords are located in the text to be processed, the weight of the keyword group is determined to be the first weight; when only one keyword is located in the text to be processed, the weight of the keyword group is determined to be the second weight, wherein the first weight is greater than the second weight.

10. A device for determining text similarity, characterized in that: include: An acquisition module, used for acquiring two target texts, wherein the target texts are text information of multimedia to be processed; A tag module, used to determine the structured tag of each target text; the structured tag includes a topic tag and a content tag; the topic tag includes a topic identifier, a topic probability score and a topic meaning word; the content tag includes a keyword extracted from the target text, a set obtained by combining a plurality of topic meaning words corresponding to the topic tag of the target text, a preset name set and a preset type set; The similarity module is used to determine the similarity between two target texts according to the structured tag of each target text.

11. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the text similarity determination method described in any one of claims 1-9 above.

12. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the text similarity determination method described in any one of claims 1 to 9.

Images