Text tag extraction method, device, equipment and storage medium

Abstract

The embodiment of the present invention discloses a method, device, equipment and storage medium for extracting text labels. The method comprises: obtaining each text of the label to be extracted, and vectorizing each of the texts to obtain a text vector corresponding to the corresponding text; clustering each of the text vectors to obtain at least one text clustering result; extracting keywords from each of the text clustering results to obtain each label candidate word corresponding to each of the text clustering results; and determining the text label of each of the texts according to each of the label candidate words corresponding to each of the text clustering results. Through the above technical solution, the automatic extraction of text labels is realized, and the accuracy and comprehensiveness of text label extraction and the scalability of the label extraction method are improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to computer technology, and in particular to a text tag extraction method, device, equipment and storage medium. Background Art

[0002] In application scenarios such as information search and information recommendation, data mining is usually required, one of which is the extraction of text tags. Taking e-commerce platforms as an example, the objects of text tag extraction are usually product-related information, such as product details (referred to as item introduction details), product specifications and comments, etc. Among them, the item introduction details contain more detailed and comprehensive product description information, such as product usage occasions, applicable groups, and marketing labels such as "small footprint" and "strong suction"; product specifications are stored in structured forms such as tables, which include product specifications such as length, width, and height, and product extended attributes such as color, memory, and communication mode.

[0003] At present, there are roughly two methods for extracting text tags for e-commerce platforms: First, obtain text tags from the manual reporting system. For example, when a merchant puts a product on the shelf, it is required to fill in the product-related attribute information in the manual reporting system. Second, automatically extract text tags from structured product-related information such as product specifications and parameters, such as automatically identifying structured text carriers such as tables, and then extracting text tags from them. Third, automatically extract text tags from unstructured product-related information such as item introduction details and comments, such as manually marking the text tags that need to be extracted, and then automatically extracting related text tags.

[0004] In the process of realizing the present invention, the inventors found that there are at least the following problems in the prior art: (1) There are problems such as errors, omissions and inaccurate expressions in the manually reported information, and the manually reported labels are highly standardized and lack personalization; (2) Although the method of automatically extracting labels from structured information can correct manual errors and improve the efficiency of manually reporting specified attributes to a certain extent, it still has the problem of insufficient personalization; (3) Although the method of automatically extracting labels from unstructured information such as item introduction details can solve the problem of insufficient personalization of text labels to a certain extent, it requires manual data annotation, which makes the efficiency of text label extraction low and the scalability poor. Summary of the invention

[0005] The embodiments of the present invention provide a text tag extraction method, apparatus, device and storage medium to realize automatic extraction of text tags and improve the accuracy, comprehensiveness and scalability of text tag extraction.

[0006] In a first aspect, an embodiment of the present invention provides a text tag extraction method, comprising:

[0007] Obtain each text of the label to be extracted, and vectorize each of the texts to obtain a text vector corresponding to the corresponding text;

[0008] Clustering each of the text vectors to obtain at least one text clustering result;

[0009] Perform keyword extraction on each of the text clustering results to obtain each label candidate word corresponding to each of the text clustering results;

[0010] According to each of the label candidate words corresponding to each of the text clustering results, a text label of each of the texts is determined.

[0011] In a second aspect, an embodiment of the present invention further provides a text tag extraction device, the device comprising:

[0012] A text vector acquisition module is used to acquire each text of the tag to be extracted, and vectorize each of the texts to obtain a text vector corresponding to the corresponding text;

[0013] A text clustering result obtaining module, used for clustering each of the text vectors to obtain at least one text clustering result;

[0014] A label candidate word acquisition module is used to extract keywords from each of the text clustering results to obtain each label candidate word corresponding to each of the text clustering results;

[0015] The text label determination module is used to determine the text label of each of the texts according to each of the label candidate words corresponding to each of the text clustering results.

[0016] In a third aspect, an embodiment of the present invention further provides a device, the device comprising:

[0017] one or more processors;

[0018] a storage device for storing one or more programs,

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the text tag extraction method provided by any embodiment of the present invention.

[0020] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the text tag extraction method provided by any embodiment of the present invention.

[0021] The embodiment of the present invention obtains each text of the label to be extracted, and vectorizes each text to obtain the text vector corresponding to the corresponding text; clusters each text vector to obtain at least one text clustering result. It realizes the unsupervised classification of multiple texts, reduces the human intervention in the label extraction process, and improves the accuracy of subsequent label extraction. By extracting keywords from each text clustering result, each label candidate word corresponding to each text clustering result is obtained; according to each label candidate word corresponding to each text clustering result, the text label of each text is determined. It realizes the unsupervised automatic label extraction of various texts under the same topic, solves the problem of incomplete and insufficient personalization of labels caused by only extracting labels from structured texts, and improves the personalization and comprehensiveness of text labels; at the same time, the unsupervised text label extraction solves the problems of low efficiency of label extraction and poor scalability of extraction methods caused by manual data annotation, improves the extraction efficiency of text labels, and improves the applicability of methods for label extraction under different topics, and enhances the scalability of label extraction methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1a is a flowchart of a text tag extraction method in Embodiment 1 of the present invention;

[0023] Figure 1b This is a schematic diagram of extracting text information from an item introduction details map in Embodiment 1 of the present invention;

[0024] Figure 1c is a schematic diagram of word segmentation results of text in the item introduction details diagram in the first embodiment of the present invention;

[0025] Figure 1d Schematic diagram of word vectorization results of the word vector model in the first embodiment of the present invention;

[0026] Figure 1e Schematic diagram of text clustering results in Embodiment 1 of the present invention;

[0027] Figure 1f Schematic diagram of keyword extraction results of text clustering results in Embodiment 1 of the present invention;

[0028] Figure 1g is a schematic diagram of the label clustering result in the first embodiment of the present invention;

[0029] Figure 2a is a flow chart of a text tag extraction method in Embodiment 2 of the present invention;

[0030] Figure 2b is a schematic diagram of text recognition results of an item introduction details image in the second embodiment of the present invention;

[0031] Figure 2cis a schematic diagram of the merged text in the item introduction details diagram in the second embodiment of the present invention;

[0032] Figure 2d It is a schematic diagram of the principle of text merging in the second embodiment of the present invention;

[0033] Figure 2e is a schematic diagram of candidate tags and word frequencies in the second embodiment of the present invention;

[0034] Figure 3 is a flowchart of a text label extraction method in Embodiment 3 of the present invention;

[0035] Figure 4 is a structural schematic diagram of a text label extraction device in Embodiment 4 of the present invention;

[0036] Figure 5 It is a structural schematic diagram of a device in Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0038] Embodiment 1

[0039] The text tag extraction method provided in this embodiment is applicable to the case of extracting tags from multiple texts under the same topic, and is particularly applicable to text tag extraction in e-commerce platforms. The method can be executed by a text tag extraction device, which can be implemented by software and / or hardware, and can be integrated in a device such as a personal computer or a server. Figure 1a The method of this embodiment specifically includes the following steps:

[0040] S110, obtaining each text of the label to be extracted, and vectorizing each text to obtain a text vector corresponding to the corresponding text.

[0041] Among them, a label refers to a description of a certain aspect of a text, such as a keyword that expresses the key points of the text. In e-commerce platforms, a label can be a description of a specific attribute of an item, such as the specification attributes of an item, the extended attributes of an item, and the functional attributes of an item. The functional attributes of an item can be marketing labels that fit the user's concerns, such as energy saving, quietness, small footprint, and strong suction. A text vector refers to a digital vector obtained by text conversion.

[0042] In order to automatically extract text tags, it is necessary to first obtain multiple texts containing text tags. Furthermore, in order to improve the efficiency of tag extraction and the convenience of subsequent tag classification and use, multiple texts under the same topic can be obtained. The topic here refers to the main content of the text, for example, it can be a research topic of an article, or it can be an item category in an e-commerce platform, such as a mobile phone category, a range hood category, and the like. In the embodiment of the present invention, the range hood category is taken as an example for explanation, and multiple texts under the range hood category need to be obtained. Afterwards, each obtained text needs to be digitized in order to automatically extract subsequent text tags. In this embodiment, each text is vectorized and converted into a digitized vector.

[0043] Exemplarily, vectorizing each text to obtain a text vector corresponding to the corresponding text includes: for each text, segmenting the text and removing stop words to obtain the word segmentation results to be extracted corresponding to the text, and using at least one feature vector model to vectorize each word segmentation result to be extracted to obtain the text vector corresponding to the text.

[0044] The word segmentation result to be extracted refers to the word segmentation result of the text used to extract the tag. The feature vector model refers to a model that converts text into a vector, such as a topic model such as the open source LDA (Latent Dirichlet Allocation) model, a term frequency-inverse document frequency model (Term Frequency-Inverse Document Frequency, TF-IDF) or a word vector model such as the word2vec model. The description of these models is shown in the following content.

[0045] The process of text vectorization can be processed one by one. The processing process of a single text is as follows: first, the text is segmented using a segmentation tool such as Jieba word segmentation to obtain multiple words. Then, the multiple words are removed from the stop words, that is, the punctuation, numbers, English letters and brand names under the current category in the multiple words obtained by the segmentation are removed (the brand name can be obtained from the database of the e-commerce platform through the category number to obtain the brand list under the corresponding category), and the remaining words obtained are the segmentation results to be extracted for the text. After that, each word in the segmentation results to be extracted is input into at least one feature vector model among the topic model, the word frequency-inverse file frequency model and the word vector model. After the vectorization processing of the model, the word vector of each word corresponding to the text can be obtained. The corresponding element values ​​of the word vectors of each word obtained by the same feature vector model are added together to obtain the initial text vector of the text under the vectorization dimension corresponding to the feature vector model. If only one feature vector model is used, the initial text vector obtained is the final text vector. If the above three feature vector models are used, then the three initial text vectors corresponding to the text can be obtained. At this point, the three initial text vectors are directly connected to obtain the final text vector corresponding to the text. Assuming that the vector dimensions of the three initial text vectors are 8, 10, and 12 respectively, the vector dimension of the final text vector is 30. The advantage of this setting is that a more accurate text vector can be obtained, thereby improving the accuracy of subsequent text label extraction to a certain extent.

[0046] The above term frequency-inverse document frequency model (TF-IDF) can measure the importance of a word to a text in all texts (called a corpus). Its importance increases in direct proportion to the number of times it appears in the text, but at the same time it decreases in inverse proportion to the frequency of its appearance in the corpus. Its calculation formula is as follows:

[0047]

[0048] Among them, c wd is the number of times a word appears in the text to be processed, c w is the total number of words in the text to be processed, c d is the total number of texts in the corpus, c dw is the number of texts in the corpus that contain the word.

[0049] The above-mentioned word vector model can be an existing model in the related art, or a model obtained by retraining. In this embodiment, the model is retrained using the item introduction details image in the e-commerce platform as the training data source, and the training process is as follows:

[0050] First, we run the text detection and recognition system to obtain the text information in the item description details image as corpus data, such as Figure 1b Then, the expected data is segmented to obtain Figure 1c The word segmentation results shown in the figure are used as the vocabulary for subsequent model training. Finally, the vocabulary is input into the word2vec model to be trained for model training (the parameters are set as follows: -train vocabulary.txt-output word vector.bin-cbow 1-size 200-window 5-negative 25-threads 20-binary 0), and the word vector model can be obtained. Using the trained word vector model, "appointment" and "protection" are vectorized to obtain their corresponding word vectors. The results are shown in the figure below. Figure 1d shown.

[0051] S120: Cluster each text vector to obtain at least one text clustering result.

[0052] By clustering the multiple text vectors obtained above using clustering algorithms such as K-Means or AP, the text vectors can be automatically divided into multiple categories, that is, multiple text clustering results can be obtained. The text expressions in each text clustering result are similar in meaning. According to the above text vectorization description, a text vector is obtained by vectorizing multiple words, so a text vector corresponds to a word segmentation result to be extracted, then the multiple text vectors contained in a text clustering result can be represented by the word segmentation result to be extracted corresponding to each text vector. See Figure 1e The text clustering result numbered 239 includes 14 text segmentation results corresponding to the texts to be extracted, and the text clustering result numbered 139 includes 4 text segmentation results corresponding to the texts to be extracted. It can be seen that the texts in the same text clustering result have similar meanings.

[0053] It should be noted that the same segmentation results to be extracted exist in each text clustering result. This is because different SKUs under this category may use the same text description.

[0054] S130 , performing keyword extraction on each text clustering result to obtain each label candidate word corresponding to each text clustering result.

[0055] By using a keyword extraction algorithm such as that in the Jieba tool, keywords are extracted from each text clustering result, and the top n keywords are retained according to application requirements, the keyword extraction results corresponding to each text clustering result can be obtained. These extracted keywords are used as label candidates for further generating text labels. For example, keyword extraction is performed on the text clustering results numbered 239 and 139, and the following can be obtained: Figure 1f The two sets of label candidates are shown.

[0056] It should be noted that, during the keyword extraction process, if there are overlapping keywords between the text clustering results, keyword deduplication is required.

[0057] S140 , determining a text label for each text according to each label candidate word corresponding to each text clustering result.

[0058] Since the semantics of a single keyword is not clear and distinct enough, in this embodiment, after extracting the keywords, the label candidate words in each text clustering result are merged to obtain a candidate word combination result with more certain semantics as the text label finally extracted from each text.

[0059] According to the above description, the text tag extraction in the embodiment of the present invention is mainly based on unsupervised clustering and word frequency information, so that the text tag extraction method in the embodiment of the present invention is independent of the text subject (such as the category of the item), and can be migrated between different topics at a low cost, so that the text tag extraction method in the embodiment of the present invention is highly scalable and can efficiently and conveniently serve in various scenarios based on item and user-related analysis, such as improving the accuracy of search and recommendation, providing richer item information for product-related data analysis, etc., to achieve "the right product accurately reaches the right user".

[0060] The technical solution of this embodiment obtains each text of the label to be extracted, and vectorizes each text to obtain the text vector corresponding to the corresponding text; clusters each text vector to obtain at least one text clustering result. It realizes the unsupervised classification of multiple texts, reduces the human intervention in the label extraction process, and improves the accuracy of subsequent label extraction. By extracting keywords from each text clustering result, the label candidate words corresponding to each text clustering result are obtained; according to the label candidate words corresponding to each text clustering result, the text label of each text is determined. It realizes the unsupervised automatic label extraction of various texts under the same topic, solves the problem of incomplete and insufficient personalization of labels caused by only extracting labels from structured texts, and improves the personalization and comprehensiveness of text labels; at the same time, the unsupervised text label extraction solves the problems of low efficiency of label extraction and poor scalability of extraction methods caused by manual data annotation, improves the extraction efficiency of text labels, and improves the applicability of methods for label extraction under different topics, and enhances the scalability of label extraction methods.

[0061] On the basis of the above technical scheme, after determining the text label of each text according to the label candidate words corresponding to each text clustering result, it also includes: clustering each text label to obtain at least one label clustering result; for each label clustering result, using any text label in the label clustering result as a revised text label of the corresponding label clustering result, and using each remaining text label in the label clustering result except the revised text label as a similar text label of the revised text label.

[0062] According to the above description, if there are semantically similar label candidate words in a text clustering result, then there may also be semantically similar labels in each text label corresponding to the text clustering result. In order to simplify the labels, the present embodiment performs cluster analysis on the text labels again. During specific implementation, for each text clustering result, multiple text labels corresponding to the text clustering result are clustered to obtain multiple label clustering results corresponding to the text clustering result. Any text label in the label clustering result is then used as the revised text label corresponding to the text clustering result, and the other remaining text labels in the label clustering result are used as similar text labels of the revised text label. In this way, each text clustering result corresponds to a revised text label, and the revised text label corresponds to multiple similar text labels. The revised text labels and similar text labels can be stored in a list for subsequent use.

[0063] For example Figure 1g In the example, "noise reduction and silence, decibel silence, sound pressure level noise, decibel low noise" is a label clustering result. The first text label "noise reduction and silence" is selected as the corrected text label of the label clustering result. The remaining text labels are similar text labels. Then the text label list of the label clustering result can be {"noise reduction and silence": "noise reduction and silence", "decibel silence", "sound pressure level noise", "decibel low noise"}.

[0064] Embodiment 2

[0065] Based on the above-mentioned embodiment 1, this embodiment further optimizes "obtaining each text of the label to be extracted". On this basis, "determining the text label of each text according to each label candidate word corresponding to each text clustering result" can be further optimized. The explanations of the terms that are the same or corresponding to the above-mentioned embodiments are not repeated here.

[0066] For the convenience of subsequent explanation, the application scenario in this embodiment is set to extract text labels from the item introduction details map of the e-commerce platform. In addition to the structured item specification attributes and item extension attributes, the item introduction details map also contains a large amount of unstructured item information descriptions, which contain a large number of personalized labels for items. These personalized labels can be used as corrections and supplements to manually reported item attributes and structured item attributes to improve the description of the item, and can then be used as basic data for applications such as personalized information search and recommendation.

[0067] See also Figure 2a , the text tag extraction method provided in this embodiment includes:

[0068] S210: Perform text recognition on the item introduction details image to obtain text data of the item introduction details image.

[0069] Using the text recognition system to perform text recognition on the item introduction details image, we can obtain the following Figure 2b The text data of the item description details picture shown. In addition to the structured item attributes, these text data also contain a large amount of unstructured functional description information. In addition, in addition to the text itself, the text data also records the position of the text in the picture, the font, size and color of the text, etc.

[0070] S220. Filter the texts in the text data by importance based on text attributes and preset text filtering conditions in the text data, obtain the filtered texts corresponding to the item introduction details image, and determine the texts of the tags to be extracted corresponding to the item introduction details image based on the filtered texts.

[0071] The text attributes refer to information such as the text position, the text font, the font size and the color etc. The preset text screening condition is a pre-set condition for screening texts, for example, it can be a text attribute such as the font size and the color used to characterize the importance of the text.

[0072] In the current item introduction details map, in order to facilitate users to grasp the main performance characteristics of the item, the text describing the important characteristics of the item is displayed in a larger font size or a special color. Based on this, in this embodiment, after obtaining the text data corresponding to the item introduction details map, the text attributes in the text data and the preset text filtering conditions are first used to filter the text. For example, if the preset text filtering condition is set to a font size of 20, then all texts with a font size less than 20 in the text data are removed (text with a larger font size is more relevant to the label, and small font text is generally a description of a personalized label), and the remaining text is the various filtered texts. Afterwards, the various texts of the to-be-extracted labels corresponding to the item introduction details map can be determined based on the filtered text. For example, the filtered text can be directly used as the required text, or the filtered text can be processed by text merging to obtain the required text.

[0073] Exemplarily, determining each text of the label to be extracted corresponding to the item introduction details image based on each filtered text includes: for every two vertically adjacent filtered texts, if any of the absolute value of the left edge difference, the absolute value of the center coordinate difference and the absolute value of the right edge difference between the text boxes corresponding to the two filtered texts in the text data is less than the set text box height, and the absolute value of the vertical difference between the two text boxes is less than the set text box height, then the two filtered texts are merged as the vertical merged text, wherein the set text box height is the larger value of the text box heights of the two text boxes, and the absolute value of the vertical difference is the absolute value of the distance difference between the lower edge of the upper text box and the upper edge of the lower text box; for every two horizontally adjacent vertical merged texts, if the vertical overlapping distance between the text boxes of the two vertically merged texts is less than the set text box height, then the two vertically merged texts are merged as the text of the label to be extracted corresponding to the item introduction details image.

[0074] The text in the item introduction details image is different from ordinary text. The text used to describe the same event may be arranged in an area in the image. Therefore, in this embodiment, the required texts are determined by merging the filtered texts vertically and horizontally. Figure 2c "Intelligent control, you can prepare the ingredients on your way home from get off work", "Ingredient entry management, ingredients can be entered, and there will be reminders when they are expired" and "Online recommendation of health recipes, you can also recommend recipes based on the entered ingredients" all describe the same thing, so these texts need to be merged vertically first, and then merged horizontally.

[0075] In the case of vertical mergers, see Figure 2d For each two vertically adjacent filter texts, obtain the absolute value 201 of the left edge difference between the upper text box 210 and the lower text box 220 (denoted as x l ), the absolute value of the center coordinate difference is 202 (denoted as x m) and the absolute value of the right edge difference 203 (denoted as x r ), if any of the differences is less than the set text box height 204 (denoted as h m ), then continue to determine the absolute value 205 of the vertical difference between the upper text box 210 and the lower text box 220 (recorded as y b ) is less than the set text box height 204, if so, the two filter texts corresponding to the two text boxes are merged vertically to form the following Figure 2c The vertically merged text 200 in the vertical merge. That is, the two vertically merged filter texts satisfy the relationship: x l <h m x m <h m x r <h m And y b <h m .

[0076] After the vertical merging, it is necessary to determine whether each two vertically merged texts need to be horizontally merged. For each two vertically merged texts, the vertical overlap distance 206 (denoted as y ) between the left text box 230 and the right text box 240 is obtained. o ), if the vertical overlap distance 206 is less than the set text box height 204, the two vertically merged texts corresponding to the two text boxes are merged horizontally to form a Figure 2c The text 200′ in the horizontal merged text. That is, the two vertical merged texts satisfy the relationship: y o <h m .

[0077] S230: vectorize each text to obtain a text vector corresponding to the corresponding text.

[0078] S240: Cluster each text vector to obtain at least one text clustering result.

[0079] S250: extract keywords from each text clustering result to obtain candidate label words corresponding to each text clustering result.

[0080] S260, based on the co-occurrence matrix between words obtained in advance based on each text, determine the co-occurrence frequency between each two label candidate words corresponding to each text clustering result, and screen out at least one pair of label candidate words whose co-occurrence frequency meets a preset co-occurrence frequency threshold, and generate each label reorganization word.

[0081] Among them, the co-occurrence matrix is ​​a matrix that counts the frequency of whether two words appear together in a text.

[0082] When merging label candidate words, the co-occurrence frequency between every two label candidate words is used. First, it is necessary to perform statistics on the co-occurrence frequency between every two words in the text according to the word segmentation results of each text obtained in S220 to obtain the co-occurrence matrix corresponding to all words. Afterwards, the co-occurrence frequency between every two label candidate words in the text clustering results is found using the co-occurrence matrix. In this way, the co-occurrence frequency of the combination words composed of every two label candidate words in the text clustering results can be obtained. Filter out the combination words whose co-occurrence frequency is greater than the preset co-occurrence frequency threshold (a co-occurrence frequency that can be set empirically), and these combination words are the label reorganization words corresponding to the text clustering results. According to the above process, the label reorganization words corresponding to each text clustering result can be obtained.

[0083] Exemplarily, before determining the co-occurrence frequency between each two label candidate words corresponding to each text clustering result based on the co-occurrence matrix between words obtained in advance based on each text, it also includes: using a feature vector model to determine the candidate word vector of each label candidate word corresponding to each text clustering result; based on the candidate word vectors of each two label candidate words, determining the first text similarity between the corresponding two label candidate words, and eliminating the label candidate words whose first text similarity is less than a first preset similarity threshold among the label candidate words corresponding to each text clustering result.

[0084] In order to further improve the accuracy of label candidate word merging, in this embodiment, before merging label candidate words using co-occurrence frequency, the text similarity (i.e., the first text similarity) between every two label candidate words is calculated. After that, the label candidate words whose first text similarity is less than the first preset similarity threshold (similarity set by experience) among all label candidate words in the text clustering result are eliminated, and the remaining label candidate words can be merged based on co-occurrence frequency.

[0085] S270: For each label reorganization word, if the label reorganization word exists in each text, determine the label reorganization word as a text label.

[0086] After obtaining the label reorganization word, it is necessary to determine whether the label reorganization word exists in the text obtained in S220, and if so, it is determined as a text label. If not, it cannot be used as a text label.

[0087] Exemplarily, S270 includes: if there is a label reorganization word in each text, determining the label reorganization word as a candidate label; if the word frequency of the candidate label meets a preset word frequency threshold, determining the label reorganization word as a text label.

[0088] In the process of determining the label reorganization word as the text label, on the basis of determining whether the label reorganization word exists in the text, further according to the frequency of each label reorganization word appearing in each text (i.e., word frequency, see Figure 2e The number after each tag reorganization word in the text is used for screening, so that the candidate tags whose word frequency meets the preset word frequency threshold (the word frequency set empirically according to the application requirement accuracy) are determined as text tags. The advantage of this setting is that the accuracy of the text tags is further improved.

[0089] The technical solution of this embodiment is to obtain the text data of the item introduction details picture by performing text recognition on the item introduction details picture; according to the text attributes in the text data and the preset text screening conditions, the text in the text data is screened by importance to obtain the screening texts corresponding to the item introduction details picture, and the texts of the labels to be extracted corresponding to the item introduction details picture are determined according to the screening texts. The text acquisition based on the item introduction details picture is realized, the texts with less relevance to the labels in the obtained texts are reduced, and the efficiency of subsequent text label extraction is further improved. By determining the co-occurrence frequency between each two label candidate words corresponding to each text clustering result according to the co-occurrence matrix between words obtained in advance based on each text, and screening out at least one pair of label candidate words whose co-occurrence frequency meets the preset co-occurrence frequency threshold, each label reorganization word is generated; for each label reorganization word, if there is a label reorganization word in each text, the label reorganization word is determined as a text label. The text label extraction of unstructured information in the item introduction details picture is realized, the information utilization rate of the item introduction details picture is improved, and the comprehensiveness and accuracy of the text label extraction are further improved.

[0090] Embodiment 3

[0091] This embodiment, based on the above embodiments, illustrates the steps of automatically labeling the text to be labeled, wherein the explanations of the terms that are the same or corresponding to the above embodiments are not repeated here.

[0092] S310: Obtain the text to be annotated of the label to be annotated, and perform word segmentation on the text to be annotated and remove stop words to obtain the word segmentation result to be annotated corresponding to the text to be annotated.

[0093] The text to be annotated is the text that needs to be automatically labeled. It can be an ordinary text or a text obtained from the item introduction details diagram according to S210 to S220 in Example 2. Since the automatic labeling requires the use of various text labels that have been extracted from a large amount of text, the text to be annotated should belong to the same topic as the text extracted by each text label. After obtaining the text to be annotated, it is processed by word segmentation and stop word removal to obtain the word segmentation result corresponding to the text to be annotated, that is, the word segmentation result to be annotated.

[0094] S320 , determining the text label contained in the text to be annotated according to the word segmentation result and each text label, as the annotation label of the text to be annotated.

[0095] The word segmentation results to be annotated and the text labels are subjected to correlation analysis to determine the text labels contained in the text to be annotated. These text labels can be used for automatic text labeling of the text to be annotated. The text labels here can include all modified text labels and / or similar text labels.

[0096] Exemplarily, determining the text labels contained in the text to be annotated according to the word segmentation results to be annotated and each text label includes:

[0097] A. According to the preset sliding window, determine in turn whether each text label co-occurs in the word segmentation results to be annotated, and determine the co-occurring text labels as the text labels contained in the text to be annotated.

[0098] The correlation analysis between the word segmentation results and the text labels to be annotated can be performed by analyzing whether the word pairs co-occur. In the specific implementation, a preset sliding window is set, and the annotated text is slid according to a certain step size (such as 1 character or word). In each sliding window of the text to be annotated, it is determined whether a text label co-occurs. If co-occurs, the co-occurring text labels are determined as the text labels contained in the text to be annotated.

[0099] Exemplarily, according to a preset sliding window, determining in turn whether each text tag co-occurs in the word segmentation results to be marked includes: if the preset sliding window is larger than two words, segmenting each text tag to obtain a keyword group corresponding to each text tag; according to the preset sliding window, determining in turn whether each keyword group co-occurs in the word segmentation results to be marked.

[0100] The size of the preset sliding window is used to determine whether the text label needs to be segmented. For example, if the preset sliding window is 2 words, then no segmentation is required. If the preset sliding window is larger than 2 words, then the text label needs to be segmented. This is because the text label is a combination of two words. If you want to determine whether they co-occur among 3 or more words, the probability of co-occurrence after the text label is segmented will increase (in addition to the two words appearing in close proximity, it also includes the two words appearing at intervals), which can improve the stability of determining the text label in the text to be annotated. At this time, it is necessary to determine whether the keyword group corresponding to the text label co-occurs in the sliding window corresponding to the text to be annotated.

[0101] Exemplarily, determining the text labels contained in the text to be annotated according to the word segmentation results to be annotated and each text label includes:

[0102] B. Use the feature vector model to vectorize the word segmentation results to be labeled and each text label, and obtain the vector of each word segmentation to be labeled and the vector of each text label.

[0103] In order to solve the problem that the automatically extracted text labels (i.e., the text label list) are incomplete, that is, the personalized label extraction cannot fully cover the label meaning, such as "green and environmentally friendly" and "pollution-free" are both environmentally friendly labels. If there is only "green and environmentally friendly" in the text label list, but no "pollution-free", but there is only "pollution-free" in the text corresponding to the item introduction details, if the above method is used, there will be a labeling error. Therefore, the feature vector model is introduced in this embodiment to obtain the text label vector obtained by vectorizing each text label, as well as the word vector of the combination of each two segmented words in the segmented word results to be labeled, that is, the segmented word vector to be labeled.

[0104] C. Determine the second text similarity between each to-be-annotated word segmentation and each text label in the to-be-annotated word segmentation result based on each to-be-annotated word segmentation vector and each text label vector.

[0105] Calculate the text similarity between each to-be-annotated word segmentation vector and each text label vector, that is, the second text similarity.

[0106] D. Determine the text label contained in the text to be annotated based on each second text similarity and the second preset similarity threshold.

[0107] The text labels whose second text similarity is less than the second preset similarity threshold (similarity set empirically based on the accuracy required by the application) are removed, and the remaining text labels are the text labels contained in the text to be annotated. The above process is to determine the similarity between "pollution-free" and "green and environmentally friendly", so as to determine each annotation label by using text similarity. The advantage of this setting is that it can further improve the accuracy of determining the annotation label.

[0108] It should be noted that step A and steps B to D can be performed one by one or simultaneously. When performed simultaneously, two results of determining the annotation labels can be obtained. At this time, the results of determining the two annotation labels can be combined as the final annotation labels of the text to be annotated to ensure the comprehensiveness of the annotation labels; the results of determining the two annotation labels can also be intersected, and the overlapping text labels can be used as the final annotation labels of the text to be annotated to further ensure the accuracy of the annotation labels.

[0109] The technical solution of this embodiment obtains the text to be annotated of the label to be annotated, and performs word segmentation and stop word removal on the text to be annotated to obtain the word segmentation result corresponding to the text to be annotated; and determines the text label contained in the text to be annotated as the annotated label of the text to be annotated based on the word segmentation result to be annotated and each text label. Automatic labeling of the text to be annotated is realized, which largely solves the problems of human error and subjective arbitrariness caused by manual data annotation, and improves the efficiency and accuracy of text labeling.

[0110] Embodiment 4

[0111] This embodiment provides a text label extraction device. Figure 4 , the device specifically comprises:

[0112] The text vector acquisition module 410 is used to acquire each text of the tag to be extracted, and vectorize each text to obtain a text vector corresponding to the corresponding text;

[0113] A text clustering result obtaining module 420, used to cluster each text vector to obtain at least one text clustering result;

[0114] The label candidate word acquisition module 430 is used to extract keywords from each text clustering result to obtain each label candidate word corresponding to each text clustering result;

[0115] The text label determination module 440 is used to determine the text label of each text according to each label candidate word corresponding to each text clustering result.

[0116] Optionally, the text vector acquisition module 410 is specifically used for:

[0117] For each text, the text is segmented and stop words are removed to obtain the segmentation results to be extracted corresponding to the text, and at least one feature vector model is used to vectorize each segmentation result to be extracted to obtain a text vector corresponding to the text.

[0118] Optionally, the text label determination module 440 includes:

[0119] The label reorganization word generation submodule is used to determine the co-occurrence frequency between each two label candidate words corresponding to each text clustering result based on the co-occurrence matrix between words obtained in advance based on each text, and screen out at least one pair of label candidate words whose co-occurrence frequency meets a preset co-occurrence frequency threshold to generate each label reorganization word;

[0120] The text label determination submodule is used for determining, for each label reorganization word, the label reorganization word as a text label if the label reorganization word exists in each text.

[0121] Furthermore, the text label determination module 440 further includes:

[0122] The label candidate word screening submodule is used to determine the candidate word vector of each label candidate word corresponding to each text clustering result by using a feature vector model before determining the co-occurrence frequency between every two label candidate words corresponding to each text clustering result according to the co-occurrence matrix between words obtained in advance based on each text;

[0123] According to the candidate word vectors of every two label candidate words, the first text similarity between the corresponding two label candidate words is determined, and the label candidate words whose first text similarity is less than the first preset similarity threshold in each label candidate word corresponding to each text clustering result are eliminated.

[0124] Optionally, the text label determination submodule is specifically used for:

[0125] If there are label reorganization words in each text, the label reorganization words are determined as candidate labels;

[0126] If the word frequency of the candidate tag meets the preset word frequency threshold, the tag reorganization word is determined as the text tag.

[0127] Optionally, the text vector acquisition module 410 includes:

[0128] The text data acquisition submodule is used to perform text recognition on the item introduction details image to obtain the text data of the item introduction details image when the application scenario is to extract text labels from the item introduction details image;

[0129] The text acquisition submodule is used to filter the text in the text data by importance based on the text attributes in the text data and preset text filtering conditions, obtain the filtered texts corresponding to the item introduction details picture, and determine the texts of the labels to be extracted corresponding to the item introduction details picture based on the filtered texts.

[0130] Among them, the text acquisition submodule is specifically used for:

[0131] For every two vertically adjacent filter texts, if any of the absolute values ​​of the left edge difference, the center coordinate difference, and the right edge difference between the text boxes corresponding to the two filter texts in the text data is less than the set text box height, and the absolute value of the vertical difference between the two text boxes is less than the set text box height, then the two filter texts are merged as the vertical merged text, wherein the set text box height is the larger value of the text box heights of the two text boxes, and the absolute value of the vertical difference is the absolute value of the distance difference between the lower edge of the upper text box and the upper edge of the lower text box;

[0132] For every two horizontally adjacent vertically merged texts, if the vertical overlapping distance between the text boxes of the two vertically merged texts is less than the set text box height, the two vertically merged texts are merged as the text of the label to be extracted corresponding to the item introduction details image.

[0133] Optionally, based on the above device, the device further includes a text label correction module, which is used to:

[0134] After determining the text label of each text according to each label candidate word corresponding to each text clustering result, clustering each text label to obtain at least one label clustering result;

[0135] For each label clustering result, any text label in the label clustering result is used as a revised text label of the corresponding label clustering result, and each remaining text label in the label clustering result except the revised text label is used as a similar text label of the revised text label.

[0136] Optionally, based on the above device, the device further includes a text label marking module, and the text label marking module includes:

[0137] The submodule for obtaining the segmentation result to be annotated is used to obtain the text to be annotated of the label to be annotated, and to perform segmentation and remove stop words on the text to be annotated to obtain the segmentation result to be annotated corresponding to the text to be annotated;

[0138] The annotation label determination submodule is used to determine the text label contained in the text to be annotated according to the word segmentation result and each text label, as the annotation label of the text to be annotated.

[0139] Optionally, the label determination submodule is specifically used for:

[0140] According to the preset sliding window, it is determined in turn whether each text label co-occurs in the word segmentation results to be annotated, and the co-occurring text labels are determined as the text labels contained in the text to be annotated.

[0141] Furthermore, the labeling tag determines that the submodule is specifically used for:

[0142] If the preset sliding window is larger than two words, each text label is segmented to obtain the keyword group corresponding to each text label;

[0143] According to the preset sliding window, determine in turn whether each keyword group co-occurs in the word segmentation results to be marked.

[0144] Optionally, the label determination submodule is specifically used for:

[0145] Using the feature vector model, the word segmentation results to be labeled and each text label are vectorized respectively to obtain the vectors of each word segmentation to be labeled and each text label vector;

[0146] Determine the second text similarity between each to-be-annotated word segmentation and each text label in the to-be-annotated word segmentation result according to each to-be-annotated word segmentation vector and each text label vector;

[0147] According to each second text similarity and a second preset similarity threshold, a text label included in the text to be annotated is determined.

[0148] Through a text label extraction device of embodiment 4 of the present invention, unsupervised automatic label extraction of various texts under the same topic is realized, which solves the problem of incomplete and insufficient personalization of labels caused by only extracting labels from structured texts, and improves the personalization and comprehensiveness of text labels; at the same time, unsupervised text label extraction solves the problems of low label extraction efficiency and poor scalability of extraction methods caused by manual data annotation, improves the extraction efficiency of text labels, and improves the applicability of methods for label extraction under different topics, thereby enhancing the scalability of label extraction methods.

[0149] The text tag extraction device provided in the embodiment of the present invention can execute the text tag extraction method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0150] It is worth noting that in the embodiment of the above-mentioned text tag extraction device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0151] Embodiment 5

[0152] See also Figure 5 This embodiment provides a device, which includes: one or more processors 520; a storage device 510, which is used to store one or more programs. When the one or more programs are executed by the one or more processors 520, the one or more processors 520 implement the text tag extraction method provided by the embodiment of the present invention, including:

[0153] Obtain each text of the label to be extracted, and vectorize each text to obtain the text vector corresponding to the corresponding text;

[0154] Clustering each text vector to obtain at least one text clustering result;

[0155] Perform keyword extraction on each text clustering result to obtain candidate words for each label corresponding to each text clustering result;

[0156] According to each label candidate word corresponding to each text clustering result, the text label of each text is determined.

[0157] Of course, those skilled in the art can understand that the processor 520 can also implement the technical solution of the text tag extraction method provided by any embodiment of the present invention.

[0158] Figure 5 The device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention. Figure 5 As shown, the device includes a processor 520, a storage device 510, an input device 530 and an output device 540; the number of processors 520 in the device can be one or more. Figure 5 A processor 520 is taken as an example; the processor 520, the storage device 510, the input device 530 and the output device 540 in the device can be connected via a bus or other means. Figure 5 The connection via bus 550 is taken as an example.

[0159] The storage device 510, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the text tag extraction method in the embodiment of the present invention (for example, the text vector acquisition module, text clustering result acquisition module, tag candidate word acquisition module and text tag determination module in the text tag extraction device).

[0160] The storage device 510 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the storage device 510 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the storage device 510 may further include a memory remotely arranged relative to the processor 520, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0161] The input device 530 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 540 may include a display device such as a display screen.

[0162] Embodiment 6

[0163] This embodiment provides a storage medium containing computer executable instructions. When the computer executable instructions are executed by a computer processor, they are used to perform a text tag extraction method. The method includes:

[0164] Obtain each text of the label to be extracted, and vectorize each text to obtain the text vector corresponding to the corresponding text;

[0165] Clustering each text vector to obtain at least one text clustering result;

[0166] Perform keyword extraction on each text clustering result to obtain candidate words for each label corresponding to each text clustering result;

[0167] According to each label candidate word corresponding to each text clustering result, the text label of each text is determined.

[0168] Of course, the computer executable instructions of a storage medium including computer executable instructions provided in an embodiment of the present invention are not limited to the above method operations, and can also execute related operations in the text tag extraction method provided in any embodiment of the present invention.

[0169] Through the above description of the implementation method, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions to enable a device (which can be a personal computer, server, or network device, etc.) to execute the text label extraction method provided by each embodiment of the present invention.

[0170] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A text label extraction method, characterized in that: include: Obtaining each text of the tag to be extracted under the same topic, and vectorizing each of the texts to obtain a text vector corresponding to the corresponding text; Clustering each of the text vectors to obtain at least one text clustering result; Perform keyword extraction on each of the text clustering results to obtain each label candidate word corresponding to each of the text clustering results; Determining a text label for each of the texts according to each of the label candidate words corresponding to each of the text clustering results; Wherein, determining the text label of each of the texts according to each of the label candidate words corresponding to each of the text clustering results includes: According to the co-occurrence matrix between words obtained in advance based on each of the texts, determine the co-occurrence frequency between each two of the label candidate words corresponding to each of the text clustering results, and screen out at least one pair of the label candidate words whose co-occurrence frequency meets a preset co-occurrence frequency threshold, and generate each label reorganization word; For each of the label reorganized words, if the label reorganized word exists in each of the texts, the label reorganized word is determined as the text label; the co-occurrence matrix is ​​a matrix that counts the frequency of whether two words appear together in a text.

2. The method according to claim 1, characterized in that: Vectorizing each of the texts to obtain a text vector corresponding to the corresponding text includes: For each of the texts, the text is segmented and stop words are removed to obtain the segmentation results to be extracted corresponding to the text, and each of the segmentation results to be extracted is vectorized using at least one feature vector model to obtain a text vector corresponding to the text.

3. The method according to claim 1, characterized in that Before determining the co-occurrence frequency between each two of the label candidate words corresponding to each of the text clustering results according to the co-occurrence matrix between words obtained in advance based on each of the texts, the method further includes: Determine the candidate word vector of each of the label candidate words corresponding to each of the text clustering results by using a feature vector model; Based on the candidate word vectors of each two of the label candidate words, the first text similarity between the corresponding two label candidate words is determined, and the label candidate words whose first text similarity is less than a first preset similarity threshold in each of the label candidate words corresponding to each of the text clustering results are eliminated.

4. The method according to claim 1, characterized in that If the label reorganization word exists in each of the texts, determining the label reorganization word as the text label includes: If the label reorganization word exists in each of the texts, the label reorganization word is determined as a candidate label; If the word frequency of the candidate tag meets a preset word frequency threshold, the tag reorganization word is determined as the text tag.

5. The method according to claim 1, characterized in that When the application scenario is to extract text labels from the item description details image, the text of the labels to be extracted includes: Performing text recognition on the item introduction details image to obtain text data of the item introduction details image; According to the text attributes in the text data and the preset text filtering conditions, the text in the text data is filtered by importance to obtain the filtered texts corresponding to the item introduction details picture, and the texts of the labels to be extracted corresponding to the item introduction details picture are determined based on the filtered texts.

6. The method according to claim 5, characterized in that The texts of the labels to be extracted corresponding to the item introduction details image are determined according to the screening texts, including: For every two vertically adjacent filtered texts, if any of the absolute values ​​of the left edge difference, the center coordinate difference, and the right edge difference between the text boxes corresponding to the two filtered texts in the text data is less than the set text box height, and the absolute value of the vertical difference between the two text boxes is less than the set text box height, then the two filtered texts are merged as a vertical merged text, wherein the set text box height is the larger value of the text box heights of the two text boxes, and the absolute value of the vertical difference is the absolute value of the distance difference between the lower edge of the upper text box and the upper edge of the lower text box; For every two horizontally adjacent vertically merged texts, if the vertical overlapping distance between the text boxes of the two vertically merged texts is less than the set text box height, the two vertically merged texts are merged as the text of the label to be extracted corresponding to the item introduction details image.

7. The method according to claim 1, characterized in that After determining the text label of each of the texts according to each of the label candidate words corresponding to each of the text clustering results, the method further includes: Clustering each of the text labels to obtain at least one label clustering result; For each of the label clustering results, any of the text labels in the label clustering results is used as a revised text label of the corresponding label clustering result, and each remaining text label in the label clustering result except the revised text label is used as a similar text label of the revised text label.

8. The method according to claim 1, characterized in that: After determining the text label of each of the texts according to each of the label candidate words corresponding to each of the text clustering results, the method further includes: Obtain the text to be annotated of the label to be annotated, and perform word segmentation and stop word removal on the text to be annotated to obtain the word segmentation result to be annotated corresponding to the text to be annotated; According to the word segmentation result of the text to be annotated and each of the text labels, a text label included in the text to be annotated is determined as a labeling label of the text to be annotated.

9. The method according to claim 8, characterized in that Determining the text labels contained in the text to be annotated according to the word segmentation result to be annotated and each of the text labels includes: According to the preset sliding window, it is determined in turn whether the text tags co-occur in the word segmentation results to be annotated, and the co-occurring text tags are determined as the text tags contained in the text to be annotated.

10. The method according to claim 9, characterized in that According to the preset sliding window, determining in turn whether each of the text labels co-occurs in the word segmentation results to be annotated includes: If the preset sliding window is larger than two words, each of the text tags is segmented to obtain a keyword group corresponding to each of the text tags; According to the preset sliding window, it is determined in turn whether each of the keyword groups co-occurs in the word segmentation results to be marked.

11. The method according to claim 8, characterized in that Determine the text labels contained in the text to be annotated based on the word segmentation results and the text labels include: Using a feature vector model, the word segmentation results to be labeled and the text labels are respectively vectorized to obtain the word segmentation vectors to be labeled and the text label vectors; Determining, based on each of the to-be-annotated word segmentation vectors and each of the text label vectors, a second text similarity between each to-be-annotated word segmentation in the to-be-annotated word segmentation result and each of the text labels; According to each of the second text similarities and the second preset similarity threshold, a text label included in the text to be annotated is determined.

12. A text label extraction device, characterized in that: include: A text vector acquisition module is used to acquire each text of the tag to be extracted, and vectorize each of the texts to obtain a text vector corresponding to the corresponding text; A text clustering result obtaining module, used for clustering each of the text vectors to obtain at least one text clustering result; A label candidate word acquisition module is used to extract keywords from each of the text clustering results to obtain label candidate words corresponding to each of the text clustering results; A text label determination module, used to determine the text label of each of the texts according to each of the label candidate words corresponding to each of the text clustering results; Wherein, the text label determination module includes: A label reorganization word generation submodule is used to determine the co-occurrence frequency between each two label candidate words corresponding to each text clustering result according to the co-occurrence matrix between words obtained in advance based on each of the texts, and screen out at least one pair of label candidate words whose co-occurrence frequency meets a preset co-occurrence frequency threshold to generate each label reorganization word; The text label determination submodule is used to determine each of the label reorganization words as the text label if the label reorganization word exists in each of the texts; the co-occurrence matrix is ​​a matrix that counts the frequency of whether two words appear together in a text.

13. A device, characterized in that: The device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the text tag extraction method as described in any one of claims 1-11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the text tag extraction method according to any one of claims 1 to 11 is implemented.

Images