A method and device for determining text data

By obtaining text data, filtering keyword entry and deduplication of historical text data, the problems of high similarity, low efficiency and poor timeliness in the prior art are solved, and more efficient text data extraction and storage are achieved.

CN113495954BActive Publication Date: 2024-11-26BEIJING WODONG TIANJUN INFORMATION TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010201540.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-03-20
Publication Date
2024-11-26
Estimated Expiration
2040-03-20

AI Technical Summary

Technical Problem

When extracting text data, the existing methods have high similarity, low extraction efficiency and poor timeliness.

Method used

By obtaining text data, filtering according to keyword entries to obtain alternative text data, and using the historical text data in the storage device to deduplicate similarity, allocating storage sequence numbers and identification codes, and place target text data into the storage device.

Benefits of technology

It reduces the similarity of text data, improves the extraction efficiency and timeliness, and improves the convenience of text data extraction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113495954B_ABST
    Figure CN113495954B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and device for determining text data, and relates to the field of computer technology. A specific implementation of the method includes: obtaining text data; filtering the text data according to key words to obtain candidate text data; performing similarity deduplication on the candidate text data according to historical text data in a storage device to determine target text data; assigning a storage sequence number to each target text data, determining an identification code corresponding to the target text according to the storage sequence number and the type of the target text data, and placing the target text data into a storage device. This implementation reduces the similarity of text data, improves the extraction efficiency and timeliness of text data, and improves the convenience of subsequent extraction of text data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for determining text data. Background Art

[0002] Compared with traditional media, new media has greater influence, stronger cohesion, and more convenient dissemination, and has become an important platform and carrier for different interest groups to express their demands. For enterprises or individuals, how to extract text data related to their own needs from new media in real time and effectively is becoming increasingly important.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0004] The text data determined by the existing methods have high similarity, low extraction efficiency and poor timeliness. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a text data determination method and device, which can reduce the similarity of extracted text data and improve extraction efficiency and timeliness.

[0006] To achieve the above object, according to a first aspect of an embodiment of the present invention, a text data determination method is provided, comprising:

[0007] Get text data;

[0008] Filtering the text data according to the key words to obtain candidate text data;

[0009] Deduplication of candidate text data based on similarity according to historical text data in the storage device to determine target text data;

[0010] A storage serial number is assigned to each target text data, an identification code corresponding to the target text is determined according to the storage serial number and the type of the target text data, and the target text data is placed in a storage device.

[0011] Furthermore, before the step of filtering the text data according to the key words to obtain the candidate text data, the text data determination method further includes: removing duplicate text data according to the type and title of the text data.

[0012] Furthermore, the step of deduplicating identical text data according to the type and title of the text data includes: valuing the title of the identical type of text data according to a value-taking algorithm to obtain a key value corresponding to the identical type of text data, and deduplicating the identical text data according to the key value.

[0013] Furthermore, the step of performing similarity deduplication on the alternative text data based on the historical text data in the storage device to determine the target text data includes: obtaining feature vectors corresponding to the alternative text data after performing word segmentation processing on the alternative text data according to the word segmentation algorithm, performing similarity calculation based on the feature vectors corresponding to the alternative text data and the feature vectors corresponding to the historical text data, and determining the target text data based on the similarity calculation result.

[0014] Furthermore, the step of calculating similarity based on the feature vectors corresponding to the candidate text data and the feature vectors corresponding to the historical text data includes: configuring a single calculation quantity, and calculating similarity based on the feature vectors corresponding to the candidate text data and the feature vectors corresponding to the single calculation quantity of the historical text data.

[0015] Furthermore, after the step of placing the target text data into the storage device, the text data determination method also includes: sending an update notification to the subscription department, so that after the subscription department receives the update notification, it extracts the corresponding target text data from the storage device according to the text data type indicated by the identification code and the keyword entry, wherein the keyword entry is set by the subscription department.

[0016] According to a second aspect of an embodiment of the present invention, there is provided a text data determination device, comprising:

[0017] A text data acquisition module, used for acquiring text data;

[0018] A filtering module, used to filter text data according to key words to obtain candidate text data;

[0019] A target text data determination module is used to perform similarity deduplication on the candidate text data according to the historical text data in the storage device to determine the target text data;

[0020] The storage module is used to assign a storage serial number to each target text data, determine the identification code corresponding to the target text according to the storage serial number and the type of the target text data, and place the target text data into the storage device.

[0021] Furthermore, the text data determination device also includes a deduplication module. Before the step of filtering the text data according to the key words to obtain the candidate text data, the deduplication module is used to: deduplication the same text data according to the type and title of the text data.

[0022] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0023] one or more processors;

[0024] a storage device for storing one or more programs,

[0025] When one or more programs are executed by one or more processors, the one or more processors implement any of the above-mentioned text data determination methods.

[0026] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, any of the above-mentioned text data determination methods is implemented.

[0027] An embodiment of the above invention has the following advantages or beneficial effects: because it adopts the method of acquiring text data; filtering the text data according to key words to obtain alternative text data; performing similarity deduplication on the alternative text data according to the historical text data in the storage device to determine the target text data; assigning a storage sequence number to each target text data, determining the identification code corresponding to the target text according to the storage sequence number and the type of the target text data, and placing the target text data into the storage device, the technical problems of the high similarity, low extraction efficiency and poor timeliness of the text data determined by the existing method are overcome, thereby achieving the technical effect of reducing the similarity of text data, improving the extraction efficiency and timeliness of text data, and enhancing the convenience of subsequent extraction of text data.

[0028] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0030] Figure 1 is a schematic diagram of the main process of the text data determination method provided according to the first embodiment of the present invention;

[0031] Figure 2a is a schematic diagram of the main process of the text data determination method provided according to the second embodiment of the present invention;

[0032] Figure 2b yes Figure 2a An example of filtering text data based on key words in the method shown;

[0033] Figure 2c yes Figure 2a An example of calculating similarity in the method shown;

[0034] Figure 3 is a schematic diagram of main modules of a text data determination device provided according to an embodiment of the present invention;

[0035] Figure 4 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0036] Figure 5 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0038] Figure 1 Schematic diagram of the main process of the text data determination method provided according to the first embodiment of the present invention; Figure 1 As shown, the text data determination method provided by the embodiment of the present invention mainly includes:

[0039] Step S101, obtaining text data.

[0040] Specifically, according to an embodiment of the present invention, text data can be obtained from various departments within an enterprise / company or platforms or enterprises that have cooperation with the enterprise / company. Providing text data through the above channels effectively avoids a large amount of invalid text data caused by random extraction, and also avoids the repetition of text extraction methods, thereby improving the efficiency of subsequent text extraction.

[0041] Step S102: filtering the text data according to the key words to obtain candidate text data.

[0042] The text data is filtered according to the key words set by the enterprise or the departments within the enterprise, which is highly targeted and enhances the subsequent text data determination effect. The key words can be set to multiple, for example, the enterprise name, enterprise brand, etc.

[0043] According to an embodiment of the present invention, before the step of filtering the text data according to the key words to obtain the candidate text data, the text data determination method further comprises: removing duplicate text data according to the type and title of the text data.

[0044] The type of text data includes the channel type (referring to each platform or enterprise, etc.) of the text data release or the type of text data divided according to the attribute (such as literature, history, etc.). Since the same text data may be published multiple times on the same platform or enterprise, there are a large number of text data with the same title in the text data obtained from the same channel (referring to the above platform or enterprise, etc.). Through the above setting, only one text data with the same title is obtained from the multiple text data with the same title from the same channel, which reduces the number of identical text data, thereby increasing the amount of calculation for subsequent similarity calculation and improving the efficiency of text data extraction.

[0045] Further, according to an embodiment of the present invention, the step of deduplicating identical text data according to the type and title of the text data includes: evaluating the titles of the identical type of text data according to a value-taking algorithm to obtain key values ​​corresponding to the identical type of text data, and deduplicating the identical text data according to the key values.

[0046] Specifically, since the type of each text data is unique, the title of the same type of text data is subjected to a value-taking operation (such as using the message-digest algorithm MD5 (Message-Digest Algorithm 5), a hash algorithm, etc.), and the text data with the same key value is deduplicated, and only one of them is retained.

[0047] Step S103: perform similarity deduplication on the candidate text data according to the historical text data in the storage device to determine the target text data.

[0048] According to an embodiment of the present invention, the step of performing similarity deduplication on alternative text data based on historical text data in a storage device to determine target text data includes: obtaining feature vectors of the alternative text data after performing word segmentation processing on the alternative text data according to a word segmentation algorithm, performing similarity calculation based on the feature vectors corresponding to the alternative text data and the feature vectors corresponding to the historical text data, and determining the target text data based on the similarity calculation result.

[0049] Through the above settings, multiple feature terms are obtained after the alternative text data is segmented by the word segmentation algorithm, and then the feature vector corresponding to the alternative text data is determined according to the multiple feature terms. The similarity is then calculated between the feature vector corresponding to the alternative text data and the feature vector of the historical text data in the storage device, and the text data that reaches or exceeds the similarity threshold is removed, which significantly reduces the similarity of the extracted text data and improves the extraction effect.

[0050] Specifically, according to an embodiment of the present invention, the step of calculating similarity based on the feature vectors corresponding to the alternative text data and the feature vectors corresponding to the historical text data includes: configuring the number of single calculations, and calculating similarity based on the feature vectors corresponding to the alternative text data and the feature vectors corresponding to the number of single calculations of historical text data.

[0051] Through the above settings, by configuring the number of calculations for each similarity calculation, the extraction efficiency and the performance of the similarity calculation device can be dynamically coordinated, so that both the extraction service and the performance can be guaranteed.

[0052] Step S104, assigning a storage serial number to each target text data, determining an identification code corresponding to the target text according to the storage serial number and the type of the target text data, and placing the target text data into a storage device.

[0053] Through the above settings, a storage sequence number is assigned to each target text data, and then an identification code is determined, which is conducive to extracting text data in the storage device and calculating similarity with the candidate text data; it is also conducive to extracting corresponding text data from the text data type indicated in the identification code according to the text data type indicated by the keyword and sending it to the subscription department corresponding to the keyword, thereby improving the text data extraction efficiency of each subscription department. At the same time, the text data with higher similarity in the candidate text data is first removed by similarity calculation, and then the obtained target text data is stored in the storage device. This avoids the problem of poor timeliness caused by first storing the acquired text data in the storage device and then extracting it regularly in the existing method.

[0054] According to an embodiment of the present invention, after the step of placing the target text data into a storage device, the text data determination method further includes: sending an update notification to a subscription department, so that after receiving the update notification, the subscription department extracts the corresponding target text data from the storage device according to the text data type indicated by the identification code and the keyword entry, wherein the keyword entry is set by the subscription department.

[0055] Specifically, according to a specific implementation of an embodiment of the present invention, the update notification can be in the form of business notification, email report, SMS notification, etc. By sending the update notification to the subscription department, it is convenient for the subscription department to extract text data according to its actual situation, thereby meeting the timeliness of the text data. The identification code indicates the type of the target text data, so that the target text data can be distributed in different sections / blocks in the storage device according to the type, which is convenient for the subscription department to directly go to the corresponding section / block of the storage device according to the type indicated by the set keyword entry to perform the extraction operation of the corresponding target text data according to the keyword entry, especially when there are many keyword entries, the above setting effectively improves the extraction efficiency of the corresponding target text data from the storage device according to different keyword entries.

[0056] According to the technical solution of the embodiment of the present invention, the text data is acquired; the text data is filtered according to key words to obtain candidate text data; the candidate text data is deduplicated based on similarity according to the historical text data in the storage device to determine the target text data; a storage sequence number is assigned to each target text data, an identification code corresponding to the target text is determined according to the storage sequence number and the type of the target text data, and the target text data is placed in the storage device. Therefore, the technical problems of the text data determined by the existing method having high similarity, low extraction efficiency and poor timeliness are overcome, thereby achieving the technical effect of reducing the similarity of text data, improving the extraction efficiency and timeliness of text data, and enhancing the convenience of subsequent extraction of text data.

[0057] Figure 2a Schematic diagram of the main process of the text data determination method provided according to the second embodiment of the present invention; Figure 2a As shown, the text data determination method provided by the embodiment of the present invention mainly includes:

[0058] Step S201, obtaining text data.

[0059] Specifically, according to an embodiment of the present invention, text data can be obtained from various departments within an enterprise / company or platforms or enterprises that have cooperation with the enterprise / company. Providing text data through the above channels effectively avoids a large amount of invalid text data caused by random extraction, and also avoids the repetition of text extraction methods, thereby improving the efficiency of subsequent text extraction.

[0060] Step S202, obtaining key values ​​corresponding to the text data of the same type by evaluating the titles of the text data of the same type according to the evaluation algorithm, and deduplicating the same text data according to the key values.

[0061] The type of text data includes the channel type (referring to each platform or enterprise, etc.) through which the text data is published or the type of text data divided according to attributes (such as literature, history, etc.). Since the same text data may be published multiple times on the same platform or enterprise, there are a large number of text data with the same title in the text data obtained from the same channel (referring to the above platform or enterprise, etc.). Through the above setting, only one text data with the same title is obtained from the same channel, which reduces the number of identical text data, thereby increasing the amount of calculation for subsequent similarity calculation and improving the efficiency of text data determination.

[0062] Specifically, since the type of each text data is unique, the title of the same type of text data is subjected to a value-taking operation (such as using the information-digest algorithm MD5, the hash algorithm, etc.), and the text data with the same key value is deduplicated, and only one of them is retained. According to a specific implementation method of an embodiment of the present invention, the title of the text data is subjected to a value-taking operation according to setNx of redis (a storage system) to obtain the key value of the text data. The key values ​​obtained by using the MD5 algorithm for text data with the same title are the same. For multiple (greater than or equal to two) text data with the same key value, only one of them is retained after deduplication.

[0063] Step S203: filtering the text data according to the key words to obtain candidate text data.

[0064] The text data is filtered according to the key words set by the enterprise or the departments within the enterprise, which is highly targeted and enhances the subsequent text data determination effect. The key words can be set to multiple, for example, the enterprise name, enterprise brand, etc.

[0065] Specifically, according to a specific implementation of an embodiment of the present invention, a DFA (Deterministic Finite Automaton) algorithm can be used to filter text data according to key words set by each subscription department (which can be a word consisting of a brand, the company name, etc.) to obtain candidate text data. It should be noted that the above algorithm is not intended to limit the present invention. Filtering according to key words can also use a hash or Trie (prefix tree or dictionary tree)-based algorithm to search for key words to complete the filtering operation.

[0066] Among them, DFA stands for Deterministic Finite Automaton. Its characteristics are: there is a finite set of states and some edges from one state to another, each edge is marked with a symbol, one state is the initial state, and some states are final states. But unlike the uncertain finite automaton, in DFA, no two edges starting from the same state are marked with the same symbol.

[0067] In simple terms, it gets the next state through the event and the current state, that is, event + state = nextstate. It can be understood that there are multiple nodes in the system, and the incoming event is passed to determine which route to take to another node, and the number of nodes is limited. Figure 2bAs shown, the above algorithm can be used to search for key words "I am Chinese", "I am a middleman", "I am a good person", and "I am a bad person" to complete subsequent filtering and determine candidate text data.

[0068] Step S204, after performing word segmentation processing on the candidate text data according to the word segmentation algorithm, obtain the feature vector corresponding to the candidate text data, perform similarity calculation based on the feature vector corresponding to the candidate text data and the feature vector corresponding to the historical text data, and determine the target text data based on the similarity calculation result.

[0069] Through the above settings, multiple feature terms are obtained after the alternative text data is segmented by the word segmentation algorithm, and then the feature vector corresponding to the alternative text data is determined according to the multiple feature terms. The similarity is then calculated between the feature vector corresponding to the alternative text data and the feature vector of the historical text data in the storage device, and the text data that reaches or exceeds the similarity threshold is removed, which significantly reduces the similarity of the extracted text data and improves the extraction effect.

[0070] Among them, according to a specific implementation of an embodiment of the present invention, the word segmentation algorithm can adopt the Han LP algorithm. HanLP is an open source Java toolkit dedicated to popularizing NLP (Neuro-Linguistic Programming) technology to production environments, supporting: Chinese word segmentation (N-shortest path word segmentation, CRF (Conditional Random Fields) word segmentation, index word segmentation, user-defined dictionary, part-of-speech tagging); named entity recognition (Chinese names, transliterated names, Japanese names, place names, entity organization name recognition); keyword extraction, automatic summarization, phrase extraction, pinyin conversion, simplified and traditional conversion, text recommendation, dependency syntactic analysis (MaxEnt (maximum entropy model) dependency syntactic analysis, neural network dependency syntactic analysis).

[0071] According to an embodiment of the present invention, historical text data in a storage device can be queried through a redis serial number. Text data stored in a redis storage device will generate a serial number of the current category according to the redis incr operation. The historical text data can be queried later according to the serial number.

[0072] According to an embodiment of the present invention, similarity calculation is performed on the iterative historical text data and the candidate text data, and similarity deduplication is performed according to the similarity calculation result to determine the target text data.

[0073] Specifically, the cosine similarity method (calculating the angle between two feature vectors to represent the similarity between the two) can be used. Figure 2cAs shown, the eigenvector a and the eigenvector b and the angle θ between them, the coordinates of the eigenvectors a and b are set to (x1, y1) and (x2, y2) respectively, then:

[0074]

[0075] Assuming that feature vector a is the feature vector corresponding to the candidate text data; feature vector b is the feature vector corresponding to the historical text data, then the similarity between the two can be expressed by cosθ, and 0≤cosθ≤1.

[0076] Furthermore, regarding the determination of the feature vector corresponding to the text data, the idea of ​​TF-IDF (termfrequency–inverse document frequency) can be adopted, and it can be expressed by terms / term frequencies. For example, the terms of the candidate text data are ["I", "love", "Marvel"], and the terms of the historical text data are ["I", "like", "Marvel", "movie"], then the set of all term frequencies is ["I", "love", "like", "Marvel", "movie"]. After converting by calculating the corresponding term frequencies, the feature vector corresponding to the candidate text is a=[1,1,0,1,0], and the feature vector corresponding to the historical text data is b=[1,0,1,1,1]. Then the similarity result is calculated by the above formula, and then the similarity is deduplicated by comparing the similarity result with the set similarity threshold to obtain the target text data.

[0077] Specifically, according to an embodiment of the present invention, the step of calculating similarity based on the feature vectors corresponding to the alternative text data and the feature vectors corresponding to the historical text data includes: configuring the number of single calculations, and calculating similarity based on the feature vectors corresponding to the alternative text data and the feature vectors corresponding to the number of single calculations of historical text data.

[0078] Through the above settings, by configuring the number of calculations for each similarity calculation, the extraction efficiency and the performance of the similarity calculation device can be dynamically coordinated, so that both the extraction service and the performance can be guaranteed.

[0079] Step S205, assigning a storage serial number to each target text data, determining an identification code corresponding to the target text according to the storage serial number and the type of the target text data, and placing the target text data into a storage device.

[0080] Through the above settings, a storage sequence number is assigned to each target text data, and then an identification code is determined, which is conducive to extracting text data in the storage device and calculating similarity with the candidate text data; it is also conducive to extracting corresponding text data from the text data type indicated in the identification code according to the text data type indicated by the keyword and sending it to the enterprise or departments within the enterprise corresponding to the keyword, thereby improving the efficiency of determining text data for each department. At the same time, the text data with higher similarity in the candidate text data is first removed through similarity calculation, and then the obtained target text data is stored in the storage device. This avoids the problem of poor timeliness caused by first storing the acquired text data in the storage device and then extracting it regularly in the existing method.

[0081] Step S206, sending an update notification to the subscription department, so that after receiving the update notification, the subscription department extracts corresponding target text data from the storage device according to the identification code and the text data type indicated by the keyword, wherein the keyword is set by the subscription department.

[0082] Specifically, by sending an update notification to the subscription department, it is convenient for the subscription department to extract text data according to its actual situation, thus meeting the timeliness of the text data. The identification code indicates the type of the target text data, so that the target text data can be distributed in different sections / blocks in the storage device according to the type, which is convenient for the subscription department to directly go to the corresponding section / block of the storage device to extract the corresponding target text data according to the keyword entry according to the type indicated by the keyword entry set by it. Especially when there are many keyword entries, the above setting effectively improves the extraction efficiency of extracting the corresponding target text data from the storage device according to different keyword entries.

[0083] According to the technical solution of the embodiment of the present invention, the text data is acquired; the text data is filtered according to key words to obtain candidate text data; the candidate text data is deduplicated based on similarity according to the historical text data in the storage device to determine the target text data; a storage sequence number is assigned to each target text data, an identification code corresponding to the target text is determined according to the storage sequence number and the type of the target text data, and the target text data is placed in the storage device. Therefore, the technical problems of the text data determined by the existing method having high similarity, low extraction efficiency and poor timeliness are overcome, thereby achieving the technical effect of reducing the similarity of text data, improving the extraction efficiency and timeliness of text data, and enhancing the convenience of subsequent extraction of text data.

[0084] Figure 3 is a schematic diagram of the main modules of the text data determination device provided according to an embodiment of the present invention; Figure 3 As shown, the text data determination device 300 provided in the embodiment of the present invention mainly includes:

[0085] The text data acquisition module 301 is used to acquire text data.

[0086] Specifically, text data can be obtained from various departments within the enterprise or from platforms or enterprises that have cooperation with the enterprise. Providing text data from the above channels effectively avoids a large amount of invalid text data caused by random extraction, avoids the repetition of text extraction methods, and improves the efficiency of subsequent text extraction.

[0087] The filtering module 302 is used to filter the text data according to the key words to obtain candidate text data.

[0088] The text data is filtered according to the key words set by the enterprise or the departments within the enterprise, which is highly targeted and enhances the subsequent text data determination effect. The key words can be set to multiple, for example, the enterprise name, enterprise brand, etc.

[0089] According to an embodiment of the present invention, the above-mentioned text data determination device 300 also includes a deduplication module. Before the step of filtering the text data according to the key words to obtain the candidate text data, the deduplication module is used to dedupe the same text data according to the type and title of the text data.

[0090] The type of text data includes the channel type (referring to each platform or enterprise, etc.) through which the text data is published or the type of text data divided according to attributes (such as literature, history, etc.). Since the same text data may be published multiple times on the same platform or enterprise, there are a large number of text data with the same title in the text data obtained from the same channel (referring to the above platform or enterprise, etc.). Through the above setting, only one text data with the same title is obtained from the same channel, which reduces the number of identical text data, thereby increasing the amount of calculation for subsequent similarity calculation and improving the efficiency of text data determination.

[0091] Further, according to an embodiment of the present invention, the deduplication module is also used to: perform value extraction on the titles of the same type of text data according to a value extraction algorithm to obtain key values ​​corresponding to the same type of text data, and perform deduplication of the same text data according to the key values.

[0092] Specifically, since the type of each text data is unique, the title of the same type of text data is subjected to a value-taking operation (such as using the message-digest algorithm MD5 (Message-Digest Algorithm 5), a hash algorithm, etc.), and the text data with the same key value is deduplicated, and only one of them is retained.

[0093] The target text data determination module 303 is used to perform similarity deduplication on the candidate text data according to the historical text data in the storage device to determine the target text data.

[0094] According to an embodiment of the present invention, the target text data determination module 303 is also used to: obtain feature vectors of the alternative text data after performing word segmentation processing on the alternative text data according to the word segmentation algorithm, perform similarity calculation based on the feature vectors corresponding to the alternative text data and the feature vectors corresponding to the historical text data, and determine the target text data based on the similarity calculation result.

[0095] Through the above settings, multiple feature terms are obtained after the alternative text data is segmented by the word segmentation algorithm, and then the feature vector corresponding to the alternative text data is determined according to the multiple feature terms. The similarity is then calculated between the feature vector corresponding to the alternative text data and the feature vector of the historical text data in the storage device, and the text data that reaches or exceeds the similarity threshold is removed, which significantly reduces the similarity of the extracted text data and improves the extraction effect.

[0096] Specifically, according to an embodiment of the present invention, the target text data determination module 303 is further used to: configure a single calculation quantity, and perform similarity calculation based on feature vectors corresponding to the candidate text data and feature vectors corresponding to a single calculation quantity of historical text data.

[0097] Through the above settings, by configuring the number of calculations for each similarity calculation, the extraction efficiency and the performance of the similarity calculation device can be dynamically coordinated, so that both the extraction service and the performance can be guaranteed.

[0098] The storage module 304 is used to assign a storage serial number to each target text data, determine the identification code corresponding to the target text according to the storage serial number and the type of the target text data, and place the target text data into a storage device.

[0099] Through the above settings, a storage sequence number is assigned to each target text data, and then an identification code is determined, which is conducive to extracting text data in the storage device and calculating similarity with the candidate text data; it is also conducive to extracting corresponding text data from the text data type indicated in the identification code according to the text data type indicated by the keyword and sending it to the enterprise or departments within the enterprise corresponding to the keyword, thereby improving the efficiency of determining text data for each department. At the same time, the text data with higher similarity in the candidate text data is first removed through similarity calculation, and then the obtained target text data is stored in the storage device. This avoids the problem of poor timeliness caused by first storing the acquired text data in the storage device and then extracting it regularly in the existing method.

[0100] According to an embodiment of the present invention, the above-mentioned text data determination device 300 also includes an update notification module, which is used to: send an update notification to the subscription department, so that after the subscription department receives the update notification, it extracts the corresponding target text data from the storage device according to the text data type indicated by the identification code and the keyword entry, wherein the keyword entry is set by the subscription department.

[0101] Specifically, by sending an update notification to the subscription department, it is convenient for the subscription department to extract text data according to its actual situation, thus meeting the timeliness of the text data. The identification code indicates the type of the target text data, so that the target text data can be distributed in different sections / blocks in the storage device according to the type, which is convenient for the subscription department to directly go to the corresponding section / block of the storage device to extract the corresponding target text data according to the keyword entry according to the type indicated by the keyword entry set by it. Especially when there are many keyword entries, the above setting effectively improves the extraction efficiency of extracting the corresponding target text data from the storage device according to different keyword entries, and improves the convenience of subsequent extraction of text data.

[0102] According to the technical solution of the embodiment of the present invention, the text data is acquired; the text data is filtered according to key words to obtain candidate text data; the candidate text data is deduplicated based on similarity according to the historical text data in the storage device to determine the target text data; a storage sequence number is assigned to each target text data, an identification code corresponding to the target text is determined according to the storage sequence number and the type of the target text data, and the target text data is placed in the storage device. Therefore, the technical problems of the text data determined by the existing method having high similarity, low extraction efficiency and poor timeliness are overcome, thereby achieving the technical effect of reducing the similarity of text data, improving the extraction efficiency and timeliness of text data, and enhancing the convenience of subsequent extraction of text data.

[0103] Figure 4 An exemplary system architecture 400 is shown to which the text data determination method or text data determination apparatus according to the embodiment of the present invention can be applied.

[0104] like Figure 4 As shown, the system architecture 400 may include terminal devices 401, 402, 403, a network 404 and a server 405 (this architecture is only an example, and the components included in the specific architecture may be adjusted according to the specific application). The network 404 is used to provide a medium for a communication link between the terminal devices 401, 402, 403 and the server 405. The network 404 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0105] Users can use terminal devices 401, 402, 403 to interact with server 405 through network 404 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 401, 402, 403, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0106] The terminal devices 401 , 402 , and 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0107] The server 405 may be a server that provides various services, such as a background management server (only for example) that provides support for shopping websites browsed by users using the terminal devices 401, 402, and 403. The background management server may analyze and process the received text data, keyword data, and other data, and feed back the processing results (such as candidate text data, target text data - only for example) to the terminal device.

[0108] It should be noted that the text data determination method provided in the embodiment of the present invention is generally executed by the server 405 , and accordingly, the text data determination device is generally disposed in the server 405 .

[0109] It should be understood that Figure 4 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0110] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer system 500 of a terminal device suitable for implementing an embodiment of the present invention. Figure 5 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0111] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0112] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0113] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present invention are executed.

[0114] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0115] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0116] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be arranged in a processor, for example, it may be described as: a processor includes a text data acquisition module, a filtering module, and a target text data determination module. The names of these modules do not constitute a limitation on the modules themselves in certain circumstances, for example, the text data acquisition module may also be described as a "module for acquiring text data".

[0117] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining text data; filtering the text data according to key words to obtain candidate text data; performing similarity deduplication on the candidate text data according to the historical text data in the storage device to determine the target text data; assigning a storage sequence number to each target text data, determining the identification code corresponding to the target text according to the storage sequence number and the type of the target text data, and placing the target text data into the storage device.

[0118] According to the technical solution of the embodiment of the present invention, the technical solution of the embodiment of the present invention adopts the following technical means: obtaining text data; filtering the text data according to key words to obtain candidate text data; performing similarity deduplication on the candidate text data according to the historical text data in the storage device to determine the target text data; assigning a storage sequence number to each target text data, determining the identification code corresponding to the target text according to the storage sequence number and the type of the target text data, and placing the target text data into the storage device, so as to overcome the technical problems of high similarity, low extraction efficiency and poor timeliness of the text data determined by the existing method, thereby achieving the technical effect of reducing the similarity of text data, improving the extraction efficiency and timeliness of text data, and enhancing the convenience of subsequent extraction of text data.

[0119] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for determining text data, characterized in that: include: Get text data; The titles of the same type of text data are valuated according to a valuing algorithm to obtain key values ​​corresponding to the same type of text data, and the same text data is deduplicated according to the key values, so that only one text data with the same title from the same channel is obtained; the type includes the channel type of the text data release; the channel type refers to each platform or enterprise; Filtering the text data according to key words to obtain candidate text data; the key words are company names or company brands; Deduplication of the candidate text data based on similarity according to the historical text data in the storage device to determine the target text data; A storage serial number is assigned to each target text data, an identification code corresponding to the target text is determined according to the storage serial number and the type of the target text data, and the target text data is placed in a storage device.

2. The text data determination method according to claim 1, characterized in that: The step of performing similarity deduplication on the candidate text data according to the historical text data in the storage device to determine the target text data includes: performing word segmentation on the candidate text data according to a word segmentation algorithm and obtaining a feature vector corresponding to the candidate text data, performing similarity calculation based on the feature vector corresponding to the candidate text data and the feature vector corresponding to the historical text data, and performing deduplication based on the similarity calculation result to determine the target text data.

3. The text data determination method according to claim 2, characterized in that: The step of calculating similarity based on the feature vectors corresponding to the candidate text data and the feature vectors corresponding to the historical text data includes: configuring a single calculation quantity, and calculating similarity based on the feature vectors corresponding to the candidate text data and the feature vectors corresponding to the historical text data of a single calculation quantity.

4. The text data determination method according to claim 1, characterized in that: After the step of placing the target text data into a storage device, the text data determination method further includes: sending an update notification to a subscription department, so that after receiving the update notification, the subscription department extracts the corresponding target text data from the storage device according to the text data type indicated by the identification code and the keyword entry, wherein the keyword entry is set by the subscription department.

5. A text data determination device, characterized in that: include: A text data acquisition module, used for acquiring text data; A filtering module is used to obtain the key value corresponding to the text data of the same type by obtaining the title of the text data of the same type according to the value-taking algorithm, and to remove the duplication of the same text data according to the key value, so that only one text data of the same title from the same channel is obtained; the type includes the channel type of the text data release; the channel type refers to each platform or enterprise; and is also used to filter the text data according to the keyword item to obtain the candidate text data; the keyword item is the enterprise name or enterprise brand; A target text data determination module, used to perform similarity deduplication on the candidate text data according to the historical text data in the storage device to determine the target text data; The storage module is used to assign a storage serial number to each target text data, determine the identification code corresponding to the target text according to the storage serial number and the type of the target text data, and place the target text data into a storage device.

6. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

7. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.