A data processing method, a data processing device and a data processing device
By using a deep neural network model to preprocess and transform documents, the problems of low efficiency and high memory consumption in document deduplication are solved, achieving efficient and accurate document deduplication.
Patent Information
- Application Number
- CN201910960199.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-10-10
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2039-10-10
AI Technical Summary
Existing technologies require significant manpower and have low computational efficiency and high memory consumption when deduplicating documents, which affects accuracy.
A deep neural network model is used to preprocess the documents, converting them into document vectors. Then, a document computation model is used to convert the document vectors into document values, and the similarity between document values is calculated to identify duplicate documents.
It improves the efficiency and accuracy of deduplication, reduces memory usage, simplifies the calculation process, and ensures the accuracy of document comparison.
Smart Images

Figure CN112651221B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a data processing method, device and a device for data processing. Background Art
[0002] With the development of Internet technology, people can obtain various network resources through retrieval systems or recommendation systems.
[0003] Since documents on different websites are often forwarded or borrowed, there are cases where articles on the Internet have the same or similar content. Therefore, in the retrieval system or recommendation system, it is necessary to filter articles with the same or similar content to provide users with diverse network resources.
[0004] However, the previous manual deduplication method consumes a lot of manpower costs, and for network resources such as news that need to be pushed in real time, the timeliness of manual deduplication is low. The general deduplication algorithm needs to segment the articles and organize them into a vocabulary. For industrial-level vocabulary of hundreds of thousands or even millions, not only is the memory usage high during the calculation process, which is easy to cause memory overflow, but it also affects the accuracy and efficiency of deduplication calculation. Summary of the invention
[0005] The embodiments of the present invention provide a data processing method, a device and a device for data processing, which can improve the efficiency and accuracy of filtering duplicate documents.
[0006] In order to solve the above problem, an embodiment of the present invention discloses a data processing method, which includes:
[0007] Preprocessing the document to obtain a document vector corresponding to the document;
[0008] Inputting the document vector into a document calculation model, wherein the document calculation model is a deep neural network model trained based on the collected document samples;
[0009] Determining a document value corresponding to the document according to an output of the document calculation model;
[0010] By calculating the similarity between the document values corresponding to at least two documents, it is determined whether there is a duplicate document in the at least two documents.
[0011] On the other hand, an embodiment of the present invention discloses a data processing device, the device comprising:
[0012] A document processing module, used to pre-process the document to obtain a document vector corresponding to the document;
[0013] A data input module, used to input the document vector into a document calculation model, wherein the document calculation model is a deep neural network model trained based on the collected document samples;
[0014] A value determination module, used to determine a document value corresponding to the document according to the output of the document calculation model;
[0015] The document comparison module is used to determine whether there is a duplicate document in the at least two documents by calculating the similarity between the document values corresponding to the at least two documents.
[0016] In another aspect, an embodiment of the present invention discloses a device for data processing, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations:
[0017] Preprocessing the document to obtain a document vector corresponding to the document;
[0018] Inputting the document vector into a document calculation model, wherein the document calculation model is a deep neural network model trained based on the collected document samples;
[0019] Determining a document value corresponding to the document according to an output of the document calculation model;
[0020] By calculating the similarity between the document values corresponding to at least two documents, it is determined whether there is a duplicate document in the at least two documents.
[0021] On the other hand, an embodiment of the present invention discloses a machine-readable medium having instructions stored thereon, which, when executed by one or more processors, enables the device to perform one or more of the data processing methods described above.
[0022] The embodiments of the present invention include the following advantages:
[0023] The embodiment of the present invention converts documents into document vectors. Compared with industrial-grade vocabulary, vectors not only occupy less memory, but also can improve the efficiency and accuracy of calculation, thereby improving the efficiency and accuracy of filtering duplicate documents. In addition, the embodiment of the present invention further converts the document vector into a document value through a document calculation model, and determines whether the document is a duplicate by comparing the similarity between the document values. That is, the embodiment of the present invention converts the document comparison process into a numerical comparison process, which can simplify the calculation process of document comparison and further improve the efficiency and accuracy of filtering duplicate documents. Furthermore, the document calculation model of the embodiment of the present invention is a deep neural network model trained based on collected document samples. The document calculation model trained based on a large amount of sample data can improve the accuracy of converting document vectors into document values, and can further ensure the accuracy of document comparison. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0025] Figure 1 is a flow chart of steps of an embodiment of a data processing method of the present invention;
[0026] Figure 2 is a structural block diagram of an embodiment of a data processing device of the present invention;
[0027] Figure 3 is a block diagram of a device 800 for data processing according to an embodiment of the present invention; and
[0028] Figure 4 It is a schematic diagram of the structure of the server in some embodiments of the present invention. DETAILED DESCRIPTION
[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0030] Method Embodiment
[0031] Reference Figure 1 , shows a flow chart of steps of a data processing method embodiment of the present invention, which may specifically include the following steps:
[0032] Step 101: pre-process the document to obtain a document vector corresponding to the document;
[0033] Step 102: input the document vector into a document calculation model, where the document calculation model is a deep neural network model trained based on the collected document samples;
[0034] Step 103: Determine the document value corresponding to the document according to the output of the document calculation model;
[0035] Step 104: Determine whether there is a duplicate document in the at least two documents by calculating the similarity between the document values corresponding to the at least two documents.
[0036] The data processing method of the embodiment of the present invention can be used to identify duplicate documents. It can be understood that the embodiment of the present invention does not limit the specific source of the document. For example, the document can be a web page document crawled by a search engine, or can be a document stored locally by a user, or can also be a document sent or received by a user through an instant messaging application. The embodiment of the present invention does not limit the specific type of the document. For example, the document includes but is not limited to: HTML (Hyper Text Markup Language) page documents, Word (Microsoft word processor) documents, TXT (text format) documents, etc.
[0037] The data processing method of the embodiment of the present invention can be run on electronic devices, including but not limited to: personal computers, mid-range computers, computer clusters, servers, smart phones, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, car computers, set-top boxes, smart TVs, wearable devices, etc.
[0038] In order to solve the problem that when a document contains an industrial vocabulary, it occupies a large amount of memory and affects the accuracy of calculation, the embodiment of the present invention first preprocesses the document to obtain a document vector corresponding to the document. Compared with the industrial vocabulary, the vector not only occupies less memory, but also improves the efficiency and accuracy of calculation. For example, a document can be represented as a 200-dimensional vector.
[0039] Among them, the preprocessing can specifically include the following operation steps: first, the document is segmented; then, the segmented words obtained by the segmentation process are filtered to filter out stop words and punctuation marks, etc., to obtain a segmented word sequence; finally, the segmented word sequence is vectorized to obtain a document vector corresponding to the document.
[0040] Of course, the embodiment of the present invention does not limit the specific method of converting a document into a vector. For example, after segmenting a document to obtain a segmentation sequence, a word vector model (such as Word2vec or Glove) can be used to determine the vector representation of each segmentation in the segmentation sequence, and the vector of each segmentation is weighted and summed to obtain the document vector of the document.
[0041] After preprocessing the document to obtain the document vector corresponding to the document, the document vector can be input into the document calculation model, and the document value corresponding to the document can be determined according to the output of the document calculation model. The document value can be an integer value or a binary value, and the document vector can be further converted into a value after calculation by the document calculation model. Since the same or similar documents have the same or similar vector representations, the document values calculated according to the same or similar vector representations are also the same or similar. Therefore, by calculating the similarity between the document values corresponding to at least two documents, it can be determined whether there are duplicate documents in the at least two documents. It should be noted that the duplicate documents can be documents with exactly the same content. Alternatively, two documents whose proportion of the same content meets a preset proportion can also be considered as duplicate documents.
[0042] In an optional embodiment of the present invention, step 104 determines whether there is a duplicate document in the at least two documents by calculating the similarity between the document values corresponding to the at least two documents, which may specifically include:
[0043] In the case where the document value is an integer value, if there are at least two first documents with the same integer value among the integer values corresponding to the at least two documents, then the at least two first documents are determined to be duplicate documents; or
[0044] In the case where the document value is a binary value, if there are at least two second documents whose binary value difference bits are smaller than a second preset value among the binary values corresponding to the at least two documents, the at least two second documents are determined to be duplicate documents.
[0045] Taking the determination of whether two documents are duplicate documents as an example, assuming that it is necessary to determine whether document A and document B are duplicate documents, firstly, preprocess document A and document B respectively to obtain document vector A1 corresponding to document A and document vector B1 corresponding to document B. Then, document vector A1 is input into the document calculation model, and the document value A2 corresponding to document A is determined according to the output of the document calculation model; similarly, document vector B1 is input into the document calculation model, and the document value B2 corresponding to document B is determined according to the output of the document calculation model.
[0046] In the first case, if the document value A2 and the document value B2 are integer values, it is determined whether the document value A2 and the document value B2 are the same. If they are the same, it is determined that document A and document B are duplicate documents.
[0047] In the second case, the document value A2 and the document value B2 are binary values, and the number of digits of the difference between the binary values of the document value A2 and the document value B2 is determined. Assuming that the decimal representation of the document value A2 is "100" and the decimal representation of the document value B2 is "110", the document value A2 and the document value B2 are expanded bit by bit to obtain the binary representation of the document value A2 as "01100100" and the binary representation of the document value B2 as "01101110". It can be seen that there are two different values in "01100100" and "01101110", that is, the number of digits of the difference is 2. In actual applications, a second preset value can be set in advance. If the number of digits of the difference between the binary values of the two documents is less than the second preset value, it means that the two binary values are relatively close, and then the two documents can be determined to be duplicate documents.
[0048] Referring to Table 1, a specific schematic diagram of a document A and a document B is shown. As shown in Table 1, the document A and the document B include three sections of text respectively.
[0049] Table 1
[0050]
[0051] Through the data processing method of the embodiment of the present invention, it can be determined that the document value A2 corresponding to the document A in the above Table 1 and the document value B2 corresponding to the document B are the same. Therefore, it can be determined that the document A and the document B are duplicate documents.
[0052] In an optional embodiment of the present invention, after the step of preprocessing the document to obtain the document vector corresponding to the document, the method may further include: binarizing the document vector corresponding to the document to obtain the binarized vector corresponding to the document.
[0053] Inputting the document vector into a document calculation model in step 102 may specifically include: inputting a binarized vector corresponding to the document into a document calculation model.
[0054] In order to further improve the calculation speed of the document calculation model, after preprocessing the document to obtain the document vector corresponding to the document, the embodiment of the present invention can also binarize the document vector to obtain the binarized vector corresponding to the document, and input the binarized vector corresponding to the document into the document calculation model for calculation to obtain the document value corresponding to the document.
[0055] In an optional embodiment of the present invention, binarizing the document vector corresponding to the document to obtain the binarized vector corresponding to the document may specifically include:
[0056] In each dimensional vector of the document vector corresponding to the document, a dimensional vector greater than a first preset value is set to a first value, and a dimensional vector less than the first preset value is set to a second value, to obtain a binary vector corresponding to the document.
[0057] In an application example of the present invention, each dimensional vector of the document vector is compared with a first preset value (assuming that the first preset value is 0). If the current dimensional vector is greater than 0, the current dimensional vector is set to a first value (assuming that the first value is 1). If the current dimensional vector is less than 0, the current dimensional vector is set to a second value (assuming that the second value is 0). This process continues until all dimensional vectors of the document vector are converted. In this way, the document vector can be converted into a binary vector containing only 0 or 1.
[0058] It is understood that the embodiment of the present invention does not limit the specific value of the first preset value. For example, the first preset value may also be the average value of all dimensional vectors of the document vector corresponding to the current document. The first value and the second value may be 0 or 1.
[0059] In an optional embodiment of the present invention, step 103 determines the document value corresponding to the document according to the output of the document calculation model, which may specifically include:
[0060] Step S1031, obtaining hidden layer output data of the document calculation model;
[0061] Step S1032: binarize the hidden layer output data to obtain a document value corresponding to the document.
[0062] In an embodiment of the present invention, the document computing model may be a deep neural network model. After the document vector (or binarized vector) corresponding to the document is input into the document computing model, the hidden layer output data of the document computing model may be obtained.
[0063] In an optional embodiment of the present invention, the document computing model may include multiple hidden layers, and obtaining the hidden layer output data of the document computing model may specifically include: obtaining the hidden layer output data closest to the output layer.
[0064] It should be noted that the document computing model may include multiple hidden layers, and the step S1032 may not need to obtain the data output by all hidden layers of the document computing model. For example, if the document computing model includes multiple hidden layers, only the hidden layer output data generated by one of the hidden layers may be obtained. In the case of multiple hidden layers, the hidden layer output data of any hidden layer may be obtained. As a preferred embodiment, the hidden layer output data closest to the output layer may be obtained.
[0065] During the layer-by-layer processing of the document calculation model, although the data is changing, the main features will still be retained, and the closer to the output layer, the more it can reflect deeper features, such as semantic features. Therefore, obtaining the hidden layer output data closest to the output layer can reflect the deep features of the document, and then reflect the comparison results of the document from deep features such as semantics, so as to further improve the accuracy of document comparison.
[0066] The hidden layer output data may be a vector, and the vector may be binarized to obtain the document value corresponding to the document. Taking the hidden layer output data as a 64-dimensional vector as an example, the dimension vectors greater than 0 in the 64-dimensional vector may be set to 1, and the dimension vectors less than 0 may be set to 0, and then each dimension vector of the 64-dimensional vector may be converted to 0 or 1, and finally the 64-dimensional vector in which each dimension vector is 0 or 1 may be converted to an integer value or a binary value to obtain the document value corresponding to the document.
[0067] In an optional embodiment of the present invention, the document computation model may include an autoencoder.
[0068] Among them, the autoencoder can be a deep neural network model. By training the autoencoder, a nonlinear machine learning algorithm in which the input is equal to the output (in practical applications, a certain error is allowed) can be implemented.
[0069] In an embodiment of the present invention, the autoencoder may include at least one hidden layer, and the document vector input to the autoencoder passes through the input layer, each hidden layer, and the output layer in sequence. Each layer may include multiple nodes, and each node will calculate the data from the upper layer node. The data calculated by the nodes of the hidden layer is the hidden layer output data, which reflects the intermediate state between the input and output of the autoencoder. According to the hidden layer output data, the embodiment of the present invention can determine the document value corresponding to the document vector input to the autoencoder.
[0070] In specific applications, the hidden layer values of the autoencoder have the problem of uncontrollable distribution, where uncontrollable distribution means that the hidden layer output of the autoencoder can be a multi-dimensional vector, but the values of each dimension may have large differences. For example, the hidden layer outputs a 100-dimensional vector, where the maximum value may be 100 and the minimum value may be -100, resulting in a very disordered distribution of values in each dimension. If the hidden layer values are not controlled, the accuracy of the calculation results will be affected. To solve this problem, the document calculation model of the embodiment of the present invention may include a variational autoencoder.
[0071] In an optional embodiment of the present invention, the document calculation model may include a variational autoencoder; when the input of the variational autoencoder is a document vector, the variational autoencoder is trained using the square error loss as the loss function; or, when the input of the variational autoencoder is a binary vector, the variational autoencoder is trained using the maximum KL (Kullback-Leibler Divergence, relative entropy) loss as the loss function.
[0072] The network structures of the autoencoder and the variational autoencoder are very similar. The autoencoder has no special processing for the hidden layer, while the variational autoencoder has a redirection operation on the hidden layer, which can solve the problem of uncontrollable numerical distribution of the hidden layer of the autoencoder.
[0073] In an embodiment of the present invention, there are two cases where the data for the input document calculation model can be a document vector or a binary vector. For these two different types of input data, when the document calculation model is a variational autoencoder, different loss functions can be used in the process of training the variational autoencoder.
[0074] Specifically, document samples for training the variational autoencoder can be collected, and the document samples can be preprocessed to obtain the document vectors corresponding to the document samples. In the case of training the variational autoencoder according to the document vectors corresponding to the document samples, since it is hoped that the input of the variational autoencoder obtained by training is equal to the output, it is hoped that the squared difference error between the vector output by the variational autoencoder and the vector input by the variational autoencoder is as small as possible, and the squared error loss can be used as the loss function, and the document vectors corresponding to the document samples can be input into the initial variational autoencoder for learning and training to obtain a trained variational autoencoder.
[0075] Optionally, after obtaining the document vector corresponding to the document sample, the document vector corresponding to the document sample can also be binarized to obtain the binarized vector corresponding to the document sample, and the variational autoencoder can be trained based on the binarized vector. In this case, since it is hoped that the input of the variational autoencoder obtained by training is equal to the output, it is hoped that the KL distance between the vector output by the variational autoencoder and the vector input is as small as possible, the maximum KL loss can be used as the loss function, and the binarized vector corresponding to the document sample can be input into the initial variational autoencoder for learning and training to obtain a trained variational autoencoder.
[0076] Through the above two different types of input data and corresponding loss functions, a variational autoencoder (also known as a document calculation model) corresponding to the input data type can be trained.
[0077] Optionally, the document calculation model of the embodiment of the present invention may also use an autoencoder. In the case of using an autoencoder, the distribution of the hidden layer can be constrained by adding a loss function, such as adding a loss function for constraining the distribution of the hidden layer, to solve the problem of uncontrollable distribution of the hidden layer values of the autoencoder.
[0078] Similarly, when the document calculation model is an autoencoder, there are two cases where the input data is a document vector or a binary vector. For these two different types of input data, the training process of the autoencoder is similar to the training process of the variational autoencoder, and will not be repeated here. The difference is that a loss function for constraining the hidden layer distribution is added during the training of the autoencoder.
[0079] Specifically, when the input of the autoencoder is a document vector, in order to prevent the hidden layer numerical distribution of the autoencoder from being uncontrollable, it is necessary to limit the hidden layer distribution of the autoencoder to be as similar as possible to the normal distribution. Therefore, on the basis of using the square error loss as the loss function for training the autoencoder, a loss function for constraining the hidden layer distribution can also be added. For example, the maximum KL loss between the hidden layer distribution and the normal distribution can be calculated as a newly added loss function and added to the process of training the autoencoder.
[0080] Similarly, when the input of the autoencoder is a binary vector, in order to prevent the hidden layer numerical distribution of the autoencoder from being uncontrollable, it is necessary to limit the hidden layer distribution of the autoencoder to be as similar as possible to the binary distribution. Therefore, on the basis of using the maximum KL loss as the loss function, a loss function for constraining the hidden layer distribution can also be added. For example, the maximum KL loss between the hidden layer distribution and the binary distribution can be calculated as a new loss function and added to the process of training the autoencoder.
[0081] It can be understood that in the embodiment of the present invention, other more loss functions can be added to the deep neural network model, and multiple loss functions can be minimized simultaneously during the training process to further improve the accuracy of the document calculation model.
[0082] In summary, the embodiment of the present invention converts documents into document vectors. Compared with industrial-grade vocabulary, vectors not only occupy less memory, but also can improve the efficiency and accuracy of calculation, thereby improving the efficiency and accuracy of filtering duplicate documents. In addition, the embodiment of the present invention further converts the document vector into a document value through a document calculation model, and determines whether the document is a duplicate by comparing the similarity between the document values. That is, the embodiment of the present invention converts the document comparison process into a numerical comparison process, which can simplify the calculation process of document comparison and further improve the efficiency and accuracy of filtering duplicate documents. Furthermore, the document calculation model of the embodiment of the present invention is a deep neural network model trained based on collected document samples. The document calculation model trained based on a large amount of sample data can improve the accuracy of converting document vectors into document values, and can further ensure the accuracy of document comparison.
[0083] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0084] Device Embodiment
[0085] Reference Figure 2 , shows a structural block diagram of an embodiment of a data processing device of the present invention, and the device may specifically include:
[0086] The document processing module 201 is used to pre-process the document to obtain a document vector corresponding to the document;
[0087] A data input module 202, used to input the document vector into a document calculation model, wherein the document calculation model is a deep neural network model trained based on the collected document samples;
[0088] A value determination module 203, used to determine a document value corresponding to the document according to the output of the document calculation model;
[0089] The document comparison module 204 is used to determine whether there is a duplicate document in the at least two documents by calculating the similarity between the document values corresponding to the at least two documents.
[0090] Optionally, the device may further include:
[0091] A binarization module, used to perform binarization processing on the document vector corresponding to the document to obtain a binarized vector corresponding to the document;
[0092] The data input module is specifically used to input the binary vector corresponding to the document into the document calculation model.
[0093] Optionally, the binarization module is specifically used to set the dimensional vector greater than a first preset value to a first value, and the dimensional vector less than the first preset value to a second value in each dimensional vector of the document vector corresponding to the document, so as to obtain a binarized vector corresponding to the document.
[0094] Optionally, the value determination module 203 may specifically include:
[0095] A data acquisition submodule, used to acquire hidden layer output data of the document calculation model;
[0096] The data calculation submodule is used to perform binarization processing on the hidden layer output data to obtain the document value corresponding to the document.
[0097] Optionally, the document calculation model includes multiple hidden layers, and the data acquisition submodule is specifically used to obtain the output data of the hidden layer closest to the output layer.
[0098] Optionally, the document comparison module 204 may specifically include:
[0099] A first comparison submodule is configured to determine that the at least two first documents are duplicate documents if, in the case where the document value is an integer value, there are at least two first documents with the same integer value among the integer values corresponding to the at least two documents; or
[0100] The second comparison submodule is used to determine that the at least two second documents are duplicate documents if, in the case where the document value is a binary value, there are at least two second documents whose binary value difference bits are less than a second preset value among the binary values corresponding to the at least two documents.
[0101] Optionally, the document computation model may include a variational autoencoder or an autoencoder.
[0102] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0103] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0104] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0105] An embodiment of the present invention provides a device for data processing, including a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the following operations: preprocessing a document to obtain a document vector corresponding to the document; inputting the document vector into a document calculation model, wherein the document calculation model is a deep neural network model trained based on collected document samples; determining a document value corresponding to the document based on an output of the document calculation model; and determining whether there are duplicate documents in the at least two documents by calculating the similarity between the document values corresponding to at least two documents.
[0106] Figure 3 8 is a block diagram of a data processing apparatus 800 according to an exemplary embodiment. For example, the apparatus 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0107] Reference Figure 3 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .
[0108] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.
[0109] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0110] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.
[0111] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0112] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a voice information processing mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.
[0113] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.
[0114] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0115] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency information processing (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0116] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.
[0117] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0118] Figure 4 1 is a schematic diagram of the structure of the server in some embodiments of the present invention. The server 1900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1922 (for example, one or more processors) and memory 1932, and one or more storage media 1930 (for example, one or more mass storage devices) storing application programs 1942 or data 1944. Among them, the memory 1932 and the storage medium 1930 can be short-term storage or permanent storage. The program stored in the storage medium 1930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 1922 can be configured to communicate with the storage medium 1930 and execute a series of instruction operations in the storage medium 1930 on the server 1900.
[0119] The server 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input and output interfaces 1958, one or more keyboards 1956, and / or, one or more operating systems 1941, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
[0120] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a device (server or terminal), enables the device to perform Figure 1 The data processing method shown.
[0121] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a device (server or terminal), enables the device to execute a data processing method, the method comprising: preprocessing a document to obtain a document vector corresponding to the document; inputting the document vector into a document calculation model, the document calculation model being a deep neural network model trained based on collected document samples; determining a document value corresponding to the document based on an output of the document calculation model; and determining whether there are duplicate documents in the at least two documents by calculating the similarity between the document values corresponding to at least two documents.
[0122] The embodiment of the present invention discloses A1, a data processing method, including:
[0123] Preprocessing the document to obtain a document vector corresponding to the document;
[0124] Inputting the document vector into a document calculation model, wherein the document calculation model is a deep neural network model trained based on the collected document samples;
[0125] Determining a document value corresponding to the document according to an output of the document calculation model;
[0126] By calculating the similarity between the document values corresponding to at least two documents, it is determined whether there is a duplicate document in the at least two documents.
[0127] A2. According to the method described in A1, after the step of preprocessing the document to obtain the document vector corresponding to the document, the method further comprises:
[0128] Binarizing the document vector corresponding to the document to obtain a binarized vector corresponding to the document;
[0129] The step of inputting the document vector into a document calculation model comprises:
[0130] The binary vector corresponding to the document is input into the document calculation model.
[0131] A3. According to the method described in A2, binarizing the document vector corresponding to the document to obtain the binarized vector corresponding to the document includes:
[0132] In each dimensional vector of the document vector corresponding to the document, a dimensional vector greater than a first preset value is set to a first value, and a dimensional vector less than the first preset value is set to a second value, to obtain a binary vector corresponding to the document.
[0133] A4. The method according to A1, wherein determining the document value corresponding to the document according to the output of the document calculation model comprises:
[0134] Obtaining hidden layer output data of the document calculation model;
[0135] Binarization is performed on the hidden layer output data to obtain a document value corresponding to the document.
[0136] A5. The method according to A4, wherein the document computing model includes a plurality of hidden layers, and obtaining hidden layer output data of the document computing model includes:
[0137] Get the hidden layer output data closest to the output layer.
[0138] A6. The method according to A1, wherein determining whether there is a duplicate document in the at least two documents by calculating the similarity between the document values corresponding to the at least two documents respectively comprises:
[0139] In the case where the document value is an integer value, if there are at least two first documents with the same integer value among the integer values corresponding to the at least two documents, then the at least two first documents are determined to be duplicate documents; or
[0140] In the case where the document value is a binary value, if there are at least two second documents whose binary value difference bits are smaller than a second preset value among the binary values corresponding to the at least two documents, the at least two second documents are determined to be duplicate documents.
[0141] A7. According to any of the methods described in A1 to A6, the document calculation model includes a variational autoencoder or an autoencoder.
[0142] The embodiment of the present invention discloses B8, a data processing device, including:
[0143] A document processing module, used to pre-process the document to obtain a document vector corresponding to the document;
[0144] A data input module, used to input the document vector into a document calculation model, wherein the document calculation model is a deep neural network model trained based on the collected document samples;
[0145] A value determination module, used to determine a document value corresponding to the document according to the output of the document calculation model;
[0146] The document comparison module is used to determine whether there is a duplicate document in the at least two documents by calculating the similarity between the document values corresponding to the at least two documents.
[0147] B9. The device according to B8, further comprising:
[0148] A binarization module, used to perform binarization processing on the document vector corresponding to the document to obtain a binarized vector corresponding to the document;
[0149] The data input module is specifically used to input the binary vector corresponding to the document into the document calculation model.
[0150] B10. According to the device described in B9, the binarization module is specifically used to set the dimensional vector greater than the first preset value to the first value and the dimensional vector less than the first preset value to the second value in each dimensional vector of the document vector corresponding to the document, so as to obtain the binarized vector corresponding to the document.
[0151] B11. According to the device described in B8, the value determination module comprises:
[0152] A data acquisition submodule, used to acquire hidden layer output data of the document calculation model;
[0153] The data calculation submodule is used to perform binarization processing on the hidden layer output data to obtain the document value corresponding to the document.
[0154] B12. According to the device described in B11, the document calculation model includes multiple hidden layers, and the data acquisition submodule is specifically used to obtain the hidden layer output data closest to the output layer.
[0155] B13. According to the device described in B8, the document comparison module comprises:
[0156] A first comparison submodule is configured to determine that the at least two first documents are duplicate documents if, in the case where the document value is an integer value, there are at least two first documents with the same integer value among the integer values corresponding to the at least two documents; or
[0157] The second comparison submodule is used to determine that the at least two second documents are duplicate documents if, in the case where the document value is a binary value, there are at least two second documents whose binary value difference bits are less than a second preset value among the binary values corresponding to the at least two documents.
[0158] B14. According to any one of the devices described in B8 to B13, the document computing model includes a variational autoencoder or an autoencoder.
[0159] The embodiment of the present invention discloses C15, a device for data processing, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the following operations:
[0160] Preprocessing the document to obtain a document vector corresponding to the document;
[0161] Inputting the document vector into a document calculation model, wherein the document calculation model is a deep neural network model trained based on the collected document samples;
[0162] Determining a document value corresponding to the document according to an output of the document calculation model;
[0163] By calculating the similarity between the document values corresponding to at least two documents, it is determined whether there is a duplicate document in the at least two documents.
[0164] C16. The device according to C15, wherein the device is further configured to execute, by one or more processors, the one or more programs including instructions for performing the following operations:
[0165] Binarizing the document vector corresponding to the document to obtain a binarized vector corresponding to the document;
[0166] The step of inputting the document vector into a document calculation model comprises:
[0167] The binary vector corresponding to the document is input into the document calculation model.
[0168] C17. According to the apparatus described in C16, the step of binarizing the document vector corresponding to the document to obtain the binarized vector corresponding to the document comprises:
[0169] In each dimensional vector of the document vector corresponding to the document, a dimensional vector greater than a first preset value is set to a first value, and a dimensional vector less than the first preset value is set to a second value, to obtain a binary vector corresponding to the document.
[0170] C18. According to the apparatus described in C15, determining the document value corresponding to the document according to the output of the document calculation model includes:
[0171] Obtaining hidden layer output data of the document calculation model;
[0172] Binarization is performed on the hidden layer output data to obtain a document value corresponding to the document.
[0173] C19. According to the apparatus of C18, the document computing model includes a plurality of hidden layers, and the step of obtaining hidden layer output data of the document computing model includes:
[0174] Get the hidden layer output data closest to the output layer.
[0175] C20. The apparatus according to C15, wherein determining whether there is a duplicate document in the at least two documents by calculating the similarity between the document values corresponding to the at least two documents respectively comprises:
[0176] In the case where the document value is an integer value, if there are at least two first documents with the same integer value among the integer values corresponding to the at least two documents, then the at least two first documents are determined to be duplicate documents; or
[0177] In the case where the document value is a binary value, if there are at least two second documents whose binary value difference bits are smaller than a second preset value among the binary values corresponding to the at least two documents, the at least two second documents are determined to be duplicate documents.
[0178] C21. According to any one of the devices described in C15 to C20, the document computing model includes a variational autoencoder or an autoencoder.
[0179] An embodiment of the present invention discloses D22, a machine-readable medium having instructions stored thereon, which, when executed by one or more processors, enables the device to execute the data processing method as described in one or more of A1 to A7.
[0180] Those skilled in the art will readily appreciate other embodiments of the invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.
[0181] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
[0182] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
[0183] The data processing method, the data processing device and the device for data processing provided by the present invention are introduced in detail above. The principle and implementation mode of the present invention are explained by using specific examples in this article. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation mode and the scope of application. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. A data processing method, characterized in that: The method comprises: Preprocessing the document to obtain a document vector corresponding to the document, including: performing word segmentation processing on the document, and filtering the word segments obtained by the word segmentation processing to obtain a word segmentation sequence; obtaining the word segmentation vector based on the word segmentation sequence, and performing weighted summation on the word segmentation vectors to determine the document vector; Binarizing the document vector corresponding to the document to obtain a binarized vector corresponding to the document; Inputting the binarized vector corresponding to the document into a document calculation model, wherein the document calculation model is a deep neural network model trained based on the collected document samples; Determining a document value corresponding to the document according to an output of the document calculation model; By calculating the similarity between the document values corresponding to at least two documents, it is determined whether there is a duplicate document in the at least two documents.
2. The method according to claim 1, characterized in that The binarization of the document vector corresponding to the document to obtain the binarized vector corresponding to the document includes: In each dimensional vector of the document vector corresponding to the document, a dimensional vector greater than a first preset value is set to a first value, and a dimensional vector less than the first preset value is set to a second value, to obtain a binary vector corresponding to the document.
3. The method according to claim 1, characterized in that: Determining the document value corresponding to the document according to the output of the document calculation model includes: Obtaining hidden layer output data of the document calculation model; Binarization is performed on the hidden layer output data to obtain a document value corresponding to the document.
4. The method according to claim 3, characterized in that The document computing model includes a plurality of hidden layers, and obtaining hidden layer output data of the document computing model includes: Get the hidden layer output data closest to the output layer.
5. The method according to claim 1, characterized in that The step of calculating the similarity between the document values corresponding to the at least two documents to determine whether there is a duplicate document in the at least two documents includes: In the case where the document value is an integer value, if there are at least two first documents with the same integer value among the integer values corresponding to the at least two documents, then the at least two first documents are determined to be duplicate documents; or In the case where the document value is a binary value, if there are at least two second documents whose binary value difference bits are smaller than a second preset value among the binary values corresponding to the at least two documents, the at least two second documents are determined to be duplicate documents.
6. The method according to any one of claims 1 to 5, characterized in that: The document computation model includes a variational autoencoder or an autoencoder.
7. A data processing device, characterized in that: The device comprises: The document processing module is used to pre-process the document to obtain the document vector corresponding to the document, which includes: performing word segmentation processing on the document, and filtering the word segments obtained by the word segmentation processing to obtain a word segmentation sequence; obtaining the word segmentation vector based on the word segmentation sequence, and performing weighted summation on the word segmentation vectors to determine the document vector; A binarization module, used to perform binarization processing on the document vector corresponding to the document to obtain a binarized vector corresponding to the document; A data input module, used to input the binary vector corresponding to the document into a document calculation model, wherein the document calculation model is a deep neural network model trained according to the collected document samples; A value determination module, used to determine a document value corresponding to the document according to the output of the document calculation model; The document comparison module is used to determine whether there is a duplicate document in the at least two documents by calculating the similarity between the document values corresponding to the at least two documents.
8. The device according to claim 7, characterized in that The binarization module is specifically used to set the dimensional vector greater than the first preset value to the first value and the dimensional vector less than the first preset value to the second value in each dimensional vector of the document vector corresponding to the document, so as to obtain the binarized vector corresponding to the document.
9. The device according to claim 7, characterized in that The numerical value determination module comprises: A data acquisition submodule, used to acquire hidden layer output data of the document calculation model; The data calculation submodule is used to perform binarization processing on the hidden layer output data to obtain the document value corresponding to the document.
10. The device according to claim 9, characterized in that The document calculation model includes multiple hidden layers, and the data acquisition submodule is specifically used to obtain the hidden layer output data closest to the output layer.
11. The device according to claim 7, characterized in that The document comparison module comprises: A first comparison submodule is configured to determine that the at least two first documents are duplicate documents if, in the case where the document value is an integer value, there are at least two first documents with the same integer value among the integer values corresponding to the at least two documents; or The second comparison submodule is used to determine that the at least two second documents are duplicate documents if, in the case where the document value is a binary value, there are at least two second documents whose binary value difference bits are less than a second preset value among the binary values corresponding to the at least two documents.
12. The device according to any one of claims 7 to 11, characterized in that The document computation model includes a variational autoencoder or an autoencoder.
13. A device for data processing, characterized in that: The device comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs contain instructions for performing the data processing method according to any one of claims 1 to 6.
14. A machine-readable medium having instructions stored thereon, which, when executed by one or more processors, causes an apparatus to execute the data processing method according to any one of claims 1 to 6.
Citation Information
Patent Citations
Short text clustering method based on deep semantic feature learning
CN104915386A
Text similarity determining method and device, electronic equipment and system
CN105095162A
Repeated text judgment method and apparatus
CN106528508A
Constructing method for unsupervised learning unified characteristic extractor
CN108304359A
Chinese patent text similarity calculation method
CN108549634A
