File-based key data extraction method and device, equipment and medium
By introducing current and historical word popularity matrices into the document extraction model and processing document vectors to output key data, the problem of insufficient extraction accuracy in existing technologies is solved, and more accurate and comprehensive key data extraction is achieved.
Patent Information
- Application Number
- CN202411056748.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-08
AI Technical Summary
Existing methods for extracting keywords and summaries from documents cannot accurately capture changes in readers' real-time interests, resulting in insufficient extraction accuracy.
By obtaining the file vector of the current file and inputting it into the trained key data extraction model, and combining the word popularity matrix of the current file and historical files, the first intermediate vector is processed to output key data, taking into account word popularity changes and historical word frequency information.
It improves the accuracy and comprehensiveness of key data, avoids inaccuracies caused by information loss, and enables real-time and accurate extraction of key data.
Smart Images

Figure CN118916412A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of data processing technology, and in particular to a file-based key data extraction method, device, equipment and medium. Background Art
[0002] At present, the extraction of keywords and abstracts of documents is more based on the content of the reference document, and the relevant extraction is based on the content. However, the interests of readers and social hot spots are changing in real time, and the unchanging extraction rules cannot accurately grasp the reading interests of readers. Therefore, how to improve the accuracy of extracting key data in documents is crucial. Summary of the invention
[0003] The present invention provides a file-based key data extraction method, device, equipment and medium to improve the accuracy of extracting key data from a file.
[0004] According to one aspect of the present invention, a method for extracting key data based on a file is provided, comprising:
[0005] Obtaining a file vector of a current file to be extracted, and inputting the file vector into a trained key data extraction model to obtain a first intermediate vector;
[0006] Based on the current word heat matrix corresponding to the current file, the first intermediate vector is processed to obtain a second intermediate vector; wherein the current word heat matrix is determined according to the current word and the corresponding current word frequency in the current file, and the historical word and the corresponding historical word frequency in the historical file;
[0007] Outputting key data of the current file according to the second intermediate vector and the file vector.
[0008] According to another aspect of the present invention, there is provided a file-based key data extraction device, comprising:
[0009] A first intermediate vector determination module, used to obtain a file vector of a current file to be extracted, and input the file vector into a trained key data extraction model to obtain a first intermediate vector;
[0010] A second intermediate vector determination module, configured to process the first intermediate vector based on a current word heat matrix corresponding to the current file to obtain a second intermediate vector; wherein the current word heat matrix is determined according to the current word and the corresponding current word frequency in the current file, and the historical word and the corresponding historical word frequency in the historical file;
[0011] A key data output module is used to output the key data of the current file according to the second intermediate vector and the file vector.
[0012] According to another aspect of the present invention, there is provided an electronic device, comprising:
[0013] one or more processors;
[0014] A memory for storing one or more programs;
[0015] When one or more programs are executed by one or more processors, the one or more processors can execute any file-based key data extraction method provided by the embodiments of the present invention.
[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions, and the computer instructions are used to enable a processor to implement any file-based key data extraction method provided by an embodiment of the present invention when executed.
[0017] The embodiment of the present invention provides a key data extraction scheme based on files, which obtains the file vector of the current file to be extracted and inputs the file vector into a trained key data extraction model to obtain a first intermediate vector; based on the current word heat matrix corresponding to the current file, the first intermediate vector is processed to obtain a second intermediate vector; wherein the current word heat matrix is determined according to the current words and the corresponding current word frequency in the current file, and the historical words and the corresponding historical word frequency in the historical file; according to the second intermediate vector and the file vector, the key data of the current file is output. The above scheme determines the key data by introducing the key data extraction model and the current word heat matrix, realizes the determination of the key data in the current file on the basis of considering the change of word heat, and improves the accuracy of the determined key data; at the same time, the current word heat matrix is determined according to the current words and the corresponding current word frequency in the current file, and the historical words and the corresponding historical word frequency in the historical file, and improves the accuracy of the determined current word heat matrix; and according to the second intermediate vector and the file vector, the key data of the current file is output, which avoids the inaccuracy of the determined key data caused by the second intermediate vector obtained by processing due to information loss, realizes the information supplement of the second intermediate vector, and improves the comprehensiveness of the determined key data.
[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0020] Figure 1A is a flow chart of a method for extracting key data based on a file provided in Embodiment 1 of the present invention;
[0021] Figure 1B is a flow chart of extracting key data by a key data extraction model provided in the first embodiment of the present invention;
[0022] Figure 2 is a flow chart of a file-based key data extraction method provided in Embodiment 2 of the present invention;
[0023] Figure 3 It is a structural schematic diagram of a file-based key data extraction device provided in Embodiment 3 of the present invention;
[0024] Figure 4 It is a structural schematic diagram of an electronic device for implementing a file-based key data extraction method provided by a fourth embodiment of the present invention. DETAILED DESCRIPTION
[0025] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.
[0026] Embodiment 1
[0027] Figure 1A It is a flowchart of a file-based key data extraction method provided in Example 1 of the present invention. This embodiment is applicable to the case of extracting key data from a file. The method can be executed by a file-based key data extraction device, which can be implemented in software and / or hardware and can be configured in an electronic device that carries the file-based key data extraction function.
[0028] See also Figure 1A The file-based key data extraction method shown includes:
[0029] S110 , obtaining a file vector of a current file to be extracted, and inputting the file vector into a trained key data extraction model to obtain a first intermediate vector.
[0030] The current file refers to a file that currently needs to extract key data. The embodiment of the present invention does not limit the type of the current file, which can be set by a technician based on experience. For example, the current file can be a text file.
[0031] The file vector refers to a vector determined based on the current file. Exemplarily, if the current file is a text file, the current file is vectorized, and the corresponding file vector is a text vector.
[0032] Among them, the key data extraction model can be used to extract key data from the current file. Key data refers to the most important data in the file. Exemplarily, the key data can include keywords and abstracts.
[0033] The first intermediate vector refers to an intermediate vector obtained after encoding the file vector. Exemplarily, after the file vector is input into the key data extraction model, it is encoded by the encoder to obtain the first intermediate vector.
[0034] Specifically, a file vector of a current file to be extracted is obtained, and the file vector is input into a trained key data extraction model, and the file vector is encoded by an encoder in the key data extraction model to obtain a first intermediate vector.
[0035] S120: Based on the current vocabulary heat matrix corresponding to the current file, the first intermediate vector is processed to obtain a second intermediate vector.
[0036] The current word heat matrix refers to a matrix used to record the usage degree of each current word. Exemplarily, the current word heat matrix is determined according to the current words and the corresponding current word frequencies in the current file, and the historical words and the corresponding historical word frequencies in the historical file.
[0037] The current word refers to a word extracted from the current file. The current word frequency can be used to quantify the importance of the current word in the current file. Exemplarily, the current file is segmented to obtain multiple current words, and the current word frequency corresponding to each current word is determined.
[0038] The historical file refers to a previous file. Exemplarily, the historical file may be a text file. The historical vocabulary refers to words extracted from the historical file. The historical word frequency may be used to quantify the importance of the historical vocabulary in the historical file set. The historical file set includes multiple historical files. Exemplarily, the historical file is segmented to obtain multiple historical vocabulary, and the historical word frequency corresponding to each historical vocabulary is determined.
[0039] The second intermediate vector refers to an intermediate vector obtained by processing the first intermediate vector based on the current vocabulary heat matrix.
[0040] S130. Output key data of the current file according to the second intermediate vector and the file vector.
[0041] Specifically, the key data extraction model obtains the key data vector based on the second intermediate vector and the file vector, decodes the key data vector through a decoder, and outputs the key data of the current file.
[0042] The embodiment of the present invention provides a key data extraction scheme based on files, which obtains the file vector of the current file to be extracted and inputs the file vector into a trained key data extraction model to obtain a first intermediate vector; based on the current word heat matrix corresponding to the current file, the first intermediate vector is processed to obtain a second intermediate vector; wherein the current word heat matrix is determined according to the current words and the corresponding current word frequency in the current file, and the historical words and the corresponding historical word frequency in the historical file; according to the second intermediate vector and the file vector, the key data of the current file is output. The above scheme determines the key data by introducing the key data extraction model and the current word heat matrix, realizes the determination of the key data in the current file on the basis of considering the change of word heat, and improves the accuracy of the determined key data; at the same time, the current word heat matrix is determined according to the current words and the corresponding current word frequency in the current file, and the historical words and the corresponding historical word frequency in the historical file, and improves the accuracy of the determined current word heat matrix; and according to the second intermediate vector and the file vector, the key data of the current file is output, which avoids the inaccuracy of the determined key data caused by the second intermediate vector obtained by processing due to information loss, realizes the information supplement of the second intermediate vector, and improves the comprehensiveness of the determined key data.
[0043] On the basis of the above technical solution, if the current file is a non-initial file in the file set to be extracted, the first intermediate vector is processed based on the current vocabulary heat matrix corresponding to the current file to obtain the second intermediate vector, including: combining the second intermediate vector of the previous file with the first intermediate vector of the current file to obtain a new first intermediate vector of the current file; based on the current vocabulary heat matrix corresponding to the current file, the new first intermediate vector is processed to obtain the second intermediate vector of the current file.
[0044] The file set to be extracted includes at least two current files. The initial file refers to the first current file in the file set for key data extraction. The previous file refers to the file in the file set to be extracted for key data extraction before the current file. The new first intermediate vector refers to the intermediate vector obtained by multiplying the second intermediate vector of the previous file by the first intermediate vector of the current file.
[0045] It can be understood that by combining the second intermediate vector of the previous file with the first intermediate vector of the current file to obtain a new first intermediate vector of the current file, and processing the new first intermediate vector to obtain the second intermediate vector, the real-time performance of the determined new first intermediate vector is improved, thereby improving the accuracy of the key data determined subsequently.
[0046] For example, see Figure 1B The flowchart of extracting key data from the key data extraction model shown in the figure. Wherein, x is the file vector of the current file input; y is the key data vector; B represents the first intermediate vector obtained by the file vector x through the encoder; A represents the current vocabulary heat matrix of the current file, which is updated according to the current file; h represents the first intermediate vector; C represents the second intermediate vector, which is assigned to h in the next operation; D represents the file vector; finally, the second intermediate vector C is added to the file vector D to obtain the final key data vector y; the key data vector y is decoded by the decoder to obtain the key data Y.
[0047] Exemplarily, the input of the key data extraction model may also be the current file. The model internally performs vector conversion on the current file to obtain a file vector, and then uses the above-mentioned method for extracting key data to obtain the key data.
[0048] Embodiment 2
[0049] Figure 2 It is a flowchart of the file-based key data extraction method provided in the second embodiment of the present invention. Based on the above embodiments, this embodiment further adds "obtaining historical files within the historical time period, and processing the current file and the historical file respectively to obtain current file associated data and historical file associated data; wherein, the current file associated data includes the current vocabulary and the corresponding current word frequency; the historical file associated data includes the historical vocabulary and the corresponding historical word frequency; selecting a reference vocabulary that matches the current vocabulary from the historical vocabulary, and using the historical word frequency corresponding to the reference vocabulary as the reference word frequency; determining the current vocabulary heat of the current vocabulary according to the current word frequency, the reference word frequency and the historical time period; generating a current vocabulary heat matrix including the current vocabulary heat" to improve the determination mechanism of the current vocabulary heat matrix. It should be noted that for the parts not described in detail in the embodiments of the present invention, reference can be made to the descriptions of other embodiments.
[0050] See also Figure 2 The file-based key data extraction method shown includes:
[0051] S210, obtaining a current file to be extracted and historical files within a historical period, and processing the current file and historical files respectively to obtain current file associated data and historical file associated data; wherein the current file associated data includes current vocabulary and corresponding current word frequency; the historical file associated data includes historical vocabulary and corresponding historical word frequency.
[0052] The historical period refers to the time period for obtaining historical files. The current file-related data refers to the data associated with the current file. The historical file-related data refers to the data associated with the historical files.
[0053] For example, based on the time of extracting key data in the current file, historical files within the historical period are obtained, such as obtaining files within the previous month as historical files.
[0054] S220: Select a reference word that matches the current word from the historical words, and use the historical word frequency corresponding to the reference word as the reference word frequency.
[0055] The reference vocabulary refers to the historical vocabulary that matches the current vocabulary, and the reference word frequency refers to the historical word frequency corresponding to the reference vocabulary.
[0056] S230: Determine the current word popularity of the current word according to the current word frequency, the reference word frequency and the historical time period.
[0057] Among them, the current word heat can be used to quantify the usage degree of the current word.
[0058] In an optional embodiment, the current word heat of the current word is determined based on the current word frequency, reference word frequency and historical time period, including: determining the current word frequency sum of the current word based on the current word frequency of the current word and the reference word frequency of the corresponding reference word; determining the number of words of the current word in the current file, and determining the average word frequency sum based on the number of words and the current word frequency sum of each current word; determining the word frequency proportion of the current word frequency of the current word in the corresponding current word frequency sum; determining the current word heat of the current word based on the historical time period, the average word frequency sum, the word frequency proportion of the current word, the current word frequency and the reference word frequency.
[0059] The current word frequency sum can be used to quantify the importance of the current word in the current file and the historical file. Exemplarily, for any current word, the current word frequency of the current word is added to the reference word frequency of the reference word corresponding to the current word to obtain the current word frequency sum of the current word.
[0060] The word count refers to the number of current words in the current file. For example, if the current words in the current file include word M, word N, word L, and word G, the word count of the current words in the current file is 4.
[0061] The average word frequency sum refers to the average of the current word frequency sums of each current word. Exemplarily, the current word frequency sums of each current word are added together to obtain the word frequency sum value; and the ratio between the word frequency sum value and the number of words is used as the average word frequency sum.
[0062] The frequency ratio can be used to quantify the importance of the current frequency in the current frequency sum. For example, for any current word, the ratio between the current frequency of the current word and the current frequency sum of the current word is used as the frequency ratio of the current word.
[0063] It can be understood that by determining the current vocabulary heat of the current vocabulary based on the historical period, the average word frequency, the word frequency proportion of the current vocabulary, the current word frequency and the reference word frequency, the accuracy of the determined current vocabulary heat is improved.
[0064] In an optional embodiment, the current word heat of the current word is determined based on the historical time period, the average word frequency, the word frequency proportion of the current word, the current word frequency and the reference word frequency, including: determining the initial word heat based on the average word frequency, the word frequency proportion of the current word, the current word frequency and the reference word frequency; determining the word frequency cooling coefficient of the current word based on the historical time period, the current word frequency of the current word and the reference word frequency; determining the current word heat of the current word based on the initial word heat and the word frequency cooling coefficient.
[0065] The initial word heat refers to the word heat of the current word determined for the first time. The word frequency cooling coefficient refers to the word heat cooling value of the current word in the historical period.
[0066] Exemplarily, the initial word heat can be determined by the following formula:
[0067]
[0068] Among them, B_Hot(w) represents the initial word heat; Atp(w) represents the current word frequency; Tp(avg) represents the average word frequency; R(w) represents the word frequency ratio; Btp(w) represents the reference word frequency of the reference word corresponding to the current word; w represents the current word.
[0069] Exemplarily, the word frequency cooling coefficient may be determined in the following manner:
[0070]
[0071] in, represents the word frequency cooling coefficient; t represents the time difference, that is, the historical period.
[0072] It can be understood that by determining the current vocabulary heat of the current vocabulary based on the initial vocabulary heat and the word frequency cooling coefficient, it is possible to determine the current vocabulary heat on the basis of taking into account the cooling problem of the vocabulary heat, thereby improving the accuracy of the determined current vocabulary heat.
[0073] In an optional embodiment, the current vocabulary heat of the current vocabulary is determined according to the initial vocabulary heat and the word frequency cooling coefficient, including: determining an initial weight of the initial vocabulary heat and determining a coefficient weight of the word frequency cooling coefficient; determining the current vocabulary heat of the current vocabulary according to the initial weight, the coefficient weight, the initial vocabulary heat and the word frequency cooling coefficient.
[0074] Among them, the initial weight can be used to quantify the importance of the initial word heat in the current word heat. The coefficient weight can be used to quantify the importance of the word frequency cooling coefficient in the current word heat. The embodiment of the present invention does not impose any restrictions on the size of the initial weight and the coefficient weight, which can be set by technicians based on experience, and it only needs to ensure that the sum of the initial weight and the coefficient weight is 1.
[0075] For example, the current word heat can be determined by the following formula:
[0076]
[0077] Among them, H(w) represents the current word heat; α represents the initial weight; β represents the coefficient weight.
[0078] It can be understood that by introducing the initial weight and the coefficient weight, the current vocabulary heat of the current vocabulary is determined according to the initial weight, the coefficient weight, the initial vocabulary heat and the word frequency cooling coefficient, thereby improving the accuracy of the determined current vocabulary heat.
[0079] S240: Generate a current word heat matrix including current word heat.
[0080] Specifically, a current word heat matrix of the current file is generated according to the current word heat of each current word.
[0081] S250 , inputting the file vector of the current file into the trained key data extraction model to obtain a first intermediate vector.
[0082] S260: Based on the current vocabulary heat matrix corresponding to the current file, the first intermediate vector is processed to obtain a second intermediate vector.
[0083] The current word heat matrix is determined according to the current words and the corresponding current word frequencies in the current file, and the historical words and the corresponding historical word frequencies in the historical file.
[0084] S270: Output key data of the current file according to the second intermediate vector and the file vector.
[0085] The embodiment of the present invention obtains historical files within the historical time period by adding and processing the current file and the historical file respectively, thereby obtaining current file associated data and historical file associated data; wherein the current file associated data includes the current vocabulary and the corresponding current word frequency; the historical file associated data includes the historical vocabulary and the corresponding historical word frequency; a reference vocabulary matching the current vocabulary is selected from the historical vocabulary, and the historical word frequency corresponding to the reference vocabulary is used as the reference word frequency; the current vocabulary heat of the current vocabulary is determined according to the current word frequency, the reference word frequency and the historical time period; a current vocabulary heat matrix including the current vocabulary heat is generated, thereby improving the determination mechanism of the current vocabulary heat matrix. The above scheme improves the accuracy of the determined current vocabulary heat by determining the current vocabulary heat of the current vocabulary according to the current word frequency, the reference word frequency and the historical time period.
[0086] Embodiment 3
[0087] Figure 3 This is a schematic diagram of the structure of a file-based key data extraction device provided in Embodiment 3 of the present invention. This embodiment is applicable to the case of extracting key data from a file, and the method can be performed by a file-based key data extraction device, which can be implemented in software and / or hardware, and can be configured in an electronic device that carries a file-based key data extraction function.
[0088] like Figure 3 As shown, the device includes: a first intermediate vector determination module 310, a second intermediate vector determination module 320 and a key data output module 330. Among them,
[0089] A first intermediate vector determination module 310 is used to obtain a file vector of a current file to be extracted, and input the file vector into a trained key data extraction model to obtain a first intermediate vector;
[0090] A second intermediate vector determination module 320 is used to process the first intermediate vector based on the current word heat matrix corresponding to the current file to obtain a second intermediate vector; wherein the current word heat matrix is determined according to the current word and the corresponding current word frequency in the current file, and the historical word and the corresponding historical word frequency in the historical file;
[0091] The key data output module 330 is used to output the key data of the current file according to the second intermediate vector and the file vector.
[0092] The embodiment of the present invention provides a key data extraction scheme based on files, which obtains the file vector of the current file to be extracted and inputs the file vector into a trained key data extraction model to obtain a first intermediate vector; based on the current word heat matrix corresponding to the current file, the first intermediate vector is processed to obtain a second intermediate vector; wherein the current word heat matrix is determined according to the current words and the corresponding current word frequency in the current file, and the historical words and the corresponding historical word frequency in the historical file; according to the second intermediate vector and the file vector, the key data of the current file is output. The above scheme determines the key data by introducing the key data extraction model and the current word heat matrix, realizes the determination of the key data in the current file on the basis of considering the change of word heat, and improves the accuracy of the determined key data; at the same time, the current word heat matrix is determined according to the current words and the corresponding current word frequency in the current file, and the historical words and the corresponding historical word frequency in the historical file, and improves the accuracy of the determined current word heat matrix; and according to the second intermediate vector and the file vector, the key data of the current file is output, which avoids the inaccuracy of the determined key data caused by the second intermediate vector obtained by processing due to information loss, realizes the information supplement of the second intermediate vector, and improves the comprehensiveness of the determined key data.
[0093] Optionally, the current vocabulary heat matrix is determined based on the following means:
[0094] The associated data determination module is used to obtain historical files within a historical period, and process the current file and the historical file respectively to obtain current file associated data and historical file associated data; wherein the current file associated data includes current vocabulary and corresponding current word frequency; the historical file associated data includes historical vocabulary and corresponding historical word frequency;
[0095] A reference word frequency determination module, used to select a reference word matching the current word from the historical words, and use the historical word frequency corresponding to the reference word as the reference word frequency;
[0096] A vocabulary heat determination module, used to determine the current vocabulary heat of the current vocabulary according to the current vocabulary frequency, the reference vocabulary frequency and the historical period;
[0097] The heat matrix generation module is used to generate a current word heat matrix including the current word heat.
[0098] Optional, vocabulary heat determination module, including:
[0099] A word frequency sum determination unit, configured to determine the current word frequency sum of the current vocabulary according to the current word frequency of the current vocabulary and the reference word frequency of the corresponding reference vocabulary;
[0100] An average word frequency sum determination unit, used to determine the word quantity of the current word in the current file, and determine the average word frequency sum according to the word quantity and the current word frequency sum of each of the current words;
[0101] A word frequency ratio determination unit, used to determine the word frequency ratio of the current word frequency of the current vocabulary in the corresponding current word frequency sum;
[0102] The word heat determination unit is used to determine the current word heat of the current word according to the historical period, the average word frequency, the word frequency proportion of the current word, the current word frequency and the reference word frequency.
[0103] Optionally, a vocabulary heat determination unit includes:
[0104] An initial heat determination subunit, configured to determine an initial word heat according to the average word frequency sum, the word frequency proportion of the current word, the current word frequency and the reference word frequency;
[0105] A cooling coefficient determination subunit, configured to determine a word frequency cooling coefficient of the current word according to the historical period, the current word frequency of the current word and the reference word frequency;
[0106] The current heat determination subunit is used to determine the current word heat of the current word according to the initial word heat and the word frequency cooling coefficient.
[0107] Optionally, the current heat determination subunit is specifically used to:
[0108] Determining an initial weight of the initial vocabulary heat, and determining a coefficient weight of the word frequency cooling coefficient;
[0109] The current word heat of the current word is determined according to the initial weight, the coefficient weight, the initial word heat and the word frequency cooling coefficient.
[0110] Optionally, if the current file is a non-initial file in the file set to be extracted, the second intermediate vector determination module 320 includes:
[0111] A new first intermediate vector determining unit, configured to combine the second intermediate vector of the previous file with the first intermediate vector of the current file to obtain a new first intermediate vector of the current file;
[0112] The second intermediate vector determining unit is used to process the new first intermediate vector based on the current vocabulary heat matrix corresponding to the current file to obtain the second intermediate vector of the current file.
[0113] The file-based key data extraction device provided in the embodiment of the present invention can execute the file-based key data extraction method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing each file-based key data extraction method.
[0114] In the technical solution of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of current documents and historical documents involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0115] Embodiment 4
[0116] Figure 4 It is a structural schematic diagram of an electronic device for implementing a file-based key data extraction method provided by Embodiment 4 of the present invention. Electronic device 410 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0117] like Figure 4 As shown, the electronic device 410 includes at least one processor 411, and a memory connected to the at least one processor 411 in communication, such as a read-only memory (ROM) 412, a random access memory (RAM) 413, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 411 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 412 or the computer program loaded from the storage unit 418 to the random access memory (RAM) 413. In the RAM 413, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 411, the ROM 412, and the RAM 413 are connected to each other via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.
[0118] Multiple components in the electronic device 410 are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, a mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a disk, an optical disk, etc.; and a communication unit 419, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 419 allows the electronic device 410 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0119] Processor 411 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. Processor 411 executes the various methods and processes described above, such as the metadata processing method in HDFS.
[0120] In some embodiments, the metadata processing method in HDFS may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 410 via the ROM 412 and / or the communication unit 419. When the computer program is loaded into the RAM 413 and executed by the processor 411, one or more steps of the metadata processing method in HDFS described above may be performed. Alternatively, in other embodiments, the processor 411 may be configured to execute the metadata processing method in HDFS in any other appropriate manner (e.g., by means of firmware).
[0121] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0122] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0123] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0124] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0125] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0126] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0127] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0128] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A file-based key data extraction method, characterized in that: include: Obtaining a file vector of a current file to be extracted, and inputting the file vector into a trained key data extraction model to obtain a first intermediate vector; Based on the current word heat matrix corresponding to the current file, the first intermediate vector is processed to obtain a second intermediate vector; wherein the current word heat matrix is determined according to the current word and the corresponding current word frequency in the current file, and the historical word and the corresponding historical word frequency in the historical file; Output key data of the current file according to the second intermediate vector and the file vector.
2. The method according to claim 1, characterized in that The current word heat matrix is determined based on the following method: Acquire historical files within a historical period, and process the current file and the historical file respectively to obtain current file association data and historical file association data; wherein the current file association data includes current vocabulary and corresponding current word frequency; the historical file association data includes historical vocabulary and corresponding historical word frequency; Selecting a reference word matching the current word from the historical words, and using the historical word frequency corresponding to the reference word as the reference word frequency; Determining the current word heat of the current word according to the current word frequency, the reference word frequency and the historical period; A current word heat matrix including the current word heat is generated.
3. The method according to claim 2, characterized in that The determining the current word heat of the current word according to the current word frequency, the reference word frequency and the historical period includes: Determining the sum of the current word frequencies of the current vocabulary according to the current word frequency of the current vocabulary and the reference word frequency of the corresponding reference vocabulary; Determine the number of words of the current word in the current file, and determine the average word frequency sum according to the number of words and the current word frequency sum of each current word; Determine the proportion of the current word frequency of the current vocabulary in the corresponding current word frequency sum; The current word heat of the current word is determined according to the historical time period, the average word frequency, the word frequency proportion of the current word, the current word frequency and the reference word frequency.
4. The method according to claim 3, characterized in that The determining the current word heat of the current word according to the historical period, the average word frequency, the word frequency proportion of the current word, the current word frequency and the reference word frequency includes: Determine the initial vocabulary heat according to the average word frequency sum, the word frequency proportion of the current vocabulary, the current word frequency and the reference word frequency; Determining a word frequency cooling coefficient of the current word according to the historical period, the current word frequency of the current word and the reference word frequency; The current word heat of the current word is determined according to the initial word heat and the word frequency cooling coefficient.
5. The method according to claim 4, characterized in that The determining the current vocabulary heat of the current vocabulary according to the initial vocabulary heat and the word frequency cooling coefficient includes: Determining an initial weight of the initial vocabulary heat, and determining a coefficient weight of the word frequency cooling coefficient; The current word heat of the current word is determined according to the initial weight, the coefficient weight, the initial word heat and the word frequency cooling coefficient.
6. According to the method of any one of claims 1 to 5, if the current file is a non-initial file in the file set to be extracted, then the first intermediate vector is processed based on the current vocabulary heat matrix corresponding to the current file to obtain the second intermediate vector, comprising: Combining the second intermediate vector of the previous file with the first intermediate vector of the current file to obtain a new first intermediate vector of the current file; Based on the current vocabulary heat matrix corresponding to the current file, the new first intermediate vector is processed to obtain a second intermediate vector of the current file.
7. A key data extraction device based on files, characterized in that: include: A first intermediate vector determination module, used to obtain a file vector of a current file to be extracted, and input the file vector into a trained key data extraction model to obtain a first intermediate vector; A second intermediate vector determination module, configured to process the first intermediate vector based on a current word heat matrix corresponding to the current file to obtain a second intermediate vector; wherein the current word heat matrix is determined according to the current word and the corresponding current word frequency in the current file, and the historical word and the corresponding historical word frequency in the historical file; A key data output module is used to output the key data of the current file according to the second intermediate vector and the file vector.
8. The device according to claim 7, characterized in that The current vocabulary heat matrix is determined based on the following means: The associated data determination module is used to obtain historical files within a historical period, and process the current file and the historical file respectively to obtain current file associated data and historical file associated data; wherein the current file associated data includes current vocabulary and corresponding current word frequency; the historical file associated data includes historical vocabulary and corresponding historical word frequency; A reference word frequency determination module, used to select a reference word matching the current word from the historical words, and use the historical word frequency corresponding to the reference word as the reference word frequency; A vocabulary heat determination module, used to determine the current vocabulary heat of the current vocabulary according to the current vocabulary frequency, the reference vocabulary frequency and the historical period; The heat matrix generation module is used to generate a current word heat matrix including the current word heat.
9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement a file-based key data extraction method as described in any one of claims 1-6.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a file-based key data extraction method as described in any one of claims 1 to 6 is implemented.
