Text data representation method and device, electronic equipment and storage medium

Abstract

The application provides a text data representation method and device, electronic equipment and storage medium. The method comprises the following steps: analyzing and determining document structure information of all element attribute information in target text data, constructing structure data corresponding to the document structure information, embedding the target text data into the structure data, and determining a vector used to represent the text data based on vector information in the structure data.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a method, apparatus, electronic device and storage medium for representing text data. Background Technology

[0002] In our daily work with Natural Language Processing (NLP) tasks, the data we encounter primarily includes structured data, semi-structured data, unstructured data, and file data. Structured data comes from databases, Excel spreadsheets, etc.; semi-structured data comes from web pages and the XML data we need to crawl; unstructured data comes from natural language used by humans; and file data comes from commonly used Office software, such as doc, docx, and pdf files. File data contains a wealth of information, including document structure such as document title, document beginning, paragraph headings, subheadings, body text, document ending, and the hierarchical relationships between them. Extracting this information more efficiently and completely is crucial for downstream NLP tasks.

[0003] Currently, spatial representation of text data primarily relies on language models. These models can be broadly categorized into static and dynamic language models. Static language models mainly include TF-IDF (Word Frequency-Inverse Document Model), Word2Vec (Static Local Model), and GloVe (Static Global Model), while dynamic language models include NNLM, LSTM, BERT, and XLNET. Static language models cannot adjust word vectors based on semantics, while dynamic language models can adjust word vectors according to semantics, but they are less adaptable to long text data. However, neither dynamic nor static language models adapt to the file structure of the data, thus compromising the file structure information and resulting in inaccurate representation of the file data. Summary of the Invention

[0004] To address the problems in the aforementioned related technologies, this application provides a method, apparatus, electronic device, and storage medium for representing text data.

[0005] This application provides a method for representing text data, the method comprising:

[0006] Get text data;

[0007] The text data is preprocessed to obtain the target text data;

[0008] The document structure information is determined based on the attribute information of all elements in the target text data, and the structure data corresponding to the document structure information is constructed, wherein an element is a line of text data in the target text data;

[0009] The target text data is embedded into the structured data to vectorize the target text data;

[0010] Based on the vector information in the structured data, a vector is determined to represent the text data.

[0011] In some embodiments, preprocessing the text data to obtain target text data includes:

[0012] Iterate through the character encoding of each element in the text data;

[0013] Remove all encodings except for those corresponding to numbers, letters, punctuation marks, and newlines to obtain the target text data.

[0014] In some embodiments, the attribute information includes: left coordinate information, right coordinate information, font weight information, and whether it is centered. Determining the document structure information based on the attribute information of all elements in the target text data includes:

[0015] The document title and paragraph title are determined based on the font weight and centering information of each element.

[0016] The paragraph text is determined based on the element's left coordinate information, right coordinate information, font weight information, and whether it is centered.

[0017] The document structure information is determined based at least on the document title, paragraph titles, and paragraph text.

[0018] In some embodiments, the attribute information further includes: location information, and the method further includes:

[0019] Input each element, its corresponding position information, and font weight information into the text classifier, and output the beginning and end of the document.

[0020] The step of determining document structure information based at least on the document title, paragraph titles, and paragraph text includes: determining document structure information based on the document title, paragraph titles, paragraph text, document beginning, and document ending.

[0021] In some embodiments, determining the vector representing the text data based on the vector information in the structured data includes:

[0022] The document title vector is determined based on the vector corresponding to the document title in the structured data.

[0023] The paragraph vector is determined based on the vectors corresponding to all paragraph titles in the structured data and the vectors corresponding to the paragraph text of each paragraph title.

[0024] The document start vector is determined based on the vector corresponding to the beginning of the document in the data structure.

[0025] The document end vector is determined based on the vector corresponding to the end of the document in the data structure.

[0026] The vector used to represent the text data is determined based on the document title vector, paragraph vector, document beginning vector, and document ending vector.

[0027] In some embodiments, paragraph headings include multiple levels of paragraph headings, and paragraph vectors are determined based on the vectors corresponding to all paragraph headings in the structured data and the vectors corresponding to the paragraph text of each paragraph heading, including:

[0028] Determine the similarity between the vectors of paragraph headings at each level and the vectors of the corresponding paragraph text in each level of paragraph heading.

[0029] Normalized similarity is obtained by normalizing the similarity between the vectors of paragraph titles at each level and the vectors of the corresponding paragraph text.

[0030] Each initial paragraph vector is determined based on the normalized similarity, the vector of each level of paragraph title, and the vector of the paragraph text corresponding to each level of paragraph title.

[0031] The paragraph vector is obtained by averaging the initial paragraph vectors.

[0032] In some embodiments, determining the vector representing the text data based on the document title vector, paragraph vector, document beginning vector, and document end vector includes:

[0033] Obtain the weight of the document title, the beginning of the document, the end of the document, and the paragraphs;

[0034] The first calculation result is obtained by multiplying the weight of the document title by the document title vector;

[0035] The second calculation result is obtained by multiplying the weight of the document's beginning by the document's beginning vector;

[0036] The third calculation result is obtained by multiplying the weight of the end of the document by the vector of the end of the document.

[0037] The fourth calculation result is obtained by multiplying the paragraph weight by the paragraph vector;

[0038] The target vector is obtained by summing the first, second, third, and fourth calculation results.

[0039] The target vector is represented as a vector of the text data.

[0040] This application provides a text data representation device, including:

[0041] The first acquisition module is used to acquire text data;

[0042] The preprocessing module is used to preprocess the text data to obtain the target text data;

[0043] The first determining module is used to determine document structure information based on the attribute information of all elements in the target text data, and to construct the structure data corresponding to the document structure information, wherein an element is a line of text data in the target text data;

[0044] An embedding module is used to embed the target text data into the structured data;

[0045] The second determining module uses vector information in the structured data to determine a vector to represent the text data.

[0046] This application provides an electronic device, including a memory and a processor. The memory stores a computer program, which, when executed by the processor, performs a text data representation method as described in any of the above embodiments.

[0047] This application provides a storage medium storing a computer program that can be executed by one or more processors and can be used to implement the text data representation method described in any of the above claims.

[0048] This application provides a method, apparatus, electronic device, and storage medium for representing text data. By analyzing the attribute information of all elements in the target text data to determine the document structure information, constructing the structure data corresponding to the document structure information, embedding the target text data into the structure data, and determining the vector to represent the text data based on the vector information in the structure data, the text data can be represented more effectively. Attached Figure Description

[0049] The present application will be described in more detail below based on embodiments and with reference to the accompanying drawings.

[0050] Figure 1 A schematic diagram illustrating the implementation process of a text data representation method provided in this application embodiment;

[0051] Figure 2 A schematic diagram illustrating the implementation process of a text data representation method provided in this application embodiment;

[0052] Figure 3 A schematic diagram of the structure of a text data representation device provided in an embodiment of this application;

[0053] Figure 4 This is a schematic diagram of the composition structure of the electronic device provided in the embodiments of this application.

[0054] In the accompanying drawings, the same parts are referred to by the same reference numerals, and the drawings are not drawn to scale. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0057] If the application documents contain similar descriptions such as "first, second, third", the following explanation shall be added: In the following description, the terms "first, second, third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0059] Before introducing the embodiments of this application, a brief overview of the problems in the related technologies will be provided.

[0060] A method for representing a file provided in related technologies includes:

[0061] Step S1: Obtain the file.

[0062] Step S2: Preprocess the file.

[0063] Step S3: Text sentence vector representation.

[0064] In related technologies, static language models are mainly used for vector representation, such as TF-IDF models for sentence vectors, Word2Vec models for weighted vector averaging, and GloVe models for weighted vector averaging, to learn spatial representations. However, static vectors cannot dynamically adjust to the textual context. TF-IDF, Word2Vec, and GloVe models are shallow models that cannot learn deep semantic features and do not adapt to document structure, resulting in the loss of document structure information.

[0065] Another method of file representation provided in related technologies includes:

[0066] Step S11: Obtain the file.

[0067] Step S12, file preprocessing.

[0068] Step S13, text cropping.

[0069] Step S14: Identify vectors using a language model.

[0070] Compared to shallow models, the above approach uses a deep model, which can include LSTM, BERT, and XLNET models. These pre-trained models are used to learn the spatial representation of text. Solving the long text problem requires selective deletion of text. This approach typically has two drawbacks: it heavily relies on text length and loses textual information in long texts; it also doesn't consider the text structure of the file, leading to the loss of that structure.

[0071] Before introducing the embodiments of this application, the terms used in the embodiments of this application will be explained.

[0072] Text Vector Space: Text vectorization (also known as "word vector model" or "vector space model") is the representation of text as real-valued vectors that can be recognized by computers. Depending on the granularity, text feature representation can be divided into several levels, such as characters, words, sentences, or paragraphs.

[0073] Language Model: A language model is a model used to calculate the probability of a sentence, that is, the probability of judging a sentence as human speech.

[0074] Word embedding is a method of converting words in text into numerical vectors. These vectors, converted to numbers, need to be used as input for analysis using standard machine learning algorithms. The word embedding process involves embedding a high-dimensional space containing the number of words into a much lower-dimensional continuous vector space. Each word or phrase is mapped to a vector in the real number field, and the result is a word vector.

[0075] Pretrained Model: Pretraining involves training a language model with a large amount of unlabeled language text to obtain a set of model parameters. These parameters are then used to initialize the model, and finally fine-tuned based on the existing language model according to the specific task.

[0076] Based on the problems existing in related technologies, this application provides a method for representing text data. The method is applied to electronic devices, specifically mobile phones, tablets, laptops, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, personal digital assistants (PDAs), wearable electronic devices, in-vehicle devices (also known as vehicle systems), virtual reality devices, servers, etc. This application does not impose any limitations on these. The functions implemented by the text data representation method provided in this application can be achieved by the processor of the electronic device calling program code, wherein the program code can be stored in a computer storage medium. This application provides a method for representing text data. Figure 1 This is a schematic diagram illustrating the implementation process of a text data representation method provided in an embodiment of this application, as shown below. Figure 1 As shown, it includes:

[0077] Step S31: Obtain text data.

[0078] In this embodiment, text data can be obtained through input from an input device or through a network. The structure of the text data includes: document title, document beginning, paragraph title, paragraph subheading, paragraph body, document ending, etc.

[0079] In some embodiments, electronic devices can call the Python third-party module pydocx to obtain text data, such that the obtained text data is in list format.

[0080] Step S32: Preprocess the text data to obtain the target text data.

[0081] In this embodiment of the application, the character encoding of each element in the text data can be traversed, and other encodings except for those corresponding to numbers, letters, punctuation marks and newlines can be removed to obtain the target text data.

[0082] In this embodiment, the Python `ord` function can be invoked to iterate through the elements, check the ASCII code of each character, and remove special characters other than numbers, letters, punctuation marks, and newline characters (\n\r) to obtain the target text data.

[0083] Step S33: Determine document structure information based on the attribute information of all elements in the target text data, and construct the structure data corresponding to the document structure information, wherein an element is a line of text data in the target text data.

[0084] In this embodiment, the attribute information includes: left coordinate information, right coordinate information, font weight information, and whether it is centered. The font includes both bold and thin fonts. The left coordinate information is the coordinate of the leftmost character of each element, and the right coordinate information is the coordinate of the rightmost character of each element. In this embodiment, the document structure information is the hierarchical relationship between the document title, document beginning, paragraph headings, paragraph subheadings, paragraph body text, and document ending.

[0085] In this embodiment, the document title and paragraph title can be determined based on the font weight and centering information of each element; the paragraph text can be determined based on the left coordinate information, right coordinate information, font weight information, and centering information of the element; and the document structure information can be determined based at least on the document title, paragraph title, and paragraph text.

[0086] For example, all elements of the document can be traversed, and the leftmost and rightmost coordinates of the entire document's text can be calculated and denoted as min_val and max_val, respectively. The current element is scanned to check if it is centered and its text thickness. If the current element's text is centered, it is considered a document title; if the text is thick and not centered, it can be considered a paragraph title. If it is not centered and the font is thin, it is considered line content, and the left coordinate of the current element is recorded as last_left and the right coordinate as last_right. If last_right > max_val, the current element is treated as a paragraph, and the last_left and last_right values ​​are updated. The next element is scanned, and its left and right coordinates are recorded as curr_left and curr_right. The size relationship between curr_left and last_right and the previous element's last_left and last_right is compared. If curr_left is less than last_left, the element needs to be merged with the previous element; if curr_right is less than last_right, the element is treated as a paragraph and the paragraphs are merged. Otherwise, the next element is searched to determine whether to merge paragraphs. Repeat this process until the entire document has been scanned, thus determining the document title, paragraph headings, and paragraph body text.

[0087] In some embodiments, the attribute information further includes: position information, which may include: the position at the beginning of the text and the position at the end of the text. Each element, its corresponding position information, and font weight information can be input into the text classifier to output the beginning and end of the document.

[0088] In this embodiment of the application, sample data can be obtained, which includes: text content, location information, text font weight information, and tag information. The tag information includes: document beginning tag and document ending tag. The initial text classifier is trained by using the text content, location information, and text font weight information as input and the tag information as output, thereby obtaining the text classifier.

[0089] The step of determining document structure information based at least on the document title, paragraph titles, and paragraph text includes: determining document structure information based on the document title, paragraph titles, paragraph text, document beginning, and document ending.

[0090] In this embodiment, paragraph titles and paragraph text can be merged into a single paragraph. The document beginning and end can be table of contents. Each structure in the structured data is constructed as a dict.key structure. dict.key consists of: doc_title, contents, and paragraph_contents. doc_title is the document title, contents is the table of contents, and paragraph_contents are the paragraphs. In this embodiment, contents can be categorized as the document beginning and document end.

[0091] Step S34: Embed the target text data into the structure data.

[0092] In this embodiment of the application, the pre-trained model BERT can be used to vectorize the text, thereby making each structure in the structured data correspond to a vector of the text.

[0093] Step S35: Based on the vector information in the structured data, determine the vector used to represent the text data.

[0094] In this embodiment, a document title vector is determined based on the vector corresponding to the document title in the structured data; a paragraph vector is determined based on the vectors corresponding to all paragraph titles in the structured data and the vectors corresponding to the paragraph text of each paragraph title; a document beginning vector is determined based on the vector corresponding to the beginning of the document in the data structure; a document ending vector is determined based on the vector corresponding to the end of the document in the data structure; and a vector representing the text data is determined based on the document title vector, paragraph vector, document beginning vector, and document ending vector.

[0095] In some embodiments, the attribute information may further include: font size. Bold, non-centered text can be considered a paragraph title, and the font size of the text corresponding to each paragraph title can be used to classify the paragraph titles into levels; the larger the font, the higher the level. For example, paragraph titles may include multiple levels, such as level one, level two, level three, etc. The similarity between the vectors of each level of paragraph titles and the vectors of the corresponding paragraph text can be determined. The similarity between the vectors of each level of paragraph titles and the vectors of the corresponding paragraph text is normalized to obtain a normalized similarity. Based on the normalized similarities, the vectors of each level of paragraph titles, and the vectors of the corresponding paragraph text, each initial paragraph vector is determined. The average of the initial paragraph vectors is calculated to obtain the paragraph vector.

[0096] Continuing the example above, the document title is vectorized to obtain the vector v_doc corresponding to [CLS], with a dimension of 1024. Headers (the beginning of the document) are vectorized; the content of the headings is a list, and each element is vectorized as v. Multiple vectors are averaged to obtain v_head. Paragraphs are vectorized, including paragraph titles and body text. Since a paragraph may contain multiple levels of headings and body text under those headings, BERT is used to vectorize them separately, resulting in vectors represented as v. The vectors for first-level headings are v1, second-level headings are v2, third-level headings are v3, and body text is v4. Then, cosine similarity is used to calculate sim1, sim2, and sim3 for v1 and v4, v2 and v4, and v3 and v4. Finally, sim1, sim2, and sim3 are normalized to obtain sim_normal1, sim_normal2, and sim_normal3. Then, the vector for this paragraph is calculated as v = 0.8*sim_normal1*v1 + 0.8*sim_normal2*v2 + 0.8*sim_normal3*v3 + 0.2*v4. The average of all paragraph vectors is then used to obtain the final paragraph vector v_para. Tails (document end) are vectorized; the heading content corresponding to Tails is a list, and each element is vectorized as v. The average of multiple vectors is then taken to obtain v_tail.

[0097] In this embodiment, determining the vector representing the text data based on the document title vector, paragraph vector, document beginning vector, and document ending vector can be achieved through the following steps: obtaining the weights of the document title, document beginning, document ending, and paragraphs; multiplying the document title weight by the document title vector to obtain a first calculation result; multiplying the document beginning weight by the document beginning vector to obtain a second calculation result; multiplying the document ending weight by the document ending vector to obtain a third calculation result; multiplying the paragraph weight by the paragraph vector to obtain a fourth calculation result; summing the first, second, third, and fourth calculation results to obtain a target vector; and representing the target vector as the vector of the text data.

[0098] Continuing the example above, the obtained v_doc, v_head, v_para, and v_tail are assigned the following weights: v_doc = w_doc = 0.2, v_para = w_para = 0.5, v_head = w_head = (1 - w_doc - w_para) / len(v_head, v_doc), and v_tail = w_tail = (1 - w_doc - w_para) / len(v_tail, v_doc). The file data representation vector is then V = w_doc * v_doc + w_para * v_para + w_head * v_head + w_tail * v_tail. Here, len() is used to obtain the length of the string.

[0099] This application provides a method for representing text data. By analyzing the attribute information of all elements in the target text data to determine the document structure information, constructing the structure data corresponding to the document structure information, embedding the target text data into the structure data, and determining the vector to represent the text data based on the vector information in the structure data, the method can better represent the text data.

[0100] Based on the foregoing embodiments, this application further provides a method for representing text data. Figure 2 This is a schematic diagram illustrating the implementation process of a text data representation method provided in an embodiment of this application, as shown below. Figure 2 As shown, it includes:

[0101] Step S41: Obtain file data.

[0102] Using the Python third-party module pydocx, the read data is formatted as a list, with each element being a line of text content from a text file.

[0103] Step S42: Preprocess the file data.

[0104] Using Python's built-in `ord` function, we can iterate through the rows of data, check the ASCII code of each character, and remove special characters except for numbers, letters, punctuation marks, and newline characters `\n` and `\r`.

[0105] Step S43, file structure parsing.

[0106] Obtain the left and right coordinates, font size, font weight, and whether the text is centered for each line of text. Iterate through all lines of the document, calculating the leftmost and rightmost values ​​of the entire document, denoted as `min_val` and `max_val` respectively. Scan the current line, checking if it is centered and its font weight. If the text is centered, it can be considered a document title; if the font weight is heavy, it can be considered a paragraph title. Otherwise, treat it as line content, recording the left and right coordinates as `last_left` and `last_right`. If `last_right` is greater than or equal to `max_val`, treat the line as a paragraph, update `last_left` and `last_right` values, and scan the next line, recording the left and right coordinates as `curr_left` and `curr_right`. Compare these values ​​with `last_left` and `last_right`. If `curr_left` is less than `last_left`, merge the line with the previous line; if `curr_right` is less than `last_right`, merge the paragraphs. Otherwise, check the next line and determine if paragraphs should be merged. Repeat this process until the entire document has been scanned.

[0107] After determining the file structure, construct the structure data. The data format is constructed as dict. The keys are doc_title, contents, and paragraph_contents, and the values ​​are the document title, table of contents, and paragraph content, respectively.

[0108] Step S44, document classifier prediction.

[0109] The content is classified into two categories: "heads" and "tails". Input features include: text content, text position, and text font weight. Outputs are: "heads" and "tails".

[0110] Step S45, text embedding.

[0111] The text is vectorized using the pre-trained model BERT.

[0112] Step S46 indicates spatial alignment.

[0113] The process is divided into four main parts: First, document title vectorization, resulting in the vector v_doc corresponding to [CLS] with a dimension of 1024. Second, the heading content corresponding to heads is a list, with each element vectorized as v. Multiple vectors are averaged to obtain v_head. Third, for paragraph headings, since a paragraph may contain multiple levels of headings and body text under those headings, BERT is used to vectorize them separately, resulting in vectors represented as v. The vectors for level 1 headings are v1, level 2 headings are v2, level 3 headings are v3, and body text is v4. Then, cosine similarity is used to calculate sim1, sim2, and sim3 for v1 and v4, v2 and v4, and v3 and v4. Finally, sim1, sim2, and sim3 are normalized to obtain sim_normal1, sim_normal2, and sim_normal3. Then, the vector for this paragraph is calculated as v = 0.8*sim_normal1*v1 + 0.8*sim_normal2*v2 + 0.8*sim_normal3*v3 + 0.2*v4. The average of the vectors for all paragraphs is then used to obtain the final vector v_para. The heading content corresponding to tails is a list, and each element is vectorized into v. The average of multiple vectors is then taken to obtain v_tail.

[0114] Step S47 indicates spatial fusion.

[0115] The obtained v_doc, v_head, v_para, and v_tail are assigned the following weights: v_doc = w_doc = 0.2, v_para = w_para = 0.5, v_head = (1 - w_doc - w_para) / len(v_head, v_doc), and v_tail = (1 - w_doc - w_para) / len(v_tail, v_doc). The resulting file data representation vector is:

[0116] V=w_doc*v_doc+w_para*v_para+w_head*v_head+w_tail*v_tail.

[0117] The method provided in this application, in terms of extracting deep semantics, selects a pre-trained model for text sentence vector generation. Regarding text length, we prune long texts according to their structural hierarchy. For extracting file structure information, we adopt a targeted strategy of parsing the text structure. Through these three aspects of adaptation, we ensure that the embedded text vectors have a good representation of the text file.

[0118] Based on the foregoing embodiments, this application provides a text data representation device. The various modules and units included in the device can be implemented by a processor in a computer device; of course, they can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0119] This application provides a text data representation device 300. Figure 3 This is a schematic diagram of the structure of a text data representation device provided in an embodiment of this application, as shown below. Figure 3 As shown, the text data representation device 300 includes:

[0120] The first acquisition module 301 is used to acquire text data;

[0121] Preprocessing module 302 is used to preprocess the text data to obtain target text data;

[0122] The first determining module 303 is used to determine document structure information based on the attribute information of all elements in the target text data, and to construct the structure data corresponding to the document structure information, wherein an element is a line of text data in the target text data;

[0123] Embedding module 304 is used to embed the target text data into the structure data;

[0124] The second determining module 305 determines a vector to represent the text data based on the vector information in the structure data.

[0125] In some embodiments, the preprocessing module 302 includes:

[0126] A traversal unit is used to traverse the character encoding of each element in the text data;

[0127] The removal unit is used to remove codes other than those corresponding to numbers, letters, punctuation marks, and newlines to obtain the target text data.

[0128] In some embodiments, the attribute information includes: left coordinate information, right coordinate information, font weight information, and whether it is centered. The first determining module 303 includes:

[0129] The first determining unit is used to determine the document title and paragraph title based on the font weight and centering information of each element;

[0130] The second determining unit is used to determine the paragraph text based on the left coordinate information, right coordinate information, font weight information, and whether it is centered.

[0131] The third determining unit is used to determine the document structure information based at least on the document title, paragraph title, and paragraph body.

[0132] In some embodiments, the attribute information further includes: location information, and the text data representation device 300 further includes:

[0133] The input module is used to input each element, its corresponding position information, and font weight information into the text classifier, and output the beginning and end of the document.

[0134] The third determining unit is used to determine the document structure information based on the document title, paragraph title, paragraph body, document beginning, and document ending.

[0135] In some embodiments, the second determining module 305 includes:

[0136] The fourth determining unit is used to determine the document title vector based on the vector corresponding to the document title in the structured data;

[0137] The fifth determining unit is used to determine the paragraph vector based on the vectors corresponding to all paragraph titles in the structure data and the vectors corresponding to the paragraph text of each paragraph title;

[0138] The sixth determining unit is used to determine the document beginning vector based on the vector corresponding to the beginning of the document in the data structure;

[0139] The seventh determining unit is used to determine the document end vector based on the vector corresponding to the end of the document in the data structure;

[0140] The eighth determining unit is used to determine a vector to represent the text data based on the document title vector, paragraph vector, document beginning vector, and document ending vector.

[0141] In some embodiments, the paragraph headings include multiple levels of paragraph headings, and the fifth determining unit includes:

[0142] The first determination subunit is used to determine the similarity between the vectors of each level of paragraph headings in each paragraph heading and the vectors of the paragraph text corresponding to each level of paragraph headings;

[0143] The normalization processing subunit is used to normalize the similarity between the vectors of paragraph titles at each level and the vectors of the corresponding paragraph text in each level of paragraph title to obtain the normalized similarity.

[0144] The second determining subunit is used to determine each initial paragraph vector based on each normalized similarity, the vector of each level of paragraph title, and the vector of the paragraph text corresponding to each level of paragraph title.

[0145] The third determining sub-unit is used to calculate the average value of each initial paragraph vector to obtain the paragraph vector.

[0146] In some embodiments, the eighth determining unit includes:

[0147] Get sub-cells to get the weight of the document title, the beginning of the document, the end of the document, and the paragraph.

[0148] The first calculation subunit is used to multiply the weight of the document title by the document title vector to obtain the first calculation result;

[0149] The second calculation subunit is used to multiply the weight of the document beginning by the document beginning vector to obtain the second calculation result;

[0150] The third calculation subunit is used to multiply the weight of the end of the document by the vector of the end of the document to obtain the third calculation result;

[0151] The fourth calculation subunit is used to multiply the paragraph weight by the paragraph vector to obtain the fourth calculation result;

[0152] The fifth calculation subunit is used to sum the first calculation result, the second calculation result, the third calculation result, and the fourth calculation result to obtain the target vector;

[0153] The subunit represents the target vector as a vector of the text data.

[0154] It should be noted that, in the embodiments of this application, if the above-described text data representation method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0155] Accordingly, embodiments of this application provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the text data representation method provided in the above embodiments.

[0156] This application provides an electronic device; Figure 4 This is a schematic diagram of the composition structure of the electronic device provided in the embodiments of this application, such as... Figure 4 As shown, the electronic device 500 includes: a processor 501, at least one communication bus 502, a user interface 503, at least one external communication interface 504, and a memory 505. The communication bus 502 is configured to enable communication between these components. The user interface 503 may include a display screen, and the external communication interface 504 may include standard wired and wireless interfaces. The processor 501 is configured to execute a method program for representing text data stored in the memory to implement the steps in the text data representation method provided in the above embodiment.

[0157] The descriptions of the above embodiments of the electronic devices and storage media are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the embodiments of the computer devices and storage media of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0158] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0159] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0160] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0161] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0162] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0163] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0164] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a controller to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0165] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for representing text data, characterized in that, include: Get text data; The text data is preprocessed to obtain the target text data; Based on the attribute information of all elements in the target text data, the document structure information is determined, and the structure data corresponding to the document structure information is constructed. Here, an element is a line of text data in the target text data. The attribute information includes: left coordinate information, right coordinate information, font weight information, and whether it is centered. The document structure information is the hierarchical relationship between the document title, document beginning, paragraph title, paragraph sub-title, paragraph body, and document ending. The target text data is embedded into the structured data to vectorize the target text data; The document title vector is determined based on the vector corresponding to the document title in the structured data. The paragraph vector is determined based on the vectors corresponding to all paragraph titles in the structured data and the vectors corresponding to the paragraph text of each paragraph title. The paragraph vector is obtained based on the average value of each initial paragraph vector. Each initial paragraph vector is determined based on the normalized similarity between the vectors of each level of paragraph titles and the vectors of the paragraph text corresponding to each level of paragraph titles, the vectors of each level of paragraph titles, and the vectors of the paragraph text corresponding to each level of paragraph titles. The document start vector is determined based on the vector corresponding to the beginning of the document in the data structure. The document end vector is determined based on the vector corresponding to the end of the document in the data structure. The vector used to represent the text data is determined based on the document title vector, paragraph vector, document beginning vector, document ending vector, document title weight, document beginning weight, document ending weight, and paragraph weight.

2. The method according to claim 1, characterized in that, The preprocessing of the text data to obtain the target text data includes: Iterate through the character encoding of each element in the text data; Remove all encodings except for those corresponding to numbers, letters, punctuation marks, and newlines to obtain the target text data.

3. The method according to claim 1, characterized in that, The process of determining document structure information based on the attribute information of all elements in the target text data includes: The document title and paragraph title are determined based on the font weight and centering information of each element. The paragraph text is determined based on the element's left coordinate information, right coordinate information, font weight information, and whether it is centered. The document structure information is determined based at least on the document title, paragraph titles, and paragraph text.

4. The method according to claim 3, characterized in that, The attribute information further includes: location information, and the method further includes: Input each element, its corresponding position information, and font weight information into the text classifier, and output the beginning and end of the document. The step of determining document structure information based at least on the document title, paragraph titles, and paragraph text includes: determining document structure information based on the document title, paragraph titles, paragraph text, document beginning, and document ending.

5. The method according to claim 1, characterized in that, Paragraph headings include multiple levels of paragraph headings. The paragraph vector is determined based on the vectors corresponding to all paragraph headings in the structured data and the vectors corresponding to the paragraph text of each paragraph heading, including: Determine the similarity between the vectors of paragraph headings at each level and the vectors of the corresponding paragraph text in each level of paragraph heading. Normalized similarity is obtained by normalizing the similarity between the vectors of paragraph titles at each level and the vectors of the corresponding paragraph text. Each initial paragraph vector is determined based on the normalized similarity, the vector of each level of paragraph title, and the vector of the paragraph text corresponding to each level of paragraph title. The paragraph vector is obtained by averaging the initial paragraph vectors.

6. The method according to claim 5, characterized in that, The process of determining the vector representing the text data based on the document title vector, paragraph vector, document beginning vector, document ending vector, document title weight, document beginning weight, document ending weight, and paragraph weight includes: Obtain the weight of the document title, the beginning of the document, the end of the document, and the paragraphs; The first calculation result is obtained by multiplying the weight of the document title by the document title vector; The second calculation result is obtained by multiplying the weight of the document's beginning by the document's beginning vector; The third calculation result is obtained by multiplying the weight of the end of the document by the vector of the end of the document. The fourth calculation result is obtained by multiplying the paragraph weight by the paragraph vector; The target vector is obtained by summing the first, second, third, and fourth calculation results. The target vector is represented as a vector of the text data.

7. A device for representing text data, characterized in that, include: The first acquisition module is used to acquire text data; The preprocessing module is used to preprocess the text data to obtain the target text data; The first determining module is used to determine document structure information based on the attribute information of all elements in the target text data, and construct the structure data corresponding to the document structure information. Here, an element is a line of text data in the target text data. The attribute information includes: left coordinate information, right coordinate information, font weight information, and whether it is centered. The document structure information is the hierarchical relationship between the document title, document beginning, paragraph title, paragraph sub-title, paragraph body, and document ending. An embedding module is used to embed the target text data into the structured data to vectorize the target text data; The second determining module is used to determine a vector to represent the text data based on the vector information in the structured data; The second determining module includes: The fourth determining unit is used to determine the document title vector based on the vector corresponding to the document title in the structured data; The fifth determining unit is used to determine the paragraph vector based on the vectors corresponding to all paragraph titles in the structural data and the vectors corresponding to the paragraph text of each paragraph title. The paragraph vector is obtained based on the average value of each initial paragraph vector. Each initial paragraph vector is determined based on the normalized similarity between the vectors of each level of paragraph titles and the vectors of the paragraph text corresponding to each level of paragraph titles, the vectors of each level of paragraph titles, and the vectors of the paragraph text corresponding to each level of paragraph titles. The sixth determining unit is used to determine the document beginning vector based on the vector corresponding to the beginning of the document in the data structure; The seventh determining unit is used to determine the document end vector based on the vector corresponding to the end of the document in the data structure; The eighth determining unit is used to determine a vector representing the text data based on the document title vector, paragraph vector, document beginning vector, document ending vector, weight of document title, weight of document beginning, weight of document ending, and weight of paragraph.

8. An electronic device, characterized in that, include: It includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, performs the method for representing text data as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores a computer program, which is executed by a computer using the method for representing text data as described in any one of claims 1-6.

Images