Text data denoising method for large model of mine industry

Abstract

The invention provides a text data denoising method for a mining industry large model. The method comprises the following steps: determining a typesetting state of a home page in a knowledge document; wherein the knowledge document is a document in a mine industry knowledge base; determining page header coordinates of the knowledge document, page footer coordinates of a home page and page footer coordinates of a non-home page in the knowledge document; cutting the home page according to the header coordinates and the footer coordinates of the home page in combination with the typesetting state of the home page, so that the cut home page only contains the text; and cutting the non-home page according to the header coordinates and the footer coordinates of the non-home page, so that the cut non-home page only contains the text. According to the method, noise data in the mine industry knowledge base can be eliminated, the quality and accuracy of the coal industry knowledge base are improved, the quality of the coal industry high-quality text data set is improved, and the influence of the noise data on the model performance is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of data processing in the mining industry, and in particular to a text data denoising method for a large model of the mining industry. Background Art

[0002] With the rapid development of the coal industry and the deepening of digital transformation, the construction and application of coal industry knowledge bases and high-quality data sets of industry texts have become increasingly important. As an important carrier for storing, managing and utilizing industry knowledge, knowledge bases play an irreplaceable role in improving the decision-making efficiency of coal enterprises, optimizing production processes, and promoting technological innovation.

[0003] However, due to the diversity of data sources and the complexity of data formats, various noise data will inevitably be mixed in, which will seriously affect the quality and accuracy of the knowledge base. Therefore, how to effectively remove noise data from the coal industry knowledge base, improve the quality and accuracy of the knowledge base, and improve the quality of text data sets is an urgent problem to be solved in the current digital transformation of the coal industry. Summary of the invention

[0004] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0005] To this end, the first purpose of this application is to propose a text data denoising method for large models in the mining industry.

[0006] The second purpose of this application is to propose a text data denoising device for large models of the mining industry.

[0007] The third objective of the present application is to provide an electronic device.

[0008] A fourth objective of the present application is to provide a computer-readable storage medium.

[0009] A fifth object of the present application is to provide a computer program product.

[0010] To achieve the above-mentioned purpose, the first embodiment of the present application proposes a text data denoising method for a large model of the mining industry, comprising: Determining the layout status of the homepage in the knowledge document; wherein the knowledge document is a document in the mining industry knowledge base; Determine the header coordinates of the knowledge document, the footer coordinates of the first page in the knowledge document, and the footer coordinates of the non-first page; According to the header coordinates and the footer coordinates of the first page, combined with the typesetting state of the first page, the first page is cropped so that the cropped first page only contains the text; The non-first page is cropped according to the header coordinates and the footer coordinates of the non-first page, so that the cropped non-first page only contains the main text.

[0011] To achieve the above-mentioned purpose, the second embodiment of the present application proposes a text data denoising device for a large model of the mining industry, comprising: A first determination module is used to determine the layout status of the homepage in the knowledge document; wherein the knowledge document is a document in the mining industry knowledge base; A second determination module is used to determine the header coordinates of the knowledge document, the footer coordinates of the first page of the knowledge document, and the footer coordinates of the non-first page; A cropping module, used for cropping the home page according to the header coordinates and the footer coordinates of the home page, combined with the typesetting state of the home page, so that the cropped home page only contains the text; The cropping module is further used to crop the non-first page according to the header coordinates and the footer coordinates of the non-first page, so that the cropped non-first page only contains the text.

[0012] To achieve the above-mentioned purpose, the third aspect of the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method as described in the embodiment of the first aspect.

[0013] To achieve the above-mentioned purpose, the fourth aspect embodiment of the present application proposes a computer-readable storage medium, in which computer-readable storage medium is stored computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the method described in the first aspect embodiment.

[0014] To achieve the above-mentioned purpose, the fifth aspect of the present application proposes a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0015] The text data denoising method for the large model of the mining industry provided in the present application crops the homepage according to the header coordinates of the knowledge document and the footer coordinates of the homepage, combined with the typesetting status of the homepage, and crops the non-homepage according to the header coordinates and the footer coordinates of the non-homepage, thereby removing the noise data in the homepage and non-homepage in the knowledge document, so that the cropped knowledge document only contains the main text, thereby achieving the purpose of eliminating noise data in the knowledge base of the mining industry, improving the quality and accuracy of the coal industry knowledge base, improving the quality of high-quality text data sets in the coal industry, and reducing the impact of noise data on model performance.

[0016] Moreover, using different methods to crop the home page and non-home page in the knowledge document can improve the accuracy of noise data removal. In addition, cropping the home page in combination with the layout status of the home page can improve the accuracy of noise data removal on the home page.

[0017] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which: Figure 1 A flowchart of a text data denoising method for a large model of the mining industry provided in an embodiment of the present application; Figure 2 A flowchart of another text data denoising method for a large model of the mining industry provided in an embodiment of the present application; Figure 3 A horizontal coordinate histogram in a double-column state of the home page provided in an embodiment of the present application; Figure 4 A horizontal coordinate histogram in a single-column state of the home page provided in an embodiment of the present application; Figure 5 A structural schematic diagram of a text data denoising device for a large model of the mining industry provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0020] The following describes a text data denoising method for a large model of the mining industry in an embodiment of the present application with reference to the accompanying drawings.

[0021] Noise data in the mining industry knowledge base may include but is not limited to erroneous information, redundant data, inconsistently formatted data, irrelevant data, etc., which will seriously affect the quality and accuracy of the knowledge base. When using the knowledge base for big data analysis and using data sets for large model incremental pre-training and other advanced applications, noise data will interfere with the model training process, reduce the performance of the model, and even lead to wrong decision results.

[0022] Therefore, how to effectively remove noise data from the coal industry knowledge base, improve the quality and accuracy of the knowledge base, and improve the quality of text data sets is an urgent problem to be solved in the current digital transformation of the coal industry.

[0023] Based on this, an embodiment of the present application proposes a text data denoising method for a large model of the mining industry. Figure 1 A flowchart of a text data denoising method for a large model of the mining industry provided in an embodiment of the present application.

[0024] like Figure 1 As shown in the figure, the text data denoising method for the large model of the mining industry includes: Step 101, determining the layout status of the home page in the knowledge document.

[0025] The knowledge document may be, for example, a journal article, a conference paper, etc., and the format of the knowledge document may be a PDF format, a CAJ format, etc.

[0026] In addition, knowledge documents usually have single-column status, double-column status, etc. Taking journal articles as an example, the first page of a journal article usually contains the introduction of the corresponding author, the author's profile, the journal article's manuscript receipt date, the journal article's acceptance date, the projects involved in the journal article, etc. Due to the different layout states of the first page, the position of this information on the first page may also be different. Therefore, the layout state of the first page in the knowledge document can be determined.

[0027] As a possible implementation method, the home page image of the knowledge document can be captured and recognized to determine the width of the text area. If the width of the text area is close to the page width, it can be determined that the typesetting state is a single-column state. If the text area is divided into two areas of similar width, it can be determined that the typesetting state is a double-column state.

[0028] As another possible implementation, a machine learning algorithm such as a support vector machine, a decision tree, etc. may be used to classify the home page image to determine the layout status of the home page.

[0029] Step 102, determining the header coordinates of the knowledge document, the footer coordinates of the first page in the knowledge document, and the footer coordinates of the non-first page.

[0030] The non-home page may refer to other pages in the knowledge document except the home page.

[0031] Since the header positions of each page in a knowledge document are usually the same, the header coordinates of any page in the knowledge document can be determined, and the header coordinates of the page are also the header coordinates of the knowledge document.

[0032] Since the footer contents of the first page and non-first page in a knowledge document may be different, the footer coordinates of the first page and non-first page can be determined separately.

[0033] For example, a rectangular coordinate system can be established with the upper left corner of the knowledge document page as the coordinate origin, with the horizontal right direction as the positive direction of the x-axis and the vertical downward direction as the positive direction of the y-axis, to determine the coordinates of the header, the footer of the first page, and the footer of the non-first page in the rectangular coordinate system of the corresponding page.

[0034] Step 103, based on the header coordinates and the footer coordinates of the first page, combined with the layout status of the first page, the first page is cropped so that the cropped first page only contains the text.

[0035] Due to the different layout status of the home page, the positions of noise data such as the corresponding author, author profile, modification date, page number, etc. may be different on the home page. In addition, there may be noise data such as the category to which the knowledge document belongs and the publication date of the knowledge document above the header. Therefore, the home page can be cropped according to the header coordinates and the footer coordinates of the home page, combined with the layout status of the home page, to remove the noise data from the home page so that the cropped home page only contains the main text.

[0036] Step 104, cropping the non-first page according to the header coordinates and the footer coordinates of the non-first page, so that the cropped non-first page only contains the main text.

[0037] For non-first page in knowledge documents, the header content of the non-first page is usually the same as that of the first page, but the non-first page generally does not include noise data such as the corresponding author, author profile, modification date, etc. Therefore, the non-first page can be cropped according to the header coordinates and the footer coordinates of the non-first page to remove the noise data from the first page, so that the cropped non-first page only contains the main text.

[0038] For example, you can start from the header coordinates of the non-first page, crop to the right along the horizontal direction, and the cropping length is the page width. Start from the footer coordinates of the non-first page, crop to the left along the horizontal direction, and the cropping length is the page width. This will crop out the noise data above the header and below the footer, so that the cropped non-first page only contains the main text.

[0039] Exemplarily, the text data denoising method of the present application can be used to denoise the knowledge documents in the mining industry knowledge base, and the denoised knowledge documents can be used to fine-tune the initial large model to obtain the mining industry large model.

[0040] In an embodiment of the present application, the homepage is cropped according to the header coordinates of the knowledge document and the footer coordinates of the homepage, combined with the typesetting status of the homepage, and the non-homepage is cropped according to the header coordinates and the footer coordinates of the non-homepage, so that the noise data in the homepage and non-homepage in the knowledge document can be removed, so that the cropped knowledge document only contains the main text, thereby achieving the purpose of eliminating noise data in the mining industry knowledge base, which can improve the quality and accuracy of the coal industry knowledge base, improve the quality of the coal industry high-quality text data set, and reduce the impact of noise data on model performance.

[0041] Moreover, using different methods to crop the home page and non-home page in the knowledge document can improve the accuracy of noise data removal. In addition, cropping the home page in combination with the layout status of the home page can improve the accuracy of noise data removal on the home page.

[0042] Figure 2 A flowchart of another text data denoising method for a large model of the mining industry provided in an embodiment of the present application.

[0043] like Figure 2 As shown in the figure, the text data denoising method for the large model of the mining industry includes: Step 201, determine the vertical coordinate of the bottom of the home page, and convert the vertical coordinate of the bottom of the home page into image coordinates.

[0044] For example, the fitz package in Python can be used to read the knowledge document in PDF format of the coal industry, extract the vertical coordinates at the bottom of the first page of the knowledge document, and convert the extracted vertical coordinates into standard image coordinates.

[0045] For example, the vertical coordinates at the bottom of the home page can be converted to image coordinates according to the following formula (1): (1) in, Indicates the vertical coordinate of the bottom of the home page. Indicates the image coordinates at the bottom of the home page. Indicates the DPI value set when saving the cropped area as an image. DPI (dots per inch) is a unit of measurement for image resolution, and a higher DPI value means the image has higher details and clarity.

[0046] Step 202, determining the cropping range of the home page image according to the image coordinates at the bottom of the home page and the size of the home page image.

[0047] The home page image may be obtained by capturing an image of the home page in the knowledge document.

[0048] Exemplarily, the size of the home page image includes the length of the home page image and the width of the home page image. The first horizontal coordinate can be determined based on the width of the home page image and the first ratio, the second horizontal coordinate can be determined based on the width of the home page image and the second ratio, the first cropping coordinate can be determined based on the first horizontal coordinate and the image coordinates at the bottom of the home page, the second cropping coordinate can be determined based on the second horizontal coordinate and the difference between the length of the home page image and the image coordinates at the bottom of the home page, and the cropping range can be determined based on the first cropping coordinate and the second cropping coordinate.

[0049] The first ratio is smaller than half of the width of the home page image, the second ratio is larger than half of the width of the home page image, and the first horizontal coordinate and the second horizontal coordinate both refer to coordinates in the home page image.

[0050] Exemplarily, the first horizontal coordinate and the second horizontal coordinate may be coordinates in an image coordinate system established with the upper left corner of the home page image as the coordinate origin, the horizontal right direction of the home page image as the positive direction of the horizontal axis, and the vertical downward direction of the home page image as the positive direction of the vertical axis.

[0051] Exemplarily, the region between the horizontal coordinate of the first clipping coordinate and the horizontal coordinate of the second clipping coordinate, and between the vertical coordinate of the first clipping coordinate and the vertical coordinate of the second clipping coordinate may be determined as the clipping range.

[0052] For example, the length of the home page image is , the width is , the first ratio is , the second ratio is , the first clipping coordinate and the second clipping coordinate can be determined using the following formula (2): (2) in, Represents the vertical coordinate in the first clipping coordinates; Represents the horizontal coordinate in the first clipping coordinate, that is, the first horizontal coordinate; Represents the vertical coordinate in the second clipping coordinates; Represents the horizontal coordinate in the second clipping coordinate system, that is, the second horizontal coordinate.

[0053] Step 203, cropping the home page image according to the cropping range to obtain a cropped image.

[0054] Exemplarily, the home page image may be cropped according to the cropping range, and the cropped image within the cropping range may be used as the cropped image.

[0055] Step 204, obtaining a horizontal coordinate histogram according to the binarized image of the cropped image.

[0056] In the present application, the cropped image may be converted into a binary image, and a horizontal coordinate histogram may be obtained based on the binary image.

[0057] The abscissa of the horizontal coordinate histogram may be the column index of the pixel points in the cropped image, and the ordinate of the horizontal coordinate histogram may be the number of white pixel points.

[0058] for example, Figure 3 It is the horizontal coordinate histogram in the double-column state of the home page. Figure 4 It is the horizontal coordinate histogram in the single column state of the home page.

[0059] Step 205, determining the layout status of the home page according to the horizontal coordinate histogram.

[0060] Since there is a blank area of ​​a certain width between the two columns in the double-column state, in this application, the layout state of the home page can be determined based on the number of white pixels in adjacent columns in the horizontal coordinate histogram.

[0061] Exemplarily, if the number of elements with consecutive equal vertical coordinates in the horizontal coordinate histogram is greater than or equal to a preset threshold, it can be determined that the layout state of the homepage is a double-column state; if the number of elements with consecutive equal vertical coordinates in the horizontal coordinate histogram is less than the preset threshold, it can be determined that the layout state of the homepage is a single-column state.

[0062] For example, the preset threshold is 10. Figure 3 The number of white pixels in the 200th to 300th columns is equal, that is, the number of columns with equal vertical coordinates is 100, which is greater than the preset threshold of 10. It can be determined that the layout state of the homepage is a double-column state. Figure 4 If the number of consecutive columns with equal vertical coordinates is not greater than the preset threshold value 10, it can be determined that the layout state of the homepage is a single-column state.

[0063] Step 206, determining the header coordinates of the knowledge document, the footer coordinates of the first page in the knowledge document, and the footer coordinates of the non-first page.

[0064] Exemplarily, the image of any page in the knowledge document can be converted into a grayscale image, and Gaussian blur calculation can be performed on the grayscale image of any page to obtain a Gaussian blurred image, edge detection can be performed on the Gaussian blurred image to obtain a header, the header in the Gaussian blurred image can be detected based on the Hough line transform to obtain the vertical coordinates of the header in the image of the page, and then the header coordinates can be determined based on the vertical coordinates of the header in the image of the page and the preset horizontal coordinates.

[0065] As an example, the vertical coordinate of the header in the image of any page can be converted into the vertical coordinate of the header in the knowledge document, and the header coordinate is determined according to the vertical coordinate of the header in the knowledge document and the preset horizontal coordinate, wherein the preset horizontal coordinate can be 0.

[0066] For example, the vertical coordinate of the header in any page image is , then the vertical coordinate of the header in the knowledge document is .

[0067] Taking the knowledge document in PDF format as an example, the header coordinates may refer to the coordinates of the header in the knowledge document in PDF format.

[0068] For example, you can use the fitz module in the PyMuPDF library to convert any page in a knowledge document in PDF format to an image. The get_pixmap(dpi=300) method can be used to convert a page into an image with a resolution of 300 DPI.

[0069] As another example, you can use OpenCV’s cv2.imread() function to read a saved image file, load it into memory, and use OpenCV’s cv2.cvtColor() function to convert the image from the BGR color space to grayscale.

[0070] In addition, the grayscale image is Gaussian blurred and the pixels can be convolved using the Gaussian function as weights to reduce image noise and details.

[0071] Exemplarily, all text boxes in the home page can be traversed to determine the text box containing the target vocabulary, and the vertical coordinate of the top edge of the text box containing the target vocabulary can be determined as the vertical coordinate of the footer of the home page, and then the footer coordinates of the home page can be determined based on the vertical coordinate of the footer of the home page and the preset horizontal coordinates.

[0072] For example, the target vocabulary may include but is not limited to "corresponding author", "author profile", "date of receipt", "date of acceptance", "funding project", "received", "accepted", "date", etc.

[0073] For example, the vertical coordinates of text boxes in non-home pages can be compared to determine the text box with the largest vertical coordinate in the non-home page, and based on whether the text box with the largest vertical coordinate in the non-home page contains a page number, the vertical coordinate of the footer of the non-home page can be determined, and then the coordinates of the footer of the non-home page can be determined based on the vertical coordinates of the footer of the non-home page and the horizontal coordinates of the footer of the non-home page.

[0074] If the text box with the largest vertical coordinate in a non-first page contains a page number, the vertical coordinate of the top edge of the text box with the largest vertical coordinate in the non-first page can be determined as the vertical coordinate of the footer of the non-first page. If the text box with the largest vertical coordinate in a non-first page does not contain a page number, it means that the text box with the largest vertical coordinate is the text box containing the main text, then the vertical coordinate of the bottom edge of the text box with the largest vertical coordinate in the non-first page can be determined as the vertical coordinate of the footer of the non-first page.

[0075] The top edge of the text box may refer to the top edge of the text box, and the bottom edge of the text box may refer to the bottom edge of the text box. It is understandable that the top edge and the bottom edge of the text box are parallel to each other, and the length of the top edge is the same as the length of the bottom edge.

[0076] Therefore, the vertical coordinates of the footer of the first page and the footer of non-first page can be determined by adopting different methods, thereby improving the accuracy of page cropping.

[0077] Step 207, based on the header coordinates and the footer coordinates of the first page, combined with the layout status of the first page, the first page is cropped so that the cropped first page only contains the text.

[0078] In this application, if the layout state of the home page is a single column state, the home page can be cropped according to the header coordinates and the footer coordinates of the home page so that the cropped home page contains only the main text.

[0079] For example, the horizontal coordinates of the footer of the first page are the page width, and you can start from the header coordinates of the first page and crop to the right horizontally, with the cropping length being the page width. You can also start from the footer coordinates of the first page and crop to the left horizontally, with the cropping length being the page width. This way, you can crop off the noise data above the header and below the footer of the first page, so that the cropped non-first page only contains the main text.

[0080] Since in a single-column state, noise data such as the corresponding author and receipt date are usually below the main text, the above cropping method can be used to crop out noise data such as the corresponding author, receipt date, page numbers that may be below the main text on the first page, and noise data above the header, so that the cropped first page only contains the main text.

[0081] If the layout status of the home page is double-column, the target vocabulary can be erased according to the footer coordinates of the home page, the page width of the knowledge document and the maximum vertical coordinates of the page to obtain the erased home page. Then, starting from the header coordinates of the home page, the erased home page can be cropped according to the page width and the vertical coordinates of the bottom edge of the text box containing the target vocabulary, so that the erased home page only contains the main text.

[0082] Exemplarily, the position of the target word can be determined according to the footer coordinates of the home page, the page width of the knowledge document and the maximum vertical coordinates of the page, and then the target word can be erased.

[0083] When the first page is in a double-column state, no matter whether the noise data such as the corresponding author, receipt date, fund project, etc. are in the left column of the double column or below the text, this cropping method can remove these noise data, and this cropping method can ensure that the page number can also be cropped when it is at the bottom of the first page.

[0084] Step 208, cropping the non-first page according to the header coordinates and the footer coordinates of the non-first page, so that the cropped non-first page only contains the main text.

[0085] In the present application, step 208 can refer to the method described in the previous embodiment and will not be described again here.

[0086] The text data denoising method for the large model of the mining industry in the embodiment of the present application can significantly improve the quality and accuracy of the coal industry knowledge base, significantly improve the quality of the high-quality text data set of the coal industry, and reduce the impact of noise data on model performance. At the same time, the denoised knowledge base high-quality text data set can be better used for RAG (Retrieval-Augmented Generation) and incremental pre-training of large models, thereby improving the performance of the model on coal industry related tasks.

[0087] In addition, the noise data removal method of the present application has certain versatility and can be extended to the knowledge base construction, high-quality text data construction and optimization in other industry fields.

[0088] In order to implement the above embodiments, the present application also proposes a text data denoising device for a large model of the mining industry.

[0089] Figure 5 A structural schematic diagram of a text data denoising device for a large model of the mining industry provided in an embodiment of the present application.

[0090] like Figure 5 As shown, the text data denoising device 500 for the large model of the mining industry includes: The first determination module 510 is used to determine the layout status of the homepage in the knowledge document; wherein the knowledge document is a document in the mining industry knowledge base; The second determination module 520 is used to determine the header coordinates of the knowledge document, the footer coordinates of the first page of the knowledge document, and the footer coordinates of the non-first page; A cropping module 530, for cropping the home page according to the header coordinates and the footer coordinates of the home page, combined with the typesetting state of the home page, so that the cropped home page only contains the text; The cropping module 530 is further used to crop the non-first page according to the header coordinates and the footer coordinates of the non-first page, so that the cropped non-first page only contains the text.

[0091] Furthermore, in a possible implementation of the embodiment of the present application, the first determining module 510 is configured to: Determine the vertical coordinate of the bottom of the home page, and convert the vertical coordinate of the bottom of the home page into image coordinates; Determine the cropping range of the home page image according to the image coordinates at the bottom of the home page and the size of the home page image; Cropping the home page image according to the cropping range to obtain a cropped image; Obtaining a binarized image of the cropped image, and obtaining a horizontal coordinate histogram based on the binarized image; The layout status of the home page is determined according to the horizontal coordinate histogram.

[0092] Further, in a possible implementation of the embodiment of the present application, the size of the home page image includes the length of the home page image and the width of the home page image, and the first determining module 510 is used to: Determine a first horizontal coordinate according to the width of the home page image and a first ratio; wherein the first ratio is less than half of the width of the home page image; Determining a second horizontal coordinate according to the width of the home page image and a second ratio; wherein the second ratio is greater than half of the width of the home page image; Determine a first cropping coordinate according to the first horizontal coordinate and the image coordinate at the bottom of the home page; Determine a second cropping coordinate according to the second horizontal coordinate and the difference between the length of the home page image and the image coordinate at the bottom of the home page; The clipping range is determined according to the first clipping coordinates and the second clipping coordinates.

[0093] Furthermore, in a possible implementation of the embodiment of the present application, the first determining module 510 is configured to: If the number of elements with equal vertical coordinates in the horizontal coordinate histogram is greater than or equal to a preset threshold, determining that the layout state of the homepage is a double-column state; If the number of elements with equal consecutive vertical coordinates in the horizontal coordinate histogram is less than a preset threshold, it is determined that the layout state of the homepage is a single-column state.

[0094] Further, in a possible implementation of the embodiment of the present application, the second determining module 520 is configured to: Converting an image of any page in the knowledge document into a grayscale image, and performing Gaussian blur calculation on the grayscale image of any page to obtain a Gaussian blurred image; Performing edge detection on the Gaussian blurred image to obtain a header; Detecting the header in the Gaussian blurred image based on Hough line transform to obtain the vertical coordinate of the header in the image of any page; The header coordinates are determined according to the vertical coordinates and preset horizontal coordinates of the header in the image of any one page.

[0095] Further, in a possible implementation of the embodiment of the present application, the second determining module 520 is configured to: Traversing all text boxes in the home page to determine a text box containing a target word; Determine the vertical coordinate of the top edge of the text box containing the target vocabulary as the vertical coordinate of the footer of the home page; The footer coordinates of the home page are determined according to the vertical coordinates of the footer of the home page and the preset horizontal coordinates.

[0096] Further, in a possible implementation of the embodiment of the present application, the second determining module 520 is configured to: Comparing the vertical coordinates of the text boxes in the non-home page to determine the text box with the largest vertical coordinate in the non-home page; If the text box with the largest vertical coordinate in the non-first page contains a page number, the vertical coordinate of the top edge of the text box with the largest vertical coordinate in the non-first page is determined as the vertical coordinate of the footer of the non-first page; If the text box with the largest vertical coordinate in the non-first page does not contain a page number, the vertical coordinate of the bottom edge of the text box with the largest vertical coordinate in the non-first page is determined as the vertical coordinate of the footer of the non-first page; Determining the horizontal coordinates of the footer of the non-home page according to the page width of the knowledge document; The footer coordinates of the non-home page are determined according to the vertical coordinates of the footer of the non-home page and the horizontal coordinates of the footer of the non-home page.

[0097] Furthermore, in a possible implementation of the embodiment of the present application, the cropping module 530 is used to: If the layout state of the home page is a single-column state, the home page is cropped according to the header coordinates of the home page and the footer coordinates of the home page, so that the cropped home page only contains the text; If the layout state of the homepage is a double-column state, erasing the target vocabulary according to the footer coordinates of the homepage, the page width of the knowledge document and the maximum vertical coordinates of the page, to obtain an erased homepage; Starting from the header coordinates of the first page, the erased first page is cropped according to the page width and the vertical coordinates of the bottom edge of the text box containing the target vocabulary, so that the erased first page only contains the text.

[0098] It should be noted that the aforementioned explanation of the embodiment of the text data denoising method for the large model of the mining industry is also applicable to the text data denoising device for the large model of the mining industry in this embodiment, and will not be repeated here.

[0099] In an embodiment of the present application, the homepage is cropped according to the header coordinates of the knowledge document and the footer coordinates of the homepage, combined with the typesetting status of the homepage, and the non-homepage is cropped according to the header coordinates and the footer coordinates of the non-homepage, so that the noise data in the homepage and non-homepage in the knowledge document can be removed, so that the cropped knowledge document only contains the main text, thereby achieving the purpose of eliminating noise data in the mining industry knowledge base, which can improve the quality and accuracy of the coal industry knowledge base, improve the quality of the coal industry high-quality text data set, and reduce the impact of noise data on model performance.

[0100] Moreover, using different methods to crop the home page and non-home page in the knowledge document can improve the accuracy of noise data removal. In addition, cropping the home page in combination with the layout status of the home page can improve the accuracy of noise data removal on the home page.

[0101] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments. In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0102] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0103] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0104] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0105] The present application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0106] In the description of the aforementioned embodiments, the descriptions with reference to the terms "as one possible implementation", "as another possible implementation", "exemplary", etc., mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they contradict each other.

[0107] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0108] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0109] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0110] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0111] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0112] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0113] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A text data denoising method for a large model in the mining industry, characterized in that: The following steps are involved: Determining the layout status of the homepage in the knowledge document; wherein the knowledge document is a document in the mining industry knowledge base; Determine the header coordinates of the knowledge document, the footer coordinates of the first page in the knowledge document, and the footer coordinates of the non-first page; According to the header coordinates and the footer coordinates of the first page, combined with the typesetting state of the first page, the first page is cropped so that the cropped first page only contains the text; The non-first page is cropped according to the header coordinates and the footer coordinates of the non-first page, so that the cropped non-first page only contains the main text.

2. The method according to claim 1, characterized in that Determining the layout status of the homepage in the knowledge document includes: Determine the vertical coordinate of the bottom of the home page, and convert the vertical coordinate of the bottom of the home page into image coordinates; Determine the cropping range of the home page image according to the image coordinates at the bottom of the home page and the size of the home page image; Cropping the home page image according to the cropping range to obtain a cropped image; Obtaining a binarized image of the cropped image, and obtaining a horizontal coordinate histogram based on the binarized image; The layout status of the home page is determined according to the horizontal coordinate histogram.

3. The method according to claim 2, characterized in that The size of the home page image includes the length of the home page image and the width of the home page image. The determining the cropping range of the home page image according to the image coordinates at the bottom of the home page and the size of the home page image includes: Determine a first horizontal coordinate according to the width of the home page image and a first ratio; wherein the first ratio is less than half of the width of the home page image; Determining a second horizontal coordinate according to the width of the home page image and a second ratio; wherein the second ratio is greater than half of the width of the home page image; Determine a first cropping coordinate according to the first horizontal coordinate and the image coordinate at the bottom of the home page; Determine a second cropping coordinate according to the second horizontal coordinate and the difference between the length of the home page image and the image coordinate at the bottom of the home page; The clipping range is determined according to the first clipping coordinates and the second clipping coordinates.

4. The method according to claim 2, characterized in that Determining the layout state of the homepage according to the horizontal coordinate histogram includes: If the number of elements with equal vertical coordinates in the horizontal coordinate histogram is greater than or equal to a preset threshold, determining that the layout state of the homepage is a double-column state; If the number of elements with equal consecutive vertical coordinates in the horizontal coordinate histogram is less than a preset threshold, it is determined that the layout state of the homepage is a single-column state.

5. The method according to claim 1, characterized in that Determining the header coordinates of the knowledge document includes: Converting an image of any page in the knowledge document into a grayscale image, and performing Gaussian blur calculation on the grayscale image of any page to obtain a Gaussian blurred image; Performing edge detection on the Gaussian blurred image to obtain a header; Detecting the header in the Gaussian blurred image based on Hough line transform to obtain the vertical coordinate of the header in the image of any page; The header coordinates are determined according to the vertical coordinates and preset horizontal coordinates of the header in the image of any one page.

6. The method according to claim 1, characterized in that Determining the footer coordinates of the first page in the knowledge document includes: Traversing all text boxes in the home page to determine a text box containing a target word; Determine the vertical coordinate of the top edge of the text box containing the target vocabulary as the vertical coordinate of the footer of the home page; The footer coordinates of the home page are determined according to the vertical coordinates of the footer of the home page and the preset horizontal coordinates.

7. The method according to claim 1, characterized in that Determining the footer coordinates of a non-home page in the knowledge document, including: Comparing the vertical coordinates of the text boxes in the non-home page to determine the text box with the largest vertical coordinate in the non-home page; If the text box with the largest vertical coordinate in the non-first page contains a page number, the vertical coordinate of the top edge of the text box with the largest vertical coordinate in the non-first page is determined as the vertical coordinate of the footer of the non-first page; If the text box with the largest vertical coordinate in the non-first page does not contain a page number, the vertical coordinate of the bottom edge of the text box with the largest vertical coordinate in the non-first page is determined as the vertical coordinate of the footer of the non-first page; Determining the horizontal coordinates of the footer of the non-home page according to the page width of the knowledge document; The footer coordinates of the non-home page are determined according to the vertical coordinates of the footer of the non-home page and the horizontal coordinates of the footer of the non-home page.

8. The method according to claim 1, characterized in that The method of cropping the home page according to the header coordinates and the footer coordinates of the home page in combination with the layout state of the home page, so that the cropped home page only contains the text, includes: If the layout state of the home page is a single-column state, the home page is cropped according to the header coordinates of the home page and the footer coordinates of the home page, so that the cropped home page only contains the text; If the layout state of the homepage is a double-column state, erasing the target vocabulary according to the footer coordinates of the homepage, the page width of the knowledge document and the maximum vertical coordinates of the page, to obtain an erased homepage; Starting from the header coordinates of the first page, the erased first page is cropped according to the page width and the vertical coordinates of the bottom edge of the text box containing the target vocabulary, so that the erased first page only contains the text.

Images