A data processing method, medium and device based on JSOUP

Abstract

The application discloses a data processing method, medium and device based on JSOUP, and comprises the following steps: S01, remote connection with a page is established, and the page is acquired; S02, information of the current page is parsed, and searching is performed based on a plurality of preset positioning elements; S03, a table is generated according to elements collected by the plurality of positioning elements; and S04, automatic matching into a database is performed according to the table. The data processing method, medium and device based on JSOUP provided by the application accelerate the data updating speed, improve the personnel data searching efficiency, increase the data searching updating frequency, and greatly reduce the probability of occurrence of the problem of data information mismatch caused by untimely data updating.

Description

TECHNICAL FIELD

[0001] The application relates to rock experimental equipment, in particular to a data processing method based on JSOUP, a medium and a device. BACKGROUND

[0002] Sometimes some public data information is maintained in a table on a website, data checking is carried out, and the present stage processing mode adopts background manual checking processing and manual maintenance of the information, but mistakes are easily caused, and the manual cost expenditure caused by checking is also relatively huge. SUMMARY

[0003] The application aims to provide a data processing method based on JSOUP, a medium and a device, and ensure efficient data crawling and warehousing.

[0004] In order to achieve the above-mentioned purpose, the application provides the following technical scheme: a data processing method based on JSOUP, comprising the following steps:

[0005] S01, a remote connection with a page is established, and the page is acquired;

[0006] S02, information of the current page is parsed, and searching is carried out based on a plurality of positioning elements;

[0007] S03, a table is generated according to a collection of elements collected by the plurality of positioning elements;

[0008] S04, automatic matching warehousing is carried out according to the table.

[0009] As a preferred, in the step S01, if login verification is needed when the remote connection is established, then the cookie needs to be taken together, and then the information is acquired.

[0010] As a preferred, the steps S02 and S03 are processed as follows:

[0011] S11, according to jsoup document.getElementsByClass(“layui-table”), each positioning element is found to be a table that needs to be processed;

[0012] S12, according to the sequence of the table header element content, the attribute name is sequentially put into the array;

[0013] S13, each row tr is acquired by document.getElementsByClass(“tr”) circulation, and an entity class is created, and the entity class is sequentially stored in the collection;

[0014] S14, continue to get the text content of the td element in each row tr through the document.getElementsByClass("td") loop after the step S13 ends.

[0015] As preferred, the table header element is a row of labels in the table, and the first row of the row column is the table header data defining the entity class and the positioning element in the class.

[0016] As preferred, the detailed processing steps in the step S13 are as follows:

[0017] S21, find the entity class set in the nth row, find the nth-1 entity, and assign the text content in the td to the corresponding attribute value in the entity.

[0018] S22, perform loop processing according to the number of merged rows, if the first td in the nth row is merged, assign the text content of the current td to the bankLevel of the nth-1, n, and n+1 entities in the entity class set.

[0019] As preferred, the other td elements outside the nth-1 entity in the nth row in the step S22 are missing, and it is necessary to determine whether the attributes in the BankAcountInfoVo have been assigned:

[0020] No, then sequentially input the content;

[0021] Yes, the corresponding attribute in the BankAcountInfoVo in the current params is not empty, and the next attribute value is automatically skipped.

[0022] As preferred, the positioning element includes an html element, a label tag, a label id, and a label class.

[0023] In the above technical solution, the data processing method, medium and device based on JSOUP provided by the application have the following beneficial effects: the page is parsed by jsoup, no matter how the table content changes, as long as the table header remains unchanged and the template content remains unchanged, the page information can be easily captured and encapsulated into an entity class for data storage, which accelerates the data updating speed, improves the personnel data retrieval efficiency, increases the data retrieval updating frequency, and greatly reduces the probability of the occurrence of the problem of data information mismatch caused by the data updating not in time. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only represent some embodiments described in the present application, and other drawings can be obtained by those skilled in the art based on these drawings.

[0025] Figure 1 The flow structure schematic diagram provided for the embodiments of the present application;

[0026] Figure 2 The code schematic diagram of positioning element analysis and classification provided for the embodiments of the present application;

[0027] Figure 3 The table structure schematic diagram provided for the embodiments of the present application.

[0028] Figure 4 The code schematic diagram of database maintenance provided for the embodiments of the present application. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only represent some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0030] Embodiment 2

[0031] As shown in the following, a data processing method based on JSOUP includes the following steps: Figures 1-2

[0032] S01, a remote connection with a page is established to obtain the page;

[0033] S02, information of the current page is parsed, and a plurality of positioning elements are searched based on the preset positioning elements;

[0034] S03, a table is generated according to the elements set collected by the plurality of positioning elements;

[0035] S04, the table is automatically matched into the database.

[0036] Specifically, the positioning elements include html elements, tags, tag ids and tag classes. In the step S01 of establishing a remote connection, if login verification is needed, the cookie needs to be taken in, and then the information is obtained.

[0037] Further, the processing steps of the step S02 and the step S03 are as follows: ​

[0038] S11, find the table to be processed according to jsoup in document.getElementsByClass("layui-table"); make each positioning element as layui-table;

[0039] S12, according to the order of the table header element content to put the attribute name into the array one by one;

[0040] S13, through document.getElementsByClass("tr") to get each row tr, and create an entity class, and store the entity class in the collection one by one;

[0041] S14, after step S13, continue to cycle through document.getElementsByClass("td") to get the text content of the td element in each row tr.

[0042] Further, the table header element is a row of labels in the table, and the first row of the column is the table header data definition entity class and the positioning element in the class.

[0043] As a further provided embodiment of the present application, the detailed processing steps in step S13 are:

[0044] S21, find the entity class collection of the nth row to find the nth-1 entity, and assign the text content in the td to the corresponding attribute value in the entity;

[0045] S22, according to the number of merged rows, if the first td of the nth row is merged, assign the text content of the current td to the bankLevel of the nth-1, nth, and nth+1 entities in the entity class collection.

[0046] In the above embodiment, in step S22, other td elements outside the nth-1 entity in the nth row are missing, and it is necessary to determine whether the attributes in BankAcountInfoVo have been assigned:

[0047] No, then input the content in order;

[0048] Yes, the corresponding attribute in BankAcountInfoVo in the current params is not empty, and the next attribute value is automatically skipped.

[0049] At this time, all values will be paired correctly, and finally the obtained collection will be stored in the database.

[0050] Embodiment 2

[0051] As Figure 3 and Figure 4As shown, the method provided in Embodiment 1 is executed to acquire page information when a remote connection is established, to acquire a Document after the connection is established, and to parse html elements and html tag, id or class positioning required table table corresponding elements element set in the page information. The code is as follows:

[0052] Elementse=document.getElementsByClass(TABLE_CLASS_NAME);

[0053] The steps of parsing the table element set are as follows:

[0054] 1) Acquire the table header, which is the attribute name of the entity class;

[0055] 2) Define the entity class and the attribute name in the class according to the table header data;

[0056] 3) Put the attribute names into the array in sequence according to the table header element content acquisition order.

[0057] As shown, the table header of the set is bank type, expiration date, face value, and acceptance bank, and we need to store the table content into the database for data maintenance. Therefore, the entity class needs to be defined to accept the parsed information, and then the entity class BankAcountInfoVo is defined. Figure 2 The attribute thereof is as shown.

[0058] The database maintenance attribute name and table header content keyword mapping relationship is as follows: bankLevel-bank type, expireDate-expiration date, rates-face value (ten thousand yuan), and bankNames-acceptance bank name. The advantage of database maintenance is that when the table is changed or other content is parsed, the data can be modified to achieve the purpose. Each row will generate a BankAcountInfoVo bank information display class, and finally the set is written into the database in batch. Figure 4

[0059] ​The parsing table header will write the attribute names of bank type, expiration date, face value, and accepting bank in order into the array params. That is [bankLevel, expireDate, rates, bankNames]. Then the data in the table body is written into the corresponding attribute in the params array according to the order of data occurrence. For example, the first row, we create a BankAcountInfoVo, a type of bank appears in the first column, and the attribute array is bankLevel. We write the data of a type of bank into bankLevel. The second column is the expiration date, which corresponds to the expireDate in the array. We write it into expireDate. According to the corresponding coordinates, the following data is assigned to the corresponding attribute using the reflection mechanism.

[0060] Among them, the table common data may exist in the cross-row cell merging situation, so special processing is needed. Through document.getElementsByClass(“td”) loop to get the text content of each row tr td element, if in the n th tr tag cycle found that td exists cross-row cell merging scene, then mark, and according to the current tr sequence, that is, the n th row, find the entity class set to find the n-1 th entity, assign the text content in the td to the corresponding attribute value in the entity (if the td is the first type of bank, assign the type of bank to params[0]bankLevel, if it is the second, assign the processing date of 2022-09-10 to params[1]expireDate), and according to the number of merged rows, the first td in the n th row is merged for three rows, and the bankLevel of the n-1 th, n th, and n+1 th entities in the entity class set is assigned to the text content of the current td. And because of cross-row merging, except for the first row, other rows will have missing td elements. At this time, attribute assignment according to the order of td appearance is not feasible, so we need to determine whether the attribute in BankAcountInfoVo has been assigned. If not, proceed to the next attribute value judgment. At this time, all values will be paired correctly. Finally, the obtained set is stored in the database in batches.

[0061] Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In a software embodiment, the methods can be tangibly embodied in a machine-readable storage medium having stored thereon instructions that can be used to program a computer to perform any of the operations described herein. The software implementation can be for example, in the form of a computer program product. The software implementation can be implemented in a centralized fashion in one computer system or network, or be distributed over a network such computer systems, some of which can implement the

[0062] The present application is described in relation to flow diagrams and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It is to be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks.

[0063] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks.

[0064] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks.

[0065] The principles and implementation of the present application are described in the detailed description of the application. The above description of the embodiments is only intended to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation and application range can be changed; therefore, the content of the description should not be understood as a limitation of the present application.

[0066] The embodiment of the present application further provides a specific implementation of an electronic device capable of implementing all steps in the method in the above-mentioned embodiment, and the electronic device specifically comprises the following contents:

[0067] a processor, a memory, a communications interface and a bus;

[0068] The processor, the memory and the communications interface can communicate with each other through the bus.

[0069] The processor is configured to call a computer program in the memory, and the processor executes the computer program to implement all steps in the method in the above-mentioned embodiment.

[0070] The embodiment of the present application further provides a computer readable storage medium capable of implementing all steps in the method in the above-mentioned embodiment, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement all steps in the method in the above-mentioned embodiment.

[0071] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, for hardware + program embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. Although the embodiments in this specification provide the method operation steps as shown in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible execution order among many steps and does not represent the only execution order. In actual device or terminal product execution, the methods can be executed in the order shown in the embodiments or drawings or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in the process, method, product, or apparatus that includes said elements is not excluded. For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing the embodiments of this specification, the functions of each module can be implemented in one or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms. This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which are executable by the processor of the computer or other programmable data processing device, produce instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0072] Those skilled in the art will appreciate that embodiments of the present specification can be devised for a method, a system, or a computer program product. Accordingly, embodiments of the present specification can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present specification can be in the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk memory, CD-ROM, optical storage medium, etc.) embodying computer usable program code. Each of the various embodiments of the present specification is described in a progressive manner, and reference can be made to other embodiments for the same or similar parts. Each embodiment focuses on the differences from other embodiments. In particular, for system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and reference can be made to the description of the method embodiments. In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the embodiments of the present specification.

[0073] In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. In addition, those skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction. The above is only an embodiment of the embodiments of the present specification and is not intended to limit the embodiments of the present specification. The embodiments of the present specification can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the embodiments of the present specification shall be included in the scope of the claims of the embodiments of the present specification.

Claims

1. A data processing method based on JSOUP, characterized in that, Includes the following steps: S01. Establish a remote connection with the page and retrieve the page; S02. Analyze the information of the current page and perform a search based on multiple preset positioning elements; S03. Generate a table based on the collection of elements from the multiple positioning elements; S04. Automatically match and store data according to the table; The processing steps S02 and S03 are as follows: S11. Based on document.getElementsByClass("layui-table") in jsoup, make each of the positioned elements layui-table to find the table that needs to be processed. S12. According to the order in which the content of the table header elements is obtained, put the attribute names into the array in sequence; S13. Use document.getElementsByClass("tr") to get each line of tr, create an entity class, and store the entity classes in the collection in sequence; S14. After step S13 is completed, continue to use document.getElementsByClass("td") to loop and get the text content of the td elements in each line of tr; The detailed processing steps in step S13 are as follows: S21. In the nth row, find the (n-1)th entity in the entity class set, and assign the text content in the td to the corresponding attribute value of the entity. S22. Process the data in a loop according to the number of rows to be merged. If the first td in the nth row is merged, assign the current td's text content to the bankLevel of the (n-1), n, and n+1th entities in the entity class set. In step S22, if other td elements besides the (n-1)th entity in the nth row are missing, it is necessary to determine whether the attributes in BankAcountInfoVo have been assigned values: If not, proceed with the content input in sequence; If yes, then the attribute in the corresponding BankAcountInfoVo within the current params is not empty, and the evaluation of the next attribute value is automatically skipped.

2. The data processing method based on JSOUP according to claim 1, characterized in that, When establishing a remote connection in step S01, if login verification is required, the cookie needs to be included before obtaining the information.

3. The data processing method based on JSOUP according to claim 1, characterized in that, The positioning elements include HTML elements, tags, tag IDs, and tag classes.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the JSOUP-based data processing method according to any one of claims 1 to 3.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the JSOUP-based data processing method according to any one of claims 1 to 3.

Images