Data processing method and device, electronic equipment and storage medium

Abstract

The application provides a data processing method and device, electronic equipment and a storage medium. The data processing method comprises the following steps: matching an analysis algorithm from a preset analysis algorithm library based on a database address; establishing a connection with the database address based on configuration information and the analysis algorithm; synchronously processing data in a database corresponding to the database address based on a synchronization algorithm to obtain synchronization data; if the synchronization data does not meet a data standard, re-matching an analysis algorithm from the preset analysis algorithm library until the synchronization data meets the data standard; and storing the synchronization data meeting the data standard. The data processing method improves the reliability of grain condition data processing.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the promotion and application of grain informatization, the types of grains that need to be collected when collecting grain information are increasing, and each type of grain corresponds to different manufacturers. This makes it difficult to unify the grain information, and the first step in unification is to extract data from each manufacturer.

[0003] In existing technologies, data from various manufacturers is extracted manually. Since the database addresses and data types differ among manufacturers, different algorithms are needed to parse the data addresses. Algorithm selection is usually done manually, but this is prone to incompatibility issues. Furthermore, manual data extraction is slow and prone to errors. Therefore, the reliability of grain condition data processing methods is poor. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a data processing method and apparatus, an electronic device, and a storage medium to solve the above problems.

[0005] According to a first aspect of the present invention, a data processing method is provided, comprising: matching a parsing algorithm from a preset parsing algorithm library based on a database address; establishing a connection with the database address based on configuration information and the parsing algorithm; performing synchronization processing on data in the database corresponding to the database address based on a synchronization algorithm to obtain synchronized data; if the synchronized data does not conform to a data standard, re-matching a parsing algorithm from the preset parsing algorithm library until the synchronized data conforms to the data standard; and storing the synchronized data that conforms to the data standard.

[0006] In another implementation of the present invention, establishing a connection with the database address based on configuration information and the parsing algorithm includes: parsing the database address using the parsing algorithm to obtain a parsing result; configuring a connection interface according to the configuration information; and establishing a connection with the database address through the connection interface and the parsing result.

[0007] In another implementation of the present invention, establishing a connection with the database address through the connection interface and the parsing result includes: determining whether the connection with the database address is successful; if the connection with the database address fails, re-matching the parsing algorithm from the preset parsing algorithm library to parse the database address until the connection with the database address is successful.

[0008] In another implementation of the present invention, the data processing method further includes: if the synchronized data does not conform to the data standard, or if the connection with the database address fails, the parsing algorithm fails to parse the database address; if all the parsing algorithms in the parsing algorithm library fail to parse the database address, a new parsing algorithm is extended by an extension plugin, and the database address is parsed by the new parsing algorithm until the database address is successfully parsed.

[0009] In another implementation of the present invention, determining whether the synchronized data conforms to the data standard includes: performing text feature analysis on the synchronized data to generate analysis results; determining whether the synchronized data conforms to the data standard based on the analysis results; if the analysis results indicate that the text features of the synchronized data are abnormal, then the synchronized data does not conform to the data standard.

[0010] In another implementation of the present invention, storing synchronized data that conforms to the data standard includes: establishing category labels for the synchronized data according to the data type; and storing the synchronized data in the corresponding label database according to the category labels.

[0011] In another implementation of the present invention, the data processing method further includes: matching configuration information from a preset configuration information database according to the database address; configuring the database address and connection interface using the configuration information; if the configuration information fails to configure the database address and connection interface, re-matching the configuration information to configure the database address and connection interface until the configuration is successful.

[0012] According to a second aspect of the present invention, a data processing apparatus is provided, comprising: a connection module: configured to match a parsing algorithm from a preset parsing algorithm library based on a database address; and further configured to establish a connection with the database address based on configuration information and the parsing algorithm; a processing module: configured to perform synchronization processing on data in the database corresponding to the database address based on a synchronization algorithm to obtain synchronized data; and further configured to, if the synchronized data does not conform to a data standard, re-match a parsing algorithm from the preset parsing algorithm library until the synchronized data conforms to the data standard; and a storage module: configured to store the synchronized data that conforms to the data standard.

[0013] According to a third aspect of the present invention, an electronic device is provided, characterized in that it includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the data processing method as described above.

[0014] According to a fourth aspect of the present invention, a computer storage medium is provided, characterized in that a computer program is stored on the computer storage medium, and when the computer program is executed by a processor, it implements the steps of the data processing method as described in any of the above claims.

[0015] In the data processing method of this invention, configuration information and a parsing algorithm matched from the parsing algorithm library are obtained, a connection is established with the database address, the data in the database is synchronized using a synchronization algorithm to obtain synchronized data, and the synchronized data that meets the data standard is stored. By automatically matching the parsing algorithm from the parsing algorithm library, the data processing efficiency is improved. At the same time, the data quality of the synchronized data is judged according to the data standard, making the synchronized data more accurate and further improving the reliability of grain condition data processing. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. By reading the detailed description of the embodiments below, the advantages and benefits of the solutions will become clear to those skilled in the art. The accompanying drawings are only for illustrating preferred embodiments and are not intended to limit the present invention. In the accompanying drawings:

[0017] Figure 1 This is a flowchart illustrating the steps of a data processing method according to an embodiment of the present invention.

[0018] Figure 2 This is a structural block diagram of a data processing apparatus according to another embodiment of the present invention.

[0019] Figure 3 This is a schematic diagram of the structure of an electronic device according to another embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art should fall within the protection scope of the present invention.

[0021] Figure 1 A flowchart illustrating the steps of a data processing method provided in an embodiment of the present invention is shown below. Figure 1 As shown, this embodiment mainly includes the following steps:

[0022] S101. Based on the database address, match the parsing algorithm from the preset parsing algorithm library.

[0023] For example, the database address is obtained, which is provided by each vendor. A suitable parsing algorithm is matched from a preset parsing algorithm library to parse the database address. The obtained database address can be directly parsed, or it can be stored in an address library for future use. Whether to store the database address is up to the user's needs, and this application does not impose specific restrictions.

[0024] S102. Based on the configuration information and parsing algorithm, establish a connection with the database address.

[0025] For example, the database address is parsed using a parsing algorithm to obtain the parsing result. Simultaneously, configuration information related to the database address is used to configure the settings. A connection is then established with the database address using the parsing result and the configuration information. The configuration information includes interface configuration information and database configuration information. The interface information is used to configure the connection interface, and the database configuration information is used to retrieve data from the database.

[0026] S103. Based on the synchronization algorithm, the data in the database corresponding to the database address is synchronized to obtain synchronized data.

[0027] For example, a pre-defined synchronization algorithm library is provided. Based on user needs, a suitable synchronization algorithm is matched to synchronize the data in the database, resulting in synchronized data. For instance, users can choose to synchronize only a portion of the data or all of the data.

[0028] S104. If the synchronized data does not conform to the data standard, re-match the parsing algorithm from the preset parsing algorithm library until the synchronized data conforms to the data standard.

[0029] For example, a test report is generated based on the acquired synchronization data. The test report is then used to determine whether the synchronization data conforms to data standards. If the synchronization data does not conform to the data standards, a new parsing algorithm needs to be re-matched from a pre-defined parsing algorithm library to resolve the database address. Data standards include: whether there are garbled characters in the synchronization data, and whether there is missing data, etc.

[0030] S105. Store the synchronized data that conforms to the data standard.

[0031] For example, synchronized data that conforms to the data standard is stored in the corresponding folder.

[0032] In the data processing method of this invention, configuration information and a parsing algorithm matched from the parsing algorithm library are obtained, a connection is established with the database address, the data in the database is synchronized using a synchronization algorithm to obtain synchronized data, and the synchronized data that meets the data standard is stored. By automatically matching the parsing algorithm from the parsing algorithm library, the data processing efficiency is improved. At the same time, the data quality of the synchronized data is judged according to the data standard, making the synchronized data more accurate and further improving the reliability of grain condition data processing.

[0033] In another implementation of the present invention, establishing a connection with the database address based on configuration information and the parsing algorithm includes: parsing the database address using the parsing algorithm to obtain a parsing result; configuring a connection interface according to the configuration information; and establishing a connection with the database address through the connection interface and the parsing result.

[0034] For example, the database address is parsed using a matching parsing algorithm to obtain the parsing result. Suitable configuration information is then matched from a preset configuration information database to configure the connection interface and the database. A connection is then established with the database address using the connection interface and the parsing result. If no matching configuration information is found in the configuration information database, a new configuration file is input into the preset configuration information database. The connection interface and the database are then configured using the new configuration file until the configuration is successful.

[0035] In the method of this invention embodiment, by matching appropriate parsing algorithms and configuration information from a preset parsing algorithm library and a preset configuration information library, the repeated input of parsing algorithms and configuration information is avoided, thus saving workload and improving the efficiency of data processing.

[0036] In another implementation of the present invention, establishing a connection with the database address through the connection interface and the parsing result includes: determining whether the connection with the database address is successful; if the connection with the database address fails, re-matching the parsing algorithm from the preset parsing algorithm library to parse the database address until the connection with the database address is successful.

[0037] For example, it is determined whether the connection to the database address is successful. If the connection to the database address fails, it means that the parsing algorithm used has failed to parse the database address. The parsing algorithm is then matched again from the preset parsing algorithm library to parse the database address until the connection to the database address is successful.

[0038] In the method of this embodiment of the invention, if the connection to the database address fails, a new parsing algorithm is automatically matched to parse the database address, thereby realizing automated processing, saving manual workload, and improving the efficiency of data processing.

[0039] In another implementation of the present invention, the data processing method further includes: if the synchronized data does not conform to the data standard, or if the connection with the database address fails, the parsing algorithm fails to parse the database address; if all the parsing algorithms in the parsing algorithm library fail to parse the database address, a new parsing algorithm is extended by an extension plugin, and the database address is parsed by the new parsing algorithm until the database address is successfully parsed.

[0040] For example, when all the parsing algorithms in the preset parsing algorithm library fail to parse the database address, a new parsing algorithm is added through an extension plugin. The new parsing algorithm is then matched and parsed with the database address until the database address is successfully parsed.

[0041] In the method of this invention embodiment, a custom algorithm is implemented through an extension plugin, which makes the data processing method more widely applicable.

[0042] In another implementation of the present invention, determining whether the synchronized data conforms to the data standard includes: performing text feature analysis on the synchronized data to generate analysis results; determining whether the synchronized data conforms to the data standard based on the analysis results; if the analysis results indicate that the text features of the synchronized data are abnormal, then the synchronized data does not conform to the data standard.

[0043] For example, the synchronized data is analyzed by semantic analysis algorithm to obtain the analysis results. The analysis results include whether there is garbled data, missing data, or abnormal data in the synchronized data. If the analysis results indicate that there is an anomaly, the synchronized data does not meet the data standard.

[0044] In the method of this invention embodiment, text feature analysis is performed on the synchronized data to determine whether the synchronized data conforms to the data standard. If it does not conform to the data standard, the data in the database needs to be re-synchronized, thus ensuring the accuracy of the synchronized data.

[0045] In another implementation of the present invention, storing synchronized data that conforms to the data standard includes: establishing category labels for the synchronized data according to the data type; and storing the synchronized data in the corresponding label database according to the category labels.

[0046] For example, synchronized data is categorized and labeled according to its data type, and different categories of data are stored in their corresponding label folders. The data type is set by the user according to their needs; for example, it can be categorized by grain type or processing step, and there are no specific restrictions on the data type.

[0047] In the method of this invention embodiment, by classifying and storing data, it is convenient for users to find the data they need, thus improving the convenience of data use.

[0048] In another implementation of the present invention, the data processing method further includes: matching configuration information from a preset configuration information database according to the database address; configuring the database address and connection interface using the configuration information; if the configuration information fails to configure the database address and connection interface, re-matching the configuration information to configure the database address and connection interface until the configuration is successful.

[0049] For example, based on the database address, corresponding configuration information is matched from a preset configuration information database. This configuration information is then used to configure the connection interface and database. The system determines whether the configuration information has successfully configured the database and connection interface. If configuration fails, configuration information is re-matched from the preset configuration information database and configured again until the connection interface and database are successfully configured. If all configuration information in the preset configuration information database fails to configure the connection interface and database, a new configuration file is input into the preset configuration information database, and the connection interface and database are configured using this new configuration file until successful configuration.

[0050] In the method of this invention embodiment, by storing configuration information in a preset configuration information database, the database address can be automatically configured, saving the time of manual input and matching, improving the efficiency of data processing. At the same time, the configuration information in the preset configuration information database gradually increases, which can meet more configuration needs and improve the accuracy of information configuration.

[0051] According to a second aspect of the present invention, a data processing apparatus 200 is provided, comprising:

[0052] Connection module 201: used to match a parsing algorithm from a preset parsing algorithm library based on the database address; also used to establish a connection with the database address based on configuration information and the parsing algorithm.

[0053] Processing module 202: It is used to synchronize the data in the database corresponding to the database address based on the synchronization algorithm to obtain synchronized data; it is also used to re-match the parsing algorithm from the preset parsing algorithm library if the synchronized data does not conform to the data standard, until the synchronized data conforms to the data standard.

[0054] Storage module 203: Used to store synchronized data that conforms to data standards.

[0055] In the data processing device 200 of this embodiment of the invention, configuration information and a parsing algorithm matched from the parsing algorithm library are obtained, a connection is established with the database address, the data in the database is synchronized using a synchronization algorithm to obtain synchronized data, and the synchronized data that meets the data standard is stored. By automatically matching the parsing algorithm from the parsing algorithm library, the data processing efficiency is improved. At the same time, the data quality of the synchronized data is judged according to the data standard, making the synchronized data more accurate and further improving the reliability of grain condition data processing.

[0056] In another implementation of the present invention, the connection module 201 is further configured to parse the database address using a parsing algorithm to obtain a parsing result; configure a connection interface according to configuration information; and establish a connection with the database address through the connection interface and the parsing result.

[0057] In another implementation of the present invention, the connection module 201 is further configured to determine whether the connection with the database address is successful; if the connection with the database address fails, the connection module 201 re-matches the parsing algorithm from the preset parsing algorithm library to parse the database address until the connection with the database address is successful.

[0058] In another implementation of the present invention, the connection module 201 is also used to synchronize data that does not conform to the data standard, or to fail to connect to the database address. If the data does not conform to the data standard, or the connection to the database address fails, the parsing algorithm fails to parse the database address. If all the parsing algorithms in the parsing algorithm library fail to parse the database address, a new parsing algorithm is extended by an extension plugin, and the database address is parsed by the new parsing algorithm until the database address is successfully parsed.

[0059] In another implementation of the present invention, the processing module 202 is further configured to perform text feature analysis on the synchronized data and generate analysis results; determine whether the synchronized data conforms to the data standard based on the analysis results; if the analysis results indicate that the text features of the synchronized data are abnormal, then the synchronized data does not conform to the data standard.

[0060] In another implementation of the present invention, the storage module 203 is further configured to establish category labels for the synchronized data according to the data type; and store the synchronized data into the corresponding label database according to the category labels.

[0061] In another implementation of the present invention, the connection module 201 is further configured to match configuration information from a preset configuration information database according to the database address; configure the database address and connection interface with the configuration information; if the configuration information fails to configure the database address and connection interface, the configuration information is re-matched to configure the database address and connection interface until the configuration is successful.

[0062] The apparatus of this embodiment is used to implement the corresponding methods in the foregoing method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. Furthermore, the functional implementation of each module in the apparatus of this embodiment can be referred to the description of the corresponding part in the foregoing method embodiments, which will also not be repeated here.

[0063] like Figure 3 As shown, the electronic device 300 may include: a processor 301, a memory 303, a communication bus 304, and a communication interface 305.

[0064] in:

[0065] The processor 301, memory 303, and communication interface 305 communicate with each other via communication bus 304.

[0066] Communication interface 305 is used to communicate with other electronic devices or servers.

[0067] The processor 301 is used to execute the program 302, specifically the steps of any of the data processing methods in the above embodiments.

[0068] Specifically, program 302 may include program code that includes computer operation instructions.

[0069] Processor 301 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0070] Memory 303 is used to store program 302. Memory 303 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0071] Specifically, program 302 can be used to cause processor 301 to execute steps to implement any of the data processing methods described in the embodiments. The specific implementation of each step in program 302 can be found in the corresponding descriptions of the steps and units executed by any of the data processing methods described above, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments.

[0072] An exemplary embodiment of this application also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods of various embodiments of this application.

[0073] The methods described above according to embodiments of the present invention can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored on a remote recording medium or a non-transitory machine-readable medium and subsequently stored on a local recording medium, downloaded via a network. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0074] Specific embodiments of the invention have now been described. Other embodiments are within the scope of the appended claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing can be advantageous.

[0075] It should be noted that all directional indicators (such as up, down, left, right, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicator will also change accordingly.

[0076] In the description of this invention, the terms "first" and "second" are used only for convenience in describing different components or names, and should not be construed as indicating or implying a sequential relationship, relative importance, or implicitly specifying the number of technical features indicated. Thus, a feature defined with "first" and "second" may explicitly or implicitly include at least one of that feature.

[0077] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0078] It should be noted that although specific embodiments of the present invention have been described in detail with reference to the accompanying drawings, this should not be construed as limiting the scope of protection of the present invention. Various modifications and variations that can be made by those skilled in the art without inventive effort within the scope described in the claims still fall within the scope of protection of the present invention.

[0079] The examples of the embodiments of the present invention are intended to concisely illustrate the technical features of the embodiments of the present invention, so that those skilled in the art can intuitively understand the technical features of the embodiments of the present invention, and are not intended to be an improper limitation of the embodiments of the present invention.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, comprising: Based on the database address, a parsing algorithm is matched from a pre-defined parsing algorithm library; Based on the configuration information and the parsing algorithm, a connection is established with the database address; The data in the database corresponding to the database address is synchronized using a synchronization algorithm to obtain synchronized data. The synchronized data is subjected to text feature analysis to generate analysis results, and the synchronized data is judged to meet the data standard based on the analysis results. If the analysis results indicate that the text features of the synchronized data are abnormal, then the synchronized data does not meet the data standard. If the synchronized data does not conform to the data standard, the parsing algorithm is re-matched from the preset parsing algorithm library until the synchronized data conforms to the data standard; Store the synchronized data that conforms to the data standard; The step of establishing a connection with the database address based on the configuration information and the parsing algorithm includes: parsing the database address using the parsing algorithm to obtain a parsing result; configuring a connection interface according to the configuration information; and establishing a connection with the database address through the connection interface and the parsing result.

2. The method as described in claim 1, characterized in that, The step of establishing a connection with the database address through the connection interface and the parsing result includes: Determine whether the connection to the database address was successful; If the connection to the database address fails, a new parsing algorithm is selected from the preset parsing algorithm library to parse the database address until a successful connection to the database address is established.

3. The method as described in claim 1 or 2, characterized in that, Also includes: If the synchronized data does not conform to the data standard, or fails to connect to the database address, then the parsing algorithm fails to parse the database address. If all the parsing algorithms in the preset parsing algorithm library fail to parse the database address, a new parsing algorithm is added through an extension plugin, and the database address is parsed using the new parsing algorithm until the database address is successfully parsed.

4. The method as described in claim 1, characterized in that, The step of storing synchronized data that conforms to the data standard includes: Establish category labels for the synchronized data based on its data type; The synchronized data is stored in the corresponding tag database according to the category tag.

5. The method as described in claim 1, characterized in that, Also includes: Based on the database address, match the configuration information from the preset configuration information database; The database address and connection interface are configured using the configuration information. If the configuration information fails to configure the database address and the connection interface, the configuration information is re-matched and configured for the database address and the connection interface until the configuration is successful.

6. A data processing apparatus, comprising: Connection module: Used to match parsing algorithms from a preset parsing algorithm library based on the database address; It is also used to establish a connection with the database address based on the configuration information and the parsing algorithm; Processing module: used to synchronize data in the database corresponding to the database address based on a synchronization algorithm to obtain synchronized data; also used to perform text feature analysis on the synchronized data, generate analysis results, and determine whether the synchronized data conforms to the data standard based on the analysis results. If the analysis results indicate that the text features of the synchronized data are abnormal, then the synchronized data does not conform to the data standard. If the synchronized data does not conform to the data standard, a new parsing algorithm is selected from the preset parsing algorithm library until the synchronized data conforms to the data standard. Storage module: Used to store synchronized data that conforms to the data standard; The connection module is further configured to parse the database address using the parsing algorithm to obtain a parsing result; configure a connection interface according to the configuration information; and establish a connection with the database address through the connection interface and the parsing result.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the data processing method as described in any one of claims 1 to 5.

8. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1 to 5.

Images