Data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a storage medium, and relates to the technical field of data processing.The method comprises the steps that to-be-processed target data is obtained; preprocessing the target data according to a preset processing rule; performing feature extraction on the preprocessed target data to obtain a plurality of target features; and performing classification based on the candidate features, and performing uplink storage on the target data according to classification. Therefore, through classification, the target data can be separately stored according to the classification, so that data acquisition can be directly performed from the corresponding classification according to requirements when data acquisition is performed from the block chain subsequently, and the use experience of a user is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] In the digital age, text data is one of the main forms of information transmission and knowledge accumulation. Its recognition and processing technology is of great significance to improving information processing efficiency and mining data value. Text data has the characteristics of unstructured, semantically rich, and diverse expressions. Unstructured means that data does not have a fixed format and structure like tabular data in a database, which makes the processing of text data more complicated. Semantically rich means that the text contains rich information and meaning, such as emotional tendencies, subject classifications, entity relationships, etc. Diverse expressions mean that text can exist in different languages, styles, and formats, which increases the difficulty of processing. Summary of the invention

[0003] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, an object of the present disclosure is to provide a data processing method.

[0005] A second objective of the present disclosure is to provide a data processing device.

[0006] A third objective of the present disclosure is to provide an electronic device.

[0007] A fourth object of the present disclosure is to provide a non-transitory computer-readable storage medium.

[0008] A fifth object of the present disclosure is to provide a computer program product.

[0009] To achieve the above-mentioned purpose, the first aspect of the present disclosure proposes a data processing method, including: obtaining target data to be processed; preprocessing the target data according to preset processing rules; extracting features from the preprocessed target data to obtain multiple target features; classifying based on the candidate features, and storing the target data on a chain according to the classification.

[0010] According to one embodiment of the present disclosure, the preprocessed target data is subjected to feature extraction to obtain multiple target features, including: performing statistical feature extraction on the target data to obtain a first feature set, and performing deep learning feature extraction on the target data to obtain a second feature set; performing feature selection and feature optimization on the first feature set and the second feature set to obtain multiple target features.

[0011] According to one embodiment of the present disclosure, the feature selection and feature optimization of the first feature set and the second feature set include: performing word frequency statistics on the candidate features in the first feature set and the second feature set to obtain the word frequency of each candidate feature; and removing candidate features whose word frequency is less than a word frequency threshold.

[0012] According to one embodiment of the present disclosure, the method further includes: generating a feature vector based on the target feature.

[0013] According to one embodiment of the present disclosure, before storing the target data on a chain according to classification, the process further includes: converting the format of the target data to generate target data in a preset format.

[0014] According to one embodiment of the present disclosure, before storing the target data on-chain according to classification, the process further includes: encrypting the target data.

[0015] According to an embodiment of the present disclosure, the preset processing rules are one or more of text cleaning, word segmentation and stop word removal.

[0016] To achieve the above-mentioned purpose, a second aspect of the present disclosure provides a data processing device, including:

[0017] According to one embodiment of the present disclosure, the data processing device is also used for: an acquisition module for acquiring target data to be processed; a preprocessing module for preprocessing the target data according to preset processing rules; an extraction module for extracting features from the preprocessed target data to obtain multiple target features; a classification module for classifying based on the candidate features and storing the target data on the chain according to the classification.

[0018] To achieve the above-mentioned purpose, the third aspect embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to implement the data processing method as described in the first aspect embodiment of the present disclosure.

[0019] To achieve the above-mentioned purpose, the fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to implement the data processing method as described in the first aspect embodiment of the present disclosure.

[0020] To achieve the above-mentioned purpose, the fifth aspect of the present disclosure proposes a computer program product, including a computer program, which is used to implement the data processing method as described in the first aspect of the present disclosure when executed by a processor.

[0021] Therefore, through classification, the target data can be stored separately according to the category, which is convenient for subsequent data acquisition from the blockchain. Data can be directly acquired from the corresponding category according to needs, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a schematic diagram of a data processing method according to an embodiment of the present disclosure;

[0023] Figure 2 is a schematic diagram of another data processing method according to an embodiment of the present disclosure;

[0024] Figure 3 is a schematic diagram of a data processing device according to an embodiment of the present disclosure;

[0025] Figure 4 is a schematic diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Embodiments of the present disclosure are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0027] The acquisition, storage, use, and processing of data in the technical solution disclosed in this disclosure are in compliance with the relevant provisions of relevant laws and regulations.

[0028] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be considered as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0029] Figure 1 is a schematic diagram of a data processing method according to an embodiment of the present disclosure, such as Figure 1 As shown, the data processing method includes the following steps:

[0030] S101, obtaining target data to be processed.

[0031] The data processing method of the embodiment of the present application can be applied to the scenario of data processing. The executor of the data processing of the embodiment of the present application can be the data processing device of the embodiment of the present application, and the data processing device can be set on an electronic device.

[0032] It should be noted that the target data in the embodiments of the present disclosure may be of various types, and no limitation is made herein. For example, the target data may be audio data, text data, and the like.

[0033] S102, pre-processing the target data according to preset processing rules.

[0034] Preprocessing is the process of cleaning, transforming and preparing raw data in data analysis or machine learning tasks. The goal of preprocessing is to make the data suitable for subsequent modeling or analysis, eliminate noise, missing values, outliers and other problems, and standardize the data format and structure to make it easier to understand and process.

[0035] The preprocessing in the embodiments of the present disclosure may be various, and no limitation is made here. For example, the preprocessing may include one or more of missing value processing, outlier detection and processing, standardization / normalization, etc.

[0036] It should be noted that the preset processing rules in the embodiments of the present disclosure can be set according to actual design requirements or data processing scenarios. By preprocessing the target data according to the preset processing rules, the efficiency and accuracy of subsequent output processing can be improved.

[0037] S103, extracting features from the preprocessed target data to obtain multiple target features.

[0038] In this embodiment, there may be multiple methods for extracting features from target data, and no limitation is made here.

[0039] In a possible implementation, the target data may be processed according to a preset feature extraction algorithm to obtain multiple target features. It should be noted that the feature extraction algorithm may be changed according to actual design requirements and data processing scenarios.

[0040] In another possible implementation, the target data may be processed by a feature extraction model to generate multiple target features. The feature extraction model is trained in advance and stored in a storage space of the electronic device to facilitate retrieval and use when needed.

[0041] S104, classify based on candidate features, and store the target data on-chain according to the classification.

[0042] In the embodiments of the present disclosure, first, target data to be processed is obtained, then the target data is preprocessed according to a preset processing rule, and then the preprocessed target data is subjected to feature extraction to obtain a plurality of target features. Finally, classification is performed based on candidate features, and the target data is stored on the chain according to the classification. Thus, through classification, the target data can be stored separately according to the classification, which is convenient for subsequent data acquisition from the blockchain. When data is acquired, it can be directly obtained from the corresponding classification according to the requirements, improving the user experience.

[0043] It should be noted that the preset processing rule is one or more of text cleaning, word segmentation, and stop word removal. For example, the preset processing rule may include text cleaning (removing irrelevant characters, correcting errors, etc.), word segmentation (decomposing text into words or lexical units), stop word removal (deleting common meaningless words such as "de", "he", "shi", etc.), and stemming or lemmatization (reducing words to their basic forms). This lays a foundation for subsequent feature extraction and classification model construction.

[0044] In the above embodiments, the preprocessed target data is subjected to feature extraction to obtain a plurality of target features. It can also be Figure 2 Further explained that the method includes:

[0045] S201, perform statistical feature extraction on the target data to obtain a first feature set, and perform deep learning feature extraction on the target data to obtain a second feature set.

[0046] It should be noted that statistical feature extraction is to extract meaningful quantitative features from the original data for subsequent analysis or modeling. These features are usually based on statistics such as mean, standard deviation, maximum value, minimum value, median, etc., and can also include some advanced statistics such as skewness, kurtosis, quartiles, etc. Statistical feature extraction can help us understand the basic characteristics of the data and simplify complex data sets.

[0047] Deep learning feature extraction is the process of automatically learning abstract features from the original data using a deep neural network. Compared with traditional statistical feature extraction, deep learning feature extraction has the following advantages:

[0048] 1. Automation: There is no need to manually design features, and the network can automatically learn useful features from the original data. 2. Hierarchy: The deep learning model can learn multi-layer abstract features, from low-level to high-level, gradually improving the abstract degree of the features. 3. Nonlinearity: The deep learning model can capture complex nonlinear relationships and adapt to diverse data patterns. 4. Modularity: Different network layers focus on different features and can be flexibly combined to achieve end-to-end learning.

[0049] Deep learning feature extraction is widely used in computer vision, natural language processing, speech recognition and other fields. For example, in image recognition, convolutional neural network (CNN) can automatically learn spatial features from images; in natural language processing, recurrent neural network (RNN) can automatically learn semantic features from text.

[0050] S202, performing feature selection and feature optimization on the first feature set and the second feature set to obtain multiple target features.

[0051] In the embodiment of the present disclosure, word frequency statistics may be performed on the candidate features in the first feature set and the second feature set to obtain the word frequency of each candidate feature, and then the candidate features whose word frequency is less than the word frequency threshold are removed.

[0052] In the disclosed embodiment, firstly, statistical feature extraction is performed on the target data to obtain a first feature set, and deep learning feature extraction is performed on the target data to obtain a second feature set, and then feature selection and feature optimization are performed on the first feature set and the second feature set to obtain multiple target features. Thus, by extracting the target data through two feature extraction methods, statistical feature extraction and deep learning feature extraction, the accuracy and effectiveness of the target features finally generated can be improved, providing an accurate data basis for subsequent data classification.

[0053] After the classification is completed, the target data can also be converted to generate target data in a preset format. The disclosed solution designs a data conversion and verification mechanism to ensure that the text data can be converted to a format according to the requirements of the target system, and to verify the integrity and consistency of the data through verification codes, hash values, etc.

[0054] In the disclosed embodiment, it is extremely important to protect the security of data during the directory chain exchange process. Data encryption can effectively prevent data leakage and tampering. In one embodiment of the present disclosure, the target data can be encrypted. Specifically, a digital signature algorithm and a hash algorithm based on blockchain can be used. The digital signature algorithm is a technology used to verify the source and integrity of a message. It includes a private key and a public key for encrypting and decrypting information. The digital signature is mainly used to verify the legality and authenticity of data exchange. The hash algorithm can convert messages of any length into a fixed-length hash value. It has the characteristics of irreversibility and anti-collision, and is mainly used to protect the security and privacy of data exchange.

[0055] In the embodiment of the present disclosure, the target data is stored on the chain according to the classification, which can achieve the data being tamper-proof and traceable, including the following processes:

[0056] 1. Smart contract writing: define the rules and logic for uploading data to the chain.

[0057] 2. Data packaging: Pack text data into transactions in a specific format.

[0058] 3. Exchange submission: Submit the exchange to the blockchain network, confirm it through the consensus mechanism and record it on the chain.

[0059] 4. Query and verification: Provide an interface for users to query on-chain data and verify its authenticity and integrity.

[0060] Corresponding to the data processing methods provided in the above-mentioned embodiments, an embodiment of the present disclosure further provides a data processing device. Since the data processing device provided in the embodiment of the present disclosure corresponds to the data processing methods provided in the above-mentioned embodiments, the implementation methods of the above-mentioned data processing methods are also applicable to the data processing device provided in the embodiment of the present disclosure and will not be described in detail in the following embodiments.

[0061] Figure 3 is a schematic diagram of a data processing device according to an embodiment of the present disclosure. Figure 3 As shown, the data processing device 300 includes: an acquisition module 310, a preprocessing module 320, an extraction module 330 and a classification module 340.

[0062] The acquisition module 310 is used to acquire target data to be processed.

[0063] The preprocessing module 320 is used to preprocess the target data according to preset processing rules.

[0064] The extraction module 330 is used to perform feature extraction on the preprocessed target data to obtain multiple target features.

[0065] The classification module 340 is used to classify based on the candidate features and store the target data on the chain according to the classification.

[0066] In one possible implementation, the extraction module 330 is also used to: perform statistical feature extraction on the target data to obtain a first feature set, and perform deep learning feature extraction on the target data to obtain a second feature set; perform feature selection and feature optimization on the first feature set and the second feature set to obtain multiple target features.

[0067] In a possible implementation, the extraction module 330 is further used to: perform word frequency statistics on the candidate features in the first feature set and the second feature set to obtain the word frequency of each candidate feature; and remove the candidate features whose word frequency is less than the word frequency threshold.

[0068] In a possible implementation, the extraction module 330 is further configured to generate a feature vector based on the target feature.

[0069] In a possible implementation, the classification module 340 is further used to: perform format conversion on the target data to generate target data in a preset format.

[0070] In a possible implementation, the classification module 340 is further used to encrypt the target data.

[0071] In a possible implementation, the preset processing rules are one or more of text cleaning, word segmentation, and stop word removal.

[0072] Therefore, through classification, the target data can be stored separately according to the category, which is convenient for subsequent data acquisition from the blockchain. Data can be directly acquired from the corresponding category according to needs, thereby improving the user experience.

[0073] In order to implement the above embodiment, the present disclosure further provides an electronic device 400, Figure 4 is a schematic diagram of an electronic device according to an embodiment of the present disclosure, such as Figure 4 As shown, the electronic device 400 includes: a processor 401 and a memory 402 connected to the processor in communication, the memory 402 stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor 401 to implement the present disclosure. Figure 1-Figure 2 The data processing method of the embodiment.

[0074] In order to implement the above embodiments, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to implement the above embodiments. Figure 1-Figure 2 The data processing method of the embodiment.

[0075] In order to implement the above embodiments, the present disclosure also provides a computer program product, including a computer program. When the computer program is executed by a processor, the computer program implements the above embodiments. Figure 1-Figure 2 The data processing method of the embodiment.

[0076] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0077] The present application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0078] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0079] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0080] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0081] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that contains, stores, communicates, propagates or transmits a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0082] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0083] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0084] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0085] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A data processing method, characterized in that: include: Obtain target data to be processed; Preprocessing the target data according to preset processing rules; Perform feature extraction on the preprocessed target data to obtain multiple target features; Classification is performed based on the candidate features, and the target data is stored on the chain according to the classification.

2. The method according to claim 1, characterized in that: The preprocessed target data is subjected to feature extraction to obtain multiple target features, including: Performing statistical feature extraction on the target data to obtain a first feature set, and performing deep learning feature extraction on the target data to obtain a second feature set; Feature selection and feature optimization are performed on the first feature set and the second feature set to obtain multiple target features.

3. The method according to claim 2, characterized in that The performing feature selection and feature optimization on the first feature set and the second feature set includes: Performing word frequency statistics on the candidate features in the first feature set and the second feature set to obtain the word frequency of each candidate feature; Remove candidate features whose word frequency is less than the word frequency threshold.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: A feature vector is generated based on the target feature.

5. The method according to claim 1, characterized in that Before storing the target data on the chain according to the classification, the method further includes: The target data is format-converted to generate target data in a preset format.

6. The method according to claim 1, characterized in that Before storing the target data on the chain according to the classification, the method further includes: The target data is encrypted.

7. The method according to claim 1, characterized in that The preset processing rules are one or more of text cleaning, word segmentation and stop word removal.

8. A data processing device, characterized in that: include: An acquisition module, used to acquire target data to be processed; A preprocessing module, used for preprocessing the target data according to a preset processing rule; An extraction module is used to extract features from the preprocessed target data to obtain multiple target features; The classification module is used to classify based on the candidate features and store the target data on the chain according to the classification.

9. An electronic device, characterized in that: Including memory. processor; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images