Data processing method based on AI model

Abstract

The invention relates to the field of data processing, and discloses a data processing method based on an AI model, and the method comprises the steps: firstly collecting data information, and packaging the data information into a data information set; performing data set classification, cleaning useless data information, performing compression arrangement, and performing a first preprocessing step to obtain first processing data information; binding a data source mark, converting the data source mark into a data feature vector, and performing a second preprocessing step to obtain a data feature portrait; performing comparison training on the data feature portrait and the first processing data information, and adjusting AI model training data in the first preprocessing step and the second preprocessing step to obtain an AI data processing model; the problems that an existing data processing model is slow in data processing process and low in efficiency, the existing data processing model is unstable, and the performance and generalization ability of an AI large model are often limited are solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data processing method based on an AI model. Background Art

[0002] At present, with the continuous growth of data processing volume and the improvement of computing power of AI big models, the application of AI big models in various fields is becoming more and more extensive.

[0003] However, in actual applications, due to the increasing diversity and complexity of data, the performance and generalization capabilities of large AI models are often limited; therefore, how to optimize the performance and generalization capabilities of large AI models and improve the stability and reliability of the models is an important issue facing the current field of artificial intelligence. Summary of the invention

[0004] The present invention intends to provide a data processing method based on an AI model to solve the problems that the existing data processing model has a slow data processing process and low efficiency, the existing data processing model is unstable, and the performance and generalization ability of large AI models are often limited.

[0005] In order to achieve the above object, the present invention provides the following method:

[0006] A data processing method based on an AI model provided by the present invention is:

[0007] S1: collecting data information received by the computer and encapsulating the data information into a data information set;

[0008] S2: classifying the data information set and clearing out useless data information;

[0009] S3: compressing and arranging the classified data information set, and performing a first preprocessing step to obtain first processed data information;

[0010] S4: Binding a data source mark to the first processed data, and converting the first processed data into a data feature vector;

[0011] S5: performing a second preprocessing step on the data feature vector to obtain a data feature portrait;

[0012] S6: By comparing the data feature portrait with the first processed data information, the AI ​​model training data of the first preprocessing step and the second preprocessing step are adjusted to obtain an AI data processing model.

[0013] Preferably, before encapsulating the data information into a data information set, it also includes: determining whether the data information contains preset processing parameters; when the data information contains preset processing parameters, obtaining a node encapsulation object from the data information; when the data information does not contain preset processing parameters, performing data processing through an AI data processing model.

[0014] Preferably, the step of collecting the data information received by the computer and encapsulating the data information into a data information set includes: using the target access behavior identification network to perform data mining processing on the data information distribution network, outputting the target distribution network data mining results corresponding to the data information distribution network, and then extracting one or more distribution network data mining results from the target distribution network data mining results and the related distribution network data mining results; performing network area segmentation processing on the target access behavior data distribution network to form multiple access behavior data distribution network areas corresponding to the target access behavior data distribution network; splicing processing on the multiple access behavior data distribution network areas to form data information corresponding to the target access behavior data distribution network; encapsulating the data information into a data information set according to different data distribution network areas.

[0015] Preferably, the step of classifying the data information set and clearing out useless data information includes: classifying the data set according to the network area segmentation processing based on the data information set, and formulating dynamic filtering rules; filtering the classified data set according to the dynamic filtering rules to clear out useless data information.

[0016] Preferably, the data information set after classification is compressed and organized, and a first preprocessing step is performed to obtain a first processed data information step, including: compressing and encapsulating the data information set after classification into multiple data shard sets; distributing the multiple data shard sets to corresponding data processing processes, so that each data processing process performs data processing on the received data shard set to generate corresponding data processing results; after data processing, receiving the data processing results fed back by each data processing process to obtain the first preprocessed data information.

[0017] Preferably, the step of distributing multiple data shard sets to corresponding data processing processes so that each data processing process performs data processing on the received data shard sets and generates corresponding data processing results includes: updating or adding the processing result of each data to be processed in the data shard set as a data element to the data to be processed, thereby updating the results of the data to be processed in the data shard set and obtaining an updated data shard set; updating the data to be processed using the processing result of each data to be processed to obtain the updated target data of each data to be processed, and the updated data shard set includes the multiple target data after the results are updated; and fusing the data processing results corresponding to the multiple data shard sets to obtain the corresponding data processing results in the data shard set.

[0018] Preferably, the step of binding the first processed data to a data source mark and converting the first processed data into a data feature vector includes: binding the first processed data to a data source mark; the first processed data includes a data identifier and behavior data, and the data identifier and the behavior data are uncoupled data; extracting the data identifier from the first processed data to obtain a sample identifier of the feature vector to be constructed; extracting the behavior data from the first processed data according to the sample identifier of the feature vector to be constructed; and feature encoding the behavior data to obtain a feature vector of the target user.

[0019] Preferably, the step of feature encoding the behavior data to obtain the feature vector of the target user includes: determining the data type of the behavior data; the data type includes character type, numeric type and classification type; if the data type of the behavior data is character type, encoding the behavior data through a first encoding method to obtain a first feature vector; the first encoding method is an encoding method for behavior data of character type; if the data type of the behavior data is numeric type, encoding the behavior data through a second encoding method to obtain a second feature vector; the second encoding method is an encoding method for behavior data of numeric type; if the data type of the behavior data is classification type, encoding the behavior data through a third encoding method to obtain a third feature vector; the third encoding method is an encoding method for behavior data of classification type.

[0020] Preferably, the step of subjecting the data feature vector to a second preprocessing step to obtain a data feature portrait includes: setting a data feature vector threshold to screen and filter the data feature vector; using an incremental learning algorithm to locally update the data feature vector based on the results of the screening and filtering to optimize the network parameters in the AI ​​data processing model to obtain a data feature portrait.

[0021] Preferably, the step of obtaining the AI ​​data processing model by comparing the data feature portrait with the first processed data information and adjusting the AI ​​model training data of the first preprocessing step and the second preprocessing step includes: setting a preset delivery indicator based on the data feature portrait, predicting the delivery indicator according to the first processed data information, and adjusting the AI ​​model training data of the first preprocessing step and the second preprocessing step; adjusting a preset adjustment parameter according to the preset delivery indicator and the predicted delivery indicator until the difference between the preset delivery indicator and the predicted delivery indicator is within a preset difference range, thereby obtaining the corrected AI data processing model.

[0022] The beneficial effects of the present invention are embodied in: the present invention collects data information received by the computer to judge preset processing parameters, and then processes the data through the AI ​​data processing model, first performs data mining, then performs network area cutting, performs secondary splicing, and finally encapsulates it into a data information set. It can perform preliminary filtering on the data before the encapsulation process and convert the basic data form, so that the data information can be more convenient to process, and then perform classification filtering. After the first preprocessing step, the data information is organized into a data fragment set for result fusion processing and converted into a feature vector, so that the data information transmission rate is faster, and then the data is encoded through three different encoding methods, and then filtered again to obtain a data feature portrait, and finally corrected to obtain an AI data processing model, so as to prevent the data processing model from making errors due to different data information format encodings during the data processing process, so that the model processes data information more stably and efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0024] Figure 1 It is a flowchart of a data processing method based on an AI model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the technical personnel in the technical field better understand the scheme of the present invention, the technical scheme in the embodiment of the present invention will be clearly and completely described below in combination with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or end including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or ends.

[0027] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0028] At present, with the continuous growth of data processing volume and the improvement of computing power of AI big models, AI big models are being used more and more widely in various fields; however, in actual application, due to the increasing diversity and complexity of data, the performance and generalization ability of AI big models are often limited; therefore, how to optimize the performance and generalization ability of AI big models and improve the stability and reliability of the models is an important issue facing the current field of artificial intelligence.

[0029] The present invention intends to provide a data processing method based on an AI model to solve the problems that the existing data processing model has a slow data processing process and low efficiency, the existing data processing model is unstable, and the performance and generalization ability of large AI models are often limited.

[0030] This method is Figure 1 As shown, a specific embodiment of the present invention provides a data processing method based on an AI model, comprising the following steps:

[0031] S1: Collect the data information received by the computer and encapsulate the data information into a data information set.

[0032] In an embodiment of the present invention, before encapsulating the data information into a data information set, it also includes: determining whether the data information contains preset processing parameters; when the data information contains preset processing parameters, obtaining a node encapsulation object from the data information; when the data information does not contain preset processing parameters, performing data processing through an AI data processing model; utilizing a target access behavior recognition network to perform data mining processing on the data information distribution network, outputting a target distribution network data mining result corresponding to the data information distribution network, and then extracting one or more distribution network data mining results from the target distribution network data mining result and the related distribution network data mining result; performing network area segmentation processing on the target access behavior data distribution network to form a plurality of access behavior data distribution network areas corresponding to the target access behavior data distribution network; performing splicing processing on a plurality of access behavior data distribution network areas to form data information corresponding to the target access behavior data distribution network; and encapsulating the data information into a data information set according to different data distribution network areas.

[0033] S2: Classify the data information set and clean up useless data information.

[0034] In the embodiment of the present invention, data sets are classified according to the network area segmentation processing based on the data information set, and dynamic filtering rules are formulated; the classified data sets are filtered according to the dynamic filtering rules to clear out useless data information.

[0035] S3: The classified data information set is compressed and sorted, and a first preprocessing step is performed to obtain first processed data information.

[0036] In an embodiment of the present invention, the classified data information set is compressed and packaged into multiple data shard sets; the multiple data shard sets are distributed to corresponding data processing processes, so that each data processing process performs data processing on the received data shard sets and generates corresponding data processing results; the processing result of each data to be processed in the data shard set is updated or added to the data to be processed as a data element, so as to achieve the result update of the data to be processed in the data shard set and obtain the updated data shard set; the processing result of each data to be processed is used to update the data to be processed, and the updated target data of each data to be processed is obtained, and the updated data shard set includes multiple target data after the result update; the data processing results corresponding to the multiple data shard sets are fused to obtain the corresponding data processing results in the data shard set; after the data processing, the data processing results fed back by each data processing process are received to obtain the first pre-processed data information.

[0037] S4: Bind the first processed data with a data source tag, and convert the first processed data into a data feature vector.

[0038] In an embodiment of the present invention, the first processed data is obtained to bind the data source tag; the first processed data includes a data identifier and behavior data, and the data identifier and the behavior data are uncoupled data; the data identifier is extracted from the first processed data to obtain a sample identifier of the feature vector to be constructed; according to the sample identifier of the feature vector to be constructed, the behavior data is extracted from the first processed data; the behavior data is feature-encoded to obtain a feature vector of the target user; the data type of the behavior data is determined; the data type includes a character type, a numeric type and a classification type; if the data type of the behavior data is a character type, the behavior data is encoded by a first encoding method to obtain a first feature vector; the first encoding method is an encoding method for character type behavior data; if the data type of the behavior data is a numeric type, the behavior data is encoded by a second encoding method to obtain a second feature vector; the second encoding method is an encoding method for numeric type behavior data; if the data type of the behavior data is a classification type, the behavior data is encoded by a third encoding method to obtain a third feature vector; the third encoding method is an encoding method for classification type behavior data.

[0039] S5: Perform a second preprocessing step on the data feature vector to obtain a data feature portrait.

[0040] In an embodiment of the present invention, a data feature vector threshold is set to screen and filter the data feature vector; an incremental learning algorithm is used to locally update the data feature vector based on the results of screening and filtering to optimize the network parameters in the AI ​​data processing model and obtain a data feature portrait.

[0041] S6: Compare and train the data feature portrait with the first processed data information to adjust the AI ​​model training data of the first preprocessing step and the second preprocessing step to obtain an AI data processing model.

[0042] In an embodiment of the present invention, a preset delivery indicator is set based on the data feature portrait, the delivery indicator is predicted based on the first processed data information, and the AI ​​model training data of the first preprocessing step and the second preprocessing step are adjusted; the preset adjustment parameters are adjusted according to the preset delivery indicator and the predicted delivery indicator until the difference between the preset delivery indicator and the predicted delivery indicator is within the preset difference range, thereby obtaining a corrected AI data processing model.

[0043] The beneficial effects of the present invention are embodied in: the present invention collects data information received by the computer to judge preset processing parameters, and then processes the data through the AI ​​data processing model, first performs data mining, then performs network area cutting, performs secondary splicing, and finally encapsulates it into a data information set. It can perform preliminary filtering on the data before the encapsulation process and convert the basic data form, so that the data information can be more convenient to process, and then perform classification filtering. After the first preprocessing step, the data information is organized into a data fragment set for result fusion processing and converted into a feature vector, so that the data information transmission rate is faster, and then the data is encoded through three different encoding methods, and then filtered again to obtain a data feature portrait, and finally corrected to obtain an AI data processing model, so as to prevent the data processing model from making errors due to different data information format encodings during the data processing process, so that the model processes data information more stably and efficiently.

[0044] The above is only an embodiment of the present invention, and the common knowledge such as the known specific technical solutions or characteristics in the solution is not described in detail here. It should be pointed out that for those skilled in the art, without departing from the solution of the present invention, several modifications and improvements can be made, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A data processing method based on an AI model, characterized in that: The method comprises: S1: Collecting data information received by the computer and encapsulating the data information into a data information set; S2: classifying the data information set and clearing out useless data information; S3: compressing and arranging the classified data information set, and performing a first preprocessing step to obtain first processed data information; S4: Binding a data source mark to the first processed data, and converting the first processed data into a data feature vector; S5: performing a second preprocessing step on the data feature vector to obtain a data feature portrait; S6: By comparing the data feature portrait with the first processed data information, the AI ​​model training data of the first preprocessing step and the second preprocessing step are adjusted to obtain an AI data processing model.

2. The data processing method based on the AI ​​model according to claim 1, characterized in that: Before encapsulating the data information into a data information set, the process further includes: Determining whether the data information contains preset processing parameters; When the data information includes preset processing parameters, obtaining a node encapsulation object from the data information; When the data information does not contain preset processing parameters, data processing is performed through the AI ​​data processing model.

3. The data processing method based on the AI ​​model according to claim 2 is characterized in that: The step of collecting the data information received by the computer and packaging the data information into a data information set includes: Using the target access behavior recognition network, data mining is performed on the data information distribution network, a target distribution network data mining result corresponding to the data information distribution network is output, and then one or more distribution network data mining results of the target distribution network data mining result and the related distribution network data mining result are extracted; Performing network area segmentation processing on the target access behavior data distribution network to form a plurality of access behavior data distribution network areas corresponding to the target access behavior data distribution network; Performing splicing processing on the multiple access behavior data distribution network areas to form data information corresponding to the target access behavior data distribution network; The data information is packaged into data information sets according to different data distribution network areas.

4. The data processing method based on the AI ​​model according to claim 3 is characterized in that: The step of classifying the data information set and clearing out useless data information comprises: Classifying the data set according to the data information set and the network area segmentation processing, and formulating dynamic filtering rules; The data sets are classified and filtered according to the dynamic filtering rules to clear out useless data information.

5. The data processing method based on the AI ​​model according to claim 1, characterized in that: The step of compressing and arranging the classified data information set, performing a first preprocessing step, and obtaining first processed data information includes: Compressing and packaging the classified data information set into multiple data fragment sets; Distribute multiple data shard sets to corresponding data processing processes, so that each data processing process processes the received data shard sets and generates corresponding data processing results; After the data processing is performed, the data processing results fed back from each data processing process are received to obtain the first pre-processed data information.

6. The data processing method based on the AI ​​model according to claim 5, characterized in that: The step of distributing the multiple data shard sets to corresponding data processing processes so that each data processing process performs data processing on the received data shard sets and generates corresponding data processing results includes: The processing result of each piece of data to be processed in the data shard set is updated or added to the piece of data to be processed as a data element, so as to update the result of the data to be processed in the data shard set and obtain an updated data shard set; Using the processing result of each piece of data to be processed, the piece of data to be processed is updated to obtain the updated target data of each piece of data to be processed, wherein the updated data shard set includes the multiple pieces of target data after the result is updated; The data processing results corresponding to the multiple data shard sets are respectively subjected to result fusion processing to obtain the corresponding data processing results in the data shard set.

7. The data processing method based on the AI ​​model according to claim 5, characterized in that: The step of binding the first processed data with a data source tag and converting the first processed data into a data feature vector includes: Binding the obtained first processed data to a data source mark; The first processed data includes a data identifier and behavior data, and the data identifier and the behavior data are uncoupled data; Extracting the data identifier from the first processed data to obtain a sample identifier of the feature vector to be constructed; extracting the behavior data from the first processed data according to the sample identifier of the feature vector to be constructed; Feature encoding is performed on the behavior data to obtain a feature vector of the target user.

8. The data processing method based on the AI ​​model according to claim 7, characterized in that: The step of encoding the behavior data to obtain the feature vector of the target user includes: Determine the data type of the behavior data; the data type includes character type, numeric type and classification type; If the data type of the behavior data is a character type, the behavior data is encoded using a first encoding method to obtain a first feature vector; the first encoding method is an encoding method for character type behavior data; If the data type of the behavior data is a digital type, the behavior data is encoded using a second encoding method to obtain a second feature vector; the second encoding method is an encoding method for digital type behavior data; If the data type of the behavior data is a classification type, the behavior data is encoded using a third encoding method to obtain a third feature vector; the third encoding method is an encoding method for behavior data of a classification type.

9. The data processing method based on the AI ​​model according to claim 7, characterized in that: The step of performing a second preprocessing step on the data feature vector to obtain a data feature portrait comprises: Setting a data feature vector threshold to filter the data feature vector; An incremental learning algorithm is used to locally update the data feature vector based on the results of screening and filtering to optimize the network parameters in the AI ​​data processing model and obtain a data feature portrait.

10. The data processing method based on the AI ​​model according to claim 9, characterized in that: The step of comparing the data feature portrait with the first processed data information to adjust the AI ​​model training data of the first preprocessing step and the second preprocessing step to obtain the AI ​​data processing model includes: Setting a preset delivery index based on the data feature portrait, predicting the delivery index according to the first processed data information, and adjusting the AI ​​model training data of the first preprocessing step and the second preprocessing step; The preset adjustment parameters are adjusted according to the preset delivery index and the predicted delivery index until the difference between the preset delivery index and the predicted delivery index is within a preset difference range, thereby obtaining the corrected AI data processing model.

Images