Data quality improvement method and device based on large model and electronic equipment

Abstract

The invention discloses a data quality improvement method and device based on a large model and electronic equipment, and relates to the field of data processing, and the method comprises a data investigation step, a data preparation step, a data extraction step, a data cleaning step and a data complementation step. The step of data investigation comprises the following steps: acquiring all test data fields recorded in a road test system; all test data fields are marked and arranged according to the number comparison of the signals, the test correlation and the commonly-used performance; wherein the commonly-used property is used as a first priority principle, and all test data fields are marked and arranged; the data preparation step comprises the step of setting a data standardization processing rule by taking the commonly-used property as a first priority principle. According to the method, the large model is adopted to extract the test data, and the structure and the field meaning of each table in the database are learned through the large model, so that the system can automatically generate the query statement according to the user instruction to extract the test data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, in particular to a data quality improvement method based on a large model, a data quality improvement device based on a large model, an electronic device, a storage medium and an operation platform. BACKGROUND

[0002] Under the background of rapid development of digitalization and intelligentization technology, commercial vehicle big data is gradually becoming the core driving force in the fields of logistics transportation, vehicle management and traffic optimization. Data quality, as the core cornerstone, is important throughout the whole chain from data collection, transmission, storage to application. If the data is missing, wrong or delayed, it will directly weaken the decision value and reliability of big data. Therefore, the present application innovatively takes Deepseek-R1 large model as the basic model and fine-tunes the model, and introduces the fine-tuned large model to intelligently read data and preliminarily screen data anomalies. DeepSeek-R1 has strong reasoning ability and can effectively cope with the diversity and complexity of road test system signal collection. By analyzing the table building statement and language description, it accurately generates SQL statements and retrieves related test data, and through the strong reasoning ability, it accurately identifies and removes abnormal values, noise and redundant information in the data. At the same time, this paper adopts LSTM neural network technology, which can efficiently capture long-term dependencies in time series data through its unique gating mechanism and memory cell design, thus showing significant advantages in filling missing data. The road test data is standardized, cleaned and verified to achieve the purpose of improving data quality. SUMMARY

[0003] The purpose of the present application is to provide a data quality improvement method based on a large model, a data quality improvement device based on a large model, an electronic device, a storage medium and an operation platform, which at least solve the problem of how to accurately identify and remove abnormal values, noise and redundant information in the data, and solve one of the technical problems of how to efficiently capture long-term dependencies in time series data.

[0004] The present application provides the following scheme:

[0005] According to the first aspect of the present application, a data quality improvement method based on a large model is provided, which comprises the steps of data investigation, data preparation, data extraction, data cleaning and data completion.

[0006] The data investigation step comprises obtaining all test data fields recorded in the road test system;

[0007] According to the comparison of the number of signals, the correlation of the test and the commonality, all test data fields are marked and arranged;

[0008] wherein the commonality is taken as the first priority principle to mark and arrange all test data fields;

[0009] The data preparation step includes setting data normalization processing rules according to the commonality as the first priority principle.

[0010] Further, the data normalization processing rules include:

[0011] Extracting units and meanings related to road test data in the road test system;

[0012] According to the units and meanings related to road test data, all test data fields are normalized;

[0013] The normalization of all test data fields includes distinguishing continuous signals and non-continuous signals and formulating naming rules;

[0014] According to the meanings related to road test data and the naming rules, table structures are designed and MySQL tables are created for data migration.

[0015] Further, the data extraction step includes:

[0016] Based on the metadata, large model training is performed;

[0017] The metadata includes obtaining metadata according to sample data and table creation statements with annotations;

[0018] wherein the roles of the road test system are obtained;

[0019] According to the roles of the road test system, identifiable instructions are generated and sent to the large model;

[0020] The metadata is fed to the large model for preliminary learning and evaluation and feedback are given;

[0021] The large model is started for testing, test data is extracted, and SQL statements are obtained.

[0022] Further, the data cleaning step includes:

[0023] Based on the fine-tuned large model, the cleaning target is determined, the normal and abnormal data sample sets are collected, and the abnormal data sample set is labeled;

[0024] The sample set includes a training set, a validation set, and a test set;

[0025] wherein the proportions of the training set, the validation set, and the test set include 70% of the training set, 15% of the validation set, and 15% of the test set.

[0026] Further, the data completion step includes importing the cleaned data and performing normalization processing based on TensorFlow using an LSTM neural network technology;

[0027] The LSTM model is constructed, the sequence data is processed using the LSTM layer, and the predicted value is output by the Dense layer;

[0028] The training data is generated and the training set and the validation set are divided;

[0029] The model is trained, and the validation set loss is monitored;

[0030] The missing values are predicted and the data is completed.

[0031] Further, the fine-tuning of the large model includes:

[0032] The large model is fine-tuned through the LoRA technology;

[0033] The LoRA updates the parameters by injecting a low-rank matrix into the original weight matrix;

[0034] Based on the LoRA, the parameters are updated by injecting a low-rank matrix into the original weight matrix, the Transformers library is integrated with the LoRA, and the PEFT library is used to provide the LoRA implementation, and the local model and the tokenizer are loaded, the dataset is loaded, the fine-tuning parameters are configured, the training parameters are set, the data processor is created, the training is started, the LoRA weight is merged, the fine-tuned large model is generated and saved;

[0035] According to the generated fine-tuned large model, the extracted data is segmented and evaluated, and the abnormal data is deleted.

[0036] According to the second aspect of the present application, a large model-based data quality improvement device is provided, which includes a data investigation module, a data preparation module, a data extraction module, a data cleaning module and a data completion module;

[0037] The data investigation module includes obtaining all test data fields recorded in the road test system;

[0038] According to the number of signals, the test correlation and the commonality, all test data fields are marked and arranged;

[0039] Among them, the commonality is taken as the first priority principle, and all test data fields are marked and arranged;

[0040] The data preparation module includes setting data normalization processing rules according to the first priority principle of commonality.

[0041] According to a third aspect of the present application, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus.

[0042] The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the data quality improvement method based on a large model.

[0043] According to a fourth aspect of the present application, a computer readable storage medium is provided, which stores a computer program executable by an electronic device, and when the computer program is run on the electronic device, the electronic device executes the steps of the data quality improvement method based on a large model.

[0044] According to a fifth aspect of the present application, an operation platform is provided, comprising:

[0045] The electronic device is used to implement the steps of the data quality improvement method based on a large model;

[0046] The processor runs a program, and when the program runs, the data output from the electronic device executes the steps of the data quality improvement method based on a large model;

[0047] The storage medium is used to store the program, and when the program runs, the data output from the electronic device executes the steps of the data quality improvement method based on a large model.

[0048] Through the above scheme, the following beneficial technical effects are obtained:

[0049] The present application uses a large model to extract test data, learns the structure and field meaning of each table in the database through a large model, so that the system can automatically generate a query statement according to user instructions to extract test data.

[0050] The present application uses a fine-tuned large model to remove abnormal data, generates three data sets in a semi-automatic manner and adds an abnormal identifier, uses LoRA technology to fine-tune the DeepSeek-R1 large model to realize test data abnormality identification and removal, and rapidly improves the identification efficiency.

[0051] The present application uses LSTM technology to complete missing data, uses a sliding window mechanism to use historical observation values as input to predict and fill in test data at missing time points. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 is a flowchart of a data quality improvement method based on a large model provided by one or more embodiments of the present application.

[0053] Figure 2is a structural diagram of a large model-based data quality improvement device provided by one or more embodiments of the present application.

[0054] Figure 3 is a schematic diagram of a large model-based data quality improvement method provided by one specific embodiment of the present application.

[0055] Figure 4 is a schematic diagram of ECU vehicle speed of test vehicle raw data provided by one specific embodiment of the present application.

[0056] Figure 5 is a schematic diagram of ECU vehicle speed of test vehicle data quality improvement provided by one specific embodiment of the present application.

[0057] Figure 6 is a structural block diagram of a large model-based data quality improvement method provided by one or more embodiments of the present application. DETAILED DESCRIPTION

[0058] The technical solutions of the present application will be described clearly and completely below in conjunction with the drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0059] Figure 1 is a flowchart of a large model-based data quality improvement method provided by one or more embodiments of the present application.

[0060] As shown in Figure 1 the large model-based data quality improvement method includes steps A1 of data investigation, A2 of data preparation, A3 of data extraction, A4 of data cleaning, and A5 of data completion;

[0061] Step A1 of data investigation includes obtaining all test data fields recorded in the road test system;

[0062] According to the comparison of the number of signals, the test correlation and the commonality, all test data fields are marked and sorted;

[0063] Among them, the commonality is taken as the first priority principle to mark and sort all test data fields;

[0064] Step A2 of data preparation includes setting data normalization processing rules according to the first priority principle of commonality.

[0065] Specifically, the present application is applicable to a large model-based data quality improvement strategy; in one specific embodiment, asFigure 3 The data subject of the road test system-based data quality improvement method shown in the embodiment is road test data, i.e., Roads system data. Since the Roads system has problems such as repeated test data fields, non-uniform, non-standard naming, etc., first, the Roads system test data in the Jiefang test data is counted, and then according to the statistical results and the characteristics of the data, the appropriate data set is selected, and the data quality improvement method of the road test system is formulated. In a specific embodiment, through investigation and statistical methods, it is found that there are 32000+ test data fields in the road test system, and the repetition is serious. Finally, 971 data with larger amount and more important meaning and 196 commonly used signals are sorted out, and it is planned to first carry out data management for the 196 commonly used signals.

[0066] In a specific embodiment, after data preparation, i.e., data exploration of the selected data set to clarify its meaning, first, data normalization processing is performed to make it have a unified format, range and unit of measurement; then according to the characteristics of the data, select the appropriate database, design and create the table structure; finally, the target data set is migrated to the designated table. Data preparation work can provide a data basis for data extraction to understand the meaning of the data and data cleaning to reduce analysis time.

[0067] In a specific embodiment, since the original names of a large number of signals in the Roads system are in English, in order to meet the principle of knowing the meaning by the name and facilitate subsequent data processing work, the units and meanings of the road test data are extracted from the related files in batches by writing programs to complete data meaning exploration. Then, data normalization processing is performed on the 196 commonly used signals, the continuous signals and the non-continuous signals are distinguished, and naming rules are formulated. Finally, the table structure is designed according to the data meaning and naming rules, the MySQL table is created, and the data migration is realized to complete the data preparation.

[0068] In this embodiment, the data normalization processing rules include:

[0069] extracting the units and meanings related to the road test data in the road test system;

[0070] normalizing all test data fields according to the units and meanings related to the road test data;

[0071] The normalization of all test data fields includes distinguishing the continuous signals and the non-continuous signals and formulating naming rules;

[0072] According to the meanings and naming rules related to the road test data, the table structure is designed, the MySQL table is created, and the data migration is realized.

[0073] In this embodiment, the steps of data extraction include:

[0074] Based on the metadata, large model training is performed;

[0075] The metadata includes obtaining metadata according to sample data and table creation statements with annotations;

[0076] Among them, the role of the road test system is obtained;

[0077] According to the role of the road test system, a recognizable instruction is generated and sent to the large model;

[0078] The metadata is fed to the large model for preliminary learning and evaluation and feedback;

[0079] The large model is started for testing, and the test data is extracted to obtain the SQL statement.

[0080] Specifically, in one specific embodiment, data extraction, i.e. extracting relevant data of the field to be processed from MySQL, writing a query statement, accurately filtering and extracting the required data. In order to realize the convenience and intelligence of data extraction, the present application introduces a large model, which generates a corresponding query statement and completes data extraction in the form of natural language dialogue.

[0081] In this embodiment, the key to data extraction is to let the large model understand the meaning of each table and each field in the database. Only if the large model understands the specific situation of data storage in advance can it use natural language to achieve intelligent extraction. Therefore, this embodiment inputs sample data and table creation statements with detailed annotations into the large model for learning, such as using the Prompt (prompt word) of DeepSeek-R1 large model. For example, first, a clear instruction is issued by the system role, and the specific content is "you are a professional MySQL generation assistant, and you need to accurately understand the customer's instruction and generate a standard SQL statement"; then, feed the metadata to the large model for preliminary learning and give the corresponding evaluation and feedback; then, start the model and test it; finally, the output SQL statement is used to realize intelligent extraction of test data through Python third-party library.

[0082] In this embodiment, the steps of data cleaning include:

[0083] Based on the fine-tuned large model, the cleaning target is determined, the sample set of normal and abnormal data is collected, and the sample set of abnormal data is labeled;

[0084] The sample set includes a training set, a validation set and a test set;

[0085] Among them, the proportion of the training set, the validation set and the test set includes 70% of the training set, 15% of the validation set and 15% of the test set.

[0086] In this embodiment, fine-tuning the large model comprises:

[0087] Fine-tuning the large model through LoRA technology;

[0088] Wherein, LoRA realizes parameter update by injecting a low-rank matrix into the original weight matrix;

[0089] Based on LoRA, the low-rank matrix is injected into the original weight matrix to realize parameter update. The Transformers library is integrated with LoRA, and the PEFT library is used to provide LoRA implementation. In sequence, the local model and tokenizer are loaded, the dataset is loaded, the fine-tuning parameters are configured, the training parameters are set, the data refiner is created, the training is started, the LoRA weight is merged, the fine-tuned large model is generated and saved;

[0090] According to the generated fine-tuned large model, the extracted data segments are evaluated and abnormal data are deleted.

[0091] Specifically, in one specific embodiment, data cleaning, i.e. removing abnormal data, lays a data foundation for the next step of data completion. The data cleaning is performed in the manner of using a large model to intelligently identify and remove abnormal data. Considering the professional and privacy nature of the test data and the limitations of general large models, this embodiment uses the locally deployed fine-tuned Deepseek-R1 large model to realize the function. Due to the limited performance of personal computers, this embodiment uses LoRA technology to fine-tune the large model. As a lightweight fine-tuning technology, LoRA can significantly reduce the demand for computing resources, achieving the effect of reducing cost and increasing efficiency. Then, the model is evaluated, and the hyperparameters are optimized according to the accuracy, running time and other dimensions. Finally, the fine-tuned large model is deployed and data cleaning is completed.

[0092] In one specific embodiment, the data cleaning is to be completed by the fine-tuned DeepSeek-R1 large model. First, the cleaning target is determined, i.e., cleaning duplicate data (multiple data with the same timestamp), jump data (data that changes greatly between adjacent time points or adjacent samples), and data that is out of the physical range (data that does not conform to physical laws or business rules). Then, the sample set of normal and abnormal data is collected and the abnormal data is labeled with three labels. The sample set is divided into training set, validation set and test set. After repeated testing, the data proportion of the three data sets is set as: training set (70%), validation set (15%), and test set (15%). In order to balance the training accuracy and efficiency, the data set is generated in a semi-automatic manner, i.e., combining manual and tool, to quickly generate a large amount of data while ensuring a certain quality. Since the task complexity of data cleaning is low, in one specific embodiment, the total amount of generated data is 5000. Then, the DeepSeek-R1 large model is fine-tuned through LoRA technology. The core principle of LoRA technology is to update the parameters by injecting a low-rank matrix (AxB) into the original weight matrix. In this embodiment, Transformers library is integrated with LoRA and PEFT library is used to provide LoRA implementation. The fine-tuned model is finally generated and saved by sequentially loading the local model and tokenizer, loading the data set, configuring the fine-tuning parameters, setting the training parameters, creating the data organizer, starting the training, and merging the LoRA weight. The model can segment the extracted data for evaluation and delete abnormal data to lay the foundation for the next data completion.

[0093] In this embodiment, the steps of data completion include importing the cleaned data and performing normalization processing based on TensorFlow using LSTM neural network technology.

[0094] The LSTM model is constructed, the sequence data is processed using the LSTM layer, and the predicted value is output by the Dense layer.

[0095] Training data is generated and divided into training set and validation set.

[0096] The model is trained and the validation set loss is monitored.

[0097] The missing values are predicted and the data is completed.

[0098] Specifically, in one specific embodiment, data completion adopts LSTM neural network technology. LSTM can capture long-term dependencies in time series well by virtue of its unique internal memory unit and gating mechanism, and use the time correlation in the data to complete the abnormal data and missing data deleted in data cleaning. Through experiments, appropriate node number and learning rate, batch size, training round number and other hyperparameters are selected, and then model training is realized through forward propagation, loss calculation, back propagation and model verification, and finally data completion missing values are generated. Taking the ECU speed signal as an example, the original data and the data quality improved data are compared as shown in Figure 4 、 5

[0099] In one specific embodiment, after data cleaning, the abnormal data such as repetition, jump and super-physical range in the test data have been effectively improved, and at the same time, there are more data missing. This embodiment uses LSTM neural network technology with the help of TensorFlow, first imports the cleaned data and performs normalization processing; then constructs the LSTM model, i.e. uses the LSTM layer to process sequence data and the Dense layer to output predicted values; generates training data and divides the training set and the validation set; then trains the model and monitors the validation set loss; finally, predicts the missing values and completes the data. Among them, multiple index evaluations are performed, i.e. the mean square error (MSE) or mean absolute error (MAE) of the completed data and the true data is calculated, and the LSTM layer learning rate and other hyperparameters are optimized to optimize the performance. Finally, the original test data abnormal values are reasonably adjusted to improve the data quality.

[0100] Figure 2 is a structural diagram of a data quality improvement device based on a large model provided by one or more embodiments of the present application.

[0101] As shown in Figure 2 , the data quality improvement device based on a large model includes a data research module, a data preparation module, a data extraction module, a data cleaning module and a data completion module;

[0102] The data research module includes obtaining all test data fields recorded in the road test system;

[0103] According to the number comparison of signals, test correlation and commonality, all test data fields are marked and arranged;

[0104] Among them, the commonality is taken as the first priority principle to mark and arrange all test data fields;

[0105] The data preparation module includes setting data normalization processing rules according to taking commonality as the first priority principle. ​

[0106] It is worth noting that although the system / device only discloses the data research module, the data preparation module, the data extraction module, the data cleaning module and the data completion module, it does not mean that the device is limited to the above basic function modules. On the contrary, the meaning expressed by the present application is that on the basis of the above basic function modules, the skilled in the art can add one or more function modules to form infinite embodiments or technical solutions in combination with the prior art. That is to say, the system / device is open rather than closed, and the protection scope of the present application cannot be limited to the above disclosed basic function modules because the present embodiment only discloses individual basic function modules.

[0107] Figure 6 is an electronic device structure block diagram of the big model-based data quality improvement method provided by one or more embodiments of the present application.

[0108] As shown in Figure 6 The present application provides an electronic device, which comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus.

[0109] The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the big model-based data quality improvement method.

[0110] The present application also provides a computer readable storage medium, which stores a computer program executable by an electronic device, and when the computer program runs on the electronic device, the electronic device executes the steps of the big model-based data quality improvement method.

[0111] The present application also provides an operation platform, which comprises:

[0112] The electronic device is used to implement the steps of the big model-based data quality improvement method.

[0113] The processor runs the program, and when the program runs, the data output from the electronic device executes the steps of the big model-based data quality improvement method.

[0114] The storage medium is used to store the program, and when the program runs, the data output from the electronic device executes the steps of the big model-based data quality improvement method.

[0115] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0116] The electronic device includes a hardware layer, an operating system layer running above the hardware layer, and an application layer running on the operating system. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory. The operating system can be any one or more computer operating systems that implement control of the electronic device through a process, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a windows operating system. In embodiments of the present application, the electronic device can be a handheld device such as a smartphone or a tablet computer, or an electronic device such as a desktop computer or a portable computer, and is not particularly limited in embodiments of the present application.

[0117] The execution subject of the electronic device control in embodiments of the present application can be the electronic device, or a functional module capable of calling and executing a program in the electronic device. The electronic device can obtain a firmware corresponding to the storage medium, and the firmware corresponding to the storage medium is provided by a supplier. The firmware corresponding to different storage media can be the same or different, and is not limited herein. After the electronic device obtains the firmware corresponding to the storage medium, the electronic device can write the firmware corresponding to the storage medium into the storage medium, specifically, burn the firmware corresponding to the storage medium into the storage medium. The process of burning the firmware into the storage medium can be implemented by using existing technology, and is not described in detail in embodiments of the present application.

[0118] The electronic device can also obtain a reset command corresponding to the storage medium, and the reset command corresponding to the storage medium is provided by a supplier. The reset command corresponding to different storage media can be the same or different, and is not limited herein.

[0119] At this time, the storage medium of the electronic device is a storage medium in which the corresponding firmware is written, and the electronic device can respond to the reset command corresponding to the storage medium in the storage medium in which the corresponding firmware is written, so that the electronic device resets the storage medium in which the corresponding firmware is written according to the reset command corresponding to the storage medium. The process of resetting the storage medium according to the reset command can be implemented by using existing technology, and is not described in detail in embodiments of the present application.

[0120] For the convenience of description, the above apparatus is described in various units, modules, and the like in terms of functions. Of course, the functions of the units and modules can be implemented in one or more software and / or hardware in implementing the present application.

[0121] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It should also be understood that terms such as those defined in a general dictionary should be interpreted as having a meaning consistent with the meaning in the context of the relevant art and should not be interpreted in an idealized or overly formal sense unless specifically defined.

[0122] For the convenience of description, the above apparatus is described in various units, modules, and the like in terms of functions. Of course, the functions of the units and modules can be implemented in one or more software and / or hardware in implementing the present application.

[0123] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and the necessary universal hardware platforms. Based on such an understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments of the present application.

[0124] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data quality improvement method based on a large model, characterized in that, The data quality improvement method based on large models includes the following steps: data research, data preparation, data extraction, data cleaning, and data completion. The steps of data research include obtaining all test data fields recorded in the road test system; Based on the comparison of signal quantity, experimental correlation, and common usage, all experimental data fields were marked and organized. Among them, the most frequently used principle is to mark and organize all experimental data fields; The steps for data preparation include setting data normalization rules based on the principle of prioritizing commonality.

2. The data quality improvement method based on a large model according to claim 1, characterized in that, The data normalization rules include: Extract the units and meanings related to road test data from the road test system; Based on the units and meanings related to road test data, all test data fields were standardized. The standardization process for all experimental data fields includes distinguishing between continuous and discontinuous signals and establishing naming rules; Based on the meaning and naming rules related to road test data, design the table structure and create MySQL tables for data migration.

3. The data quality improvement method based on a large model according to claim 2, characterized in that, The data extraction steps include: Large-scale model training is performed based on metadata. Metadata includes metadata obtained based on sample data and annotated table creation statements; Among them, the role of the road test system is acquired; Based on the role of the road test system, generate identifiable instructions and send them to the large model; Feed metadata to the large model to perform initial learning and provide evaluation and feedback; Start the large model for testing, extract the experimental data, and obtain the SQL statements.

4. The data quality improvement method based on a large model according to claim 2, characterized in that, The data cleaning steps include: Based on fine-tuning of the large model, the cleaning target is defined, sample sets of normal and abnormal data are collected, and the sample set of abnormal data is labeled. The sample set includes a training set, a validation set, and a test set; The ratio of training set, validation set, and test set is 70%, 15%, and 15%, respectively.

5. The data quality improvement method based on a large model according to claim 3, characterized in that, The data completion steps include importing the cleaned data using LSTM neural network technology based on TensorFlow and performing normalization processing. Construct an LSTM model, use LSTM layers to process sequence data, and use Dense layers to output predicted values; Generate training data and divide it into training and validation sets; Train the model and monitor the loss on the validation set; Predict missing values ​​and complete the data.

6. The data quality improvement method based on a large model according to claim 3 or 4, characterized in that, The fine-tuned large model includes: Fine-tuning of large models using LoRA technology; LoRA achieves parameter updates by injecting a low-rank matrix into the original weight matrix; Based on LoRA, parameter updates are achieved by injecting a low-rank matrix into the original weight matrix. The Transformers library is used to integrate LoRA, and the PEFT library is used to provide LoRA implementation. The process involves loading the local model and Tokenizer, loading the dataset, configuring fine-tuning parameters, setting training parameters, creating a data polisher, starting training, merging LoRA weights, generating a fine-tuned large model, and saving it. Based on the large model that has been fine-tuned, the extracted data is segmented for evaluation and outlier data is removed.

7. A data quality improvement device based on a large model, characterized in that, The data quality improvement device based on the large model includes: a data survey module, a data preparation module, a data extraction module, a data cleaning module, and a data completion module. The data survey module includes obtaining all test data fields recorded in the road test system; Based on the comparison of signal quantity, experimental correlation, and common usage, all experimental data fields were marked and organized. Among them, the most frequently used principle is to mark and organize all experimental data fields; The data preparation module includes setting data normalization rules based on the principle of prioritizing common use.

8. An electronic device, characterized in that, include: The processor, communication interface, memory, and communication bus are connected, with the processor, communication interface, and memory communicating with each other via the communication bus. The memory stores a computer program that, when executed by a processor, causes the processor to perform the steps of the data quality improvement method based on a large model as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The device stores a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of the data quality improvement method based on a large model as described in any one of claims 1 to 7.

10. A computing platform, characterized in that, include: An electronic device for implementing the steps of the data quality improvement method based on a large model as described in any one of claims 1 to 7; A processor that runs a program that, when the program is running, performs the steps of the data quality improvement method based on a large model as described in any one of claims 1 to 7 on data output from an electronic device. A storage medium for storing a program that, when running, performs the steps of the large-model-based data quality improvement method as described in any one of claims 1 to 7 on data output from an electronic device.

Images