Data construction method, data construction device, electronic equipment and storage medium

Abstract

The invention provides a data construction method, a data construction device, electronic equipment and a storage medium. The data construction method comprises the following steps: acquiring a plurality of first test data from a production environment; fitting a Gaussian mixture model based on the plurality of first test data to obtain a fitted Gaussian mixture model; and generating a keyword vector conforming to data distribution of the production environment through the fitted Gaussian mixture model, and generating second test data based on the keyword vector.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a data construction method, a data construction device, an electronic device, and a storage medium. Background Technology

[0002] In the field of software development and testing, data construction is a critical task, playing a vital role in ensuring system stability and functional correctness. As system complexity increases, the demand for diverse, real-world-relevant data during testing is also growing, making traditional manual or simple copying methods insufficient for efficient testing. Summary of the Invention

[0003] This application provides a data construction method, a data construction apparatus, an electronic device, and a storage medium.

[0004] The data construction method provided in this application includes: Multiple sets of initial test data were collected from the production environment; Based on the multiple first test data, the Gaussian mixture model is fitted to obtain the fitted Gaussian mixture model. The fitted Gaussian mixture model is used to generate keyword vectors that conform to the data distribution of the production environment, and second test data is generated based on the keyword vectors.

[0005] The data construction apparatus provided in this application includes: Data processing module: used to collect multiple first test data from the production environment; Distribution fitting module: used to fit the Gaussian mixture model based on the multiple first test data to obtain the fitted Gaussian mixture model; Data generation module: used to generate keyword vectors that conform to the data distribution of the production environment through the fitted Gaussian mixture model, and generate second test data based on the keyword vectors.

[0006] The electronic device provided in this application includes a processor and a memory. The memory is used to store computer programs, and the processor is used to call and run the computer programs stored in the memory to execute the data construction method provided in any embodiment of this application.

[0007] The storage medium provided in the embodiments of this application is used to store a computer program, which causes a computer to execute the data construction method provided in any embodiment of this application.

[0008] The data construction method, data construction apparatus, electronic device, and storage medium provided in this application first collect multiple test data points from the production environment as real data samples; secondly, a Gaussian mixture model is fitted based on these samples to establish a data distribution model consistent with the production environment; finally, keyword vectors are generated through this model, and second test data is further generated. In this way, by using a Gaussian mixture model to simulate the complex data distribution in the production environment, the generated test data is statistically closer to the actual scenario, improving the representativeness of the test data and further enhancing the reliability of the test. Attached Figure Description

[0009] Figure 1 A schematic diagram illustrating the implementation flow of the data construction method provided in the embodiments of this application; Figure 2 A schematic diagram of the data construction apparatus provided in the embodiments of this application; Figure 3 A schematic structural diagram of an electronic device provided in the embodiments of this application; Figure 4 This is a schematic structural diagram of the chip provided in an embodiment of this application. Detailed Implementation

[0010] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0011] It should be noted that, in the embodiments of this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, in the embodiments of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0012] In the description of the embodiments of this application, the term "correspondence" may indicate that there is a direct or indirect correspondence between two things, or that there is an association between two things, or that there is a relationship of instruction and being instructed, configuration and being configured, etc.

[0013] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and they all fall within the protection scope of the embodiments of this application.

[0014] In testing environments, testers face the challenge of manually building diverse test data. Typically, they copy relevant data from the production environment, import it into a database, and push it to the front-end for testing. This approach is not only time-consuming and labor-intensive, but it also usually only creates a small amount of simple test data, making it difficult to ensure data diversity and consistency in data distribution between the testing and production environments.

[0015] However, many defects are closely related to test data, and relying solely on this simple data construction method cannot detect these defects in advance during the testing phase, thus posing a potential risk to the stable online operation of the product. Therefore, how to generate data in the testing environment that is similar to and diverse in the production environment is a key challenge in improving test quality.

[0016] refer to Figure 1 , Figure 1 This is a schematic diagram illustrating the implementation flow of the data construction method provided in the embodiments of this application, such as... Figure 1 As shown, the data construction method provided in this embodiment includes the following steps: Step 101: Collect multiple first test data from the production environment.

[0017] In this embodiment, the production environment refers to the environment in which the actual business system is running, containing various types of first test data generated by real users. This first test data is highly complex and diverse. The first test data is the raw data sample extracted from the production environment and used for subsequent modeling and analysis.

[0018] In this embodiment, the first test data can be retrieved from a database, log file, or other storage medium. For example, in a public opinion monitoring system, the production environment may include a large amount of first test data in text form, such as news reports and social media comments. A Python script can be used to periodically read the first test data from the database and perform preliminary cleaning and filtering on the first test data.

[0019] In this embodiment of the application, multiple first test data can cover all types of data in the production environment.

[0020] In this embodiment, the data cleaning process can use regular expressions to match content with a specific format, while simultaneously setting a blacklist mechanism to filter inappropriate language. By using regular expressions to match content with a specific format and setting a blacklist mechanism to filter inappropriate language during the data cleaning process, the generated first test data can be ensured to be more standardized and accurate, and high-quality input can be provided for subsequent distribution fitting.

[0021] In this embodiment of the application, after collecting multiple first test data from the production environment, the method further includes: The plurality of first test data are preprocessed; the preprocessing operations include one or more of the following operations: removing irrelevant data, formatting, and length filtering.

[0022] In this embodiment, preprocessing refers to the process of initially cleaning and standardizing multiple sets of first test data before they are formally used for training or model generation. The purpose of preprocessing is to improve data quality, remove interfering factors, and make the first test data more suitable for subsequent modeling needs. For example, in a public opinion monitoring system, multiple sets of first test data may contain a large amount of irrelevant content, such as HTML tags, image links, and special symbols. This content not only occupies storage space but also affects the model training effect. Therefore, preprocessing can effectively reduce noise and improve the usability of the first test data.

[0023] In this embodiment, removing irrelevant data refers to deleting data items from the plurality of first test data that are irrelevant to the current task or do not contribute to modeling. For example, in a public opinion analysis scenario, if the test objective is to extract the sentiment tendency of user comments, then information irrelevant to sentiment (such as advertising information and website links) should be removed. Removing irrelevant data helps reduce redundant data and improves the model's generalization ability and training efficiency.

[0024] In this embodiment, formatting refers to converting first test data from different sources or with different structures into a unified format to facilitate subsequent processing. For example, some text data may contain line breaks, extra spaces, or special characters, which can affect the consistency of data parsing. Formatting ensures that all input first test data follows the same structural standard, thereby avoiding processing errors or model misjudgments caused by format differences.

[0025] In this embodiment, length filtering refers to setting a reasonable text length range based on business needs and filtering out texts that are too long or too short. Overly long texts may contain excessive noise, while overly short texts may lack sufficient semantic information; both are detrimental to model learning. By setting a reasonable length threshold, the length filtering method can retain high-quality, meaningful first test data samples, thereby improving the quality and representativeness of the first test dataset.

[0026] In this embodiment, by adding preprocessing operations to multiple first test data collected from the production environment, irrelevant data can be removed, the format can be standardized, and a first test data sample of appropriate length can be selected. After adding preprocessing operations to multiple first test data collected from the production environment, this operation can improve the accuracy and consistency of the first test data, thereby better supporting the subsequent distribution fitting and data generation process, and further generating test data that is closer to the real environment, thereby improving the efficiency and quality of the testing work.

[0027] Step 102: Based on the multiple first test data, fit the Gaussian mixture model to obtain the fitted Gaussian mixture model.

[0028] In this embodiment, the Gaussian Mixture Model (GMM) is a probabilistic model that assumes the data is composed of a superposition of several Gaussian distributions. Each Gaussian distribution is called a component and has its own mean, covariance, and weights. The GMM can be used to fit the distribution characteristics of the actual data using the Expectation-Maximization (EM) algorithm, which can be used to generate new data samples that conform to the actual data distribution.

[0029] In this embodiment of the application, the step of fitting the Gaussian mixture model based on the plurality of first test data to obtain the fitted Gaussian mixture model includes: Based on the attribute information of the multiple first test data, the multiple first test data are encoded to obtain multiple first keyword vectors; Based on the multiple first keyword vectors, the Gaussian mixture model is fitted to obtain the fitted Gaussian mixture model.

[0030] In this embodiment, attribute information refers to the key feature information contained in each piece of test data, such as type, sentiment, industry, and region. For example, the attributes and attribute values ​​are as follows: { "type": {"Type": ["Type 1", "Type 2", ... ]}, "attribute": {"Sentiment": ["Positive", "Neutral", "Negative"]}, "industry": {"Industry": ["Finance", "Real Estate", ...]}, "region": {"Region": ["City 1", "City 2", ...]}. The attribute values ​​of the data can be encoded into a vector of a specified dimension. Templates for other types of data can be constructed by adding or deleting key-value values. By encoding these attribute values, unstructured or semi-structured data can be converted into structured numerical vectors, facilitating subsequent statistical modeling and machine learning processing.

[0031] In this embodiment, encoding is a process of converting discrete or textual data into numerical form. For example, for type 1 and type 2 in the type attribute, one-hot encoding or word embedding can be used to map type 1 and type 2 to a vector representation in a high-dimensional space; similarly, for city 1 and city 2 in the region attribute, one-hot encoding or word embedding can also be used to map the region to a vector representation in a high-dimensional space. By using one-hot encoding or word embedding, the original data can be abstracted into vectors of a unified format, thus facilitating the processing and analysis of Gaussian mixture models.

[0032] In this embodiment, the first keyword vector refers to the encoded vector representation, where each dimension corresponds to a specific attribute or attribute value. The first keyword vector not only preserves the semantic information of the original data but also possesses mathematical computability, enabling subsequent distribution fitting and data generation processes to be performed more efficiently and accurately.

[0033] In this embodiment, fitting refers to the process of making the model parameters approximate the true data distribution through training. In this embodiment, the generated first keyword vector is used as input, and the mean, covariance, and weights of the model are updated through multiple iterations, thereby finally obtaining a Gaussian mixture model that can better reflect the characteristics of the data distribution. The fitting process is as follows: Model initialization: Denote each test data point as an N-dimensional vector. Choose the number K of Gaussian distributions, and randomly initialize the parameters of each Gaussian distribution: mean vector. covariance matrix and weight , where the weights sum to 1.

[0034] Expectation-maximization algorithm: E-step (Expectation step), calculating for each data point The posterior probability belonging to each Gaussian distribution k is called the degree of responsibility. The M-step (Maximization step) updates the parameters of each Gaussian distribution: weights. mean covariance .

[0035] Iteration: Repeat the E-step and M-step until convergence (e.g., the parameter change is less than a certain threshold or the maximum number of iterations is reached).

[0036] The pseudocode for the fitting process is shown in Table 1:

[0037] Table 1 The fitted Gaussian mixture model is a model with its parameters already tuned and capable of effectively characterizing the data distribution. This fitted Gaussian mixture model can be used to generate subsequent test data, ensuring that the generated data maintains statistical consistency with the original production environment data.

[0038] Based on this, in this embodiment of the application, fitting the Gaussian mixture model based on the plurality of first keyword vectors to obtain the fitted Gaussian mixture model includes: Based on the multiple first keyword vectors, the Gaussian mixture model is fitted using the expectation-maximization algorithm to obtain the fitted Gaussian mixture model.

[0039] By fitting the Gaussian mixture model based on the first keyword vector, the distribution characteristics of multiple first test data can be captured more accurately. This fitting operation improves the model's generalization ability and ensures that the generated test data has greater diversity and realism, thereby enhancing test coverage and the reliability of test results.

[0040] For a well-fitted Gaussian mixture model, the log-likelihood function can be used to evaluate the model's goodness of fit: When the fit reaches the standard of the test data, the model can be used to generate keyword vectors in batches.

[0041] In this embodiment, by encoding multiple first test data based on attribute information and fitting a Gaussian mixture model using a first keyword vector, multiple first test data that more closely resemble the real data distribution can be generated. The process of encoding multiple first test data based on attribute information and fitting a Gaussian mixture model using a first keyword vector can improve the diversity and accuracy of the multiple first test data, thereby enhancing the effectiveness of the test and significantly improving the test quality and stability of the software system.

[0042] Step 103: Generate a keyword vector that conforms to the data distribution of the production environment using the fitted Gaussian mixture model, and generate second test data based on the keyword vector.

[0043] In this embodiment, the keyword vectors are generated by the fitted Gaussian mixture model. There are multiple keyword vectors, and their distribution conforms to the data distribution in the production environment. The keyword vectors can be used as input to a large language model to generate test data that conforms to the target semantics and structure.

[0044] The test data construction method based on Gaussian mixture model provided in this application first collects first test data from the production environment, and cleans and encodes the first test data to extract key attribute information and convert it into keyword vectors. Then, a distribution fit is performed on the keyword vectors using a Gaussian mixture model, and the model parameters of the Gaussian mixture model are solved using the expectation-maximization algorithm. Finally, new keyword vectors are generated based on the fitted Gaussian mixture model, and these new keyword vectors are input into a large language model to generate second test data that conforms to the target distribution.

[0045] In this embodiment of the application, generating a keyword vector that conforms to the data distribution of the production environment using the fitted Gaussian mixture model includes: The keyword vector is obtained by sampling data based on the weight of each Gaussian component in the Gaussian mixture model.

[0046] In this embodiment, the weight represents the proportion of a specific Gaussian distribution component in the entire Gaussian mixture model. This proportion is used to determine the degree of influence of a specific Gaussian distribution component on the overall data distribution. In this embodiment, the Gaussian mixture model is used to simulate the complex distribution characteristics of test data in a production environment, thereby more accurately reflecting the diversity of real data.

[0047] In this embodiment, data sampling refers to the process of randomly selecting sample points from the corresponding Gaussian distribution based on the structure of the Gaussian mixture model and the weight ratio of each Gaussian component, according to the pre-trained Gaussian mixture model. Specifically, a class distribution is first calculated based on the weights of each Gaussian component. The process involves randomly selecting a Gaussian component index from the category distribution; then, independent sampling is performed from the Gaussian distribution corresponding to the selected Gaussian component index to generate a sample point. This data sampling process can be repeated multiple times to generate a large number of data samples that conform to the target distribution. The keyword vectors generated using this data sampling method are statistically consistent with real-world production data, thereby improving the representativeness of the test data.

[0048] Keyword vectors encode the key attributes (such as type, sentiment, industry, region, etc.) of each test data point into a fixed-dimensional numerical vector. For example, a piece of public opinion data might be encoded as [1, 0, 0, 1], indicating that the public opinion data belongs to the financial industry and has a positive sentiment. The keyword vector format facilitates mathematical modeling and machine learning processing.

[0049] In this embodiment, the weights in the Gaussian mixture model determine the contribution of each Gaussian component to the final data distribution. In practical applications, data patterns represented by Gaussian components with higher weights are more common or important. Therefore, when sampling data, the system prioritizes Gaussian components with higher weights to ensure that the generated data is closer to reality. This weight-based selection mechanism helps avoid generating overly rare or uncommon data patterns and improves the stability and predictability of test results.

[0050] In this embodiment, keyword vectors are generated by sampling data based on the weights of each Gaussian component in a Gaussian mixture model. This method ensures that the generated test data maintains a high degree of consistency with production environment data in terms of statistical distribution, thereby improving the diversity and representativeness of the test data, ultimately enhancing test quality, reducing the workload of manually constructing data, and increasing test efficiency.

[0051] In this embodiment of the application, generating the second test data based on the keyword vector includes: Based on the keyword vectors, the second test data is generated using a large language model.

[0052] In this embodiment, the large language model simulates the real-world data generation logic in a production environment based on input keyword information. Through training on a large corpus, the large language model can capture the contextual relationships and language patterns between different types of text, thereby generating meaningful text content. For example, in a public opinion monitoring system, the large language model can generate a user comment or news summary that conforms to given industry, sentiment, and regional attributes. The test data generated by the large language model is closer to real-world scenarios.

[0053] By introducing a large language model, this embodiment achieves efficient conversion from keyword vectors to specific test data, which can reduce the workload of manually writing test data, improve the efficiency of test data generation, reduce testing costs, and accelerate product iteration. Ultimately, the system can improve the overall software quality and stability.

[0054] In this embodiment of the application, the large language model is a large language model that has undergone pre-training and fine-tuning; wherein... The pre-training process includes: Based on unlabeled corpus data, the large language model is trained by autoregression to obtain a pre-trained large language model. The fine-tuning process includes: The multiple first test data are processed into multiple sequence data; Based on the multiple sequence data and the multiple first keyword vectors, the pre-trained large language model is fine-tuned to obtain the fine-tuned large language model.

[0055] In this embodiment, during the pre-training stage, a large amount of unlabeled corpus data is fed into a large language model for autoregressive pre-training. The model parameters are continuously optimized by maximizing the self-supervised loss. The goal of this stage is to learn the general features and patterns of language. Individual training corpora are serialized into sequences of a specific length. During autoregressive training, the value at the next position is continuously predicted using the values ​​of the preceding sequence, and the model parameters are optimized based on maximizing the likelihood function. ,in, These are model parameters; In this embodiment of the application, during the fine-tuning stage, the first test data after preprocessing is processed into a sequence. and the first keyword vector The data is fed into the model as the training set for training. The first keyword vector is used as the model input, and the model output is compared with the sequence. Calculate the cross-entropy loss. After fine-tuning the training, the keyword data generated in batches based on the Gaussian mixture model is fed into the large language model, and then deserialized to obtain generated data with a distribution consistent with the existing test data.

[0056] The data construction method, data construction apparatus, electronic device, and storage medium provided in this application first collect multiple test data points from the production environment as real data samples; secondly, a Gaussian mixture model is fitted based on these samples to establish a data distribution model consistent with the production environment; finally, keyword vectors are generated through this model, and second test data is further generated. In this way, by using a Gaussian mixture model to simulate the complex data distribution in the production environment, the generated test data is statistically closer to the actual scenario, improving the representativeness of the test data and further enhancing the reliability of the test.

[0057] This application also provides a data construction apparatus, referring to... Figure 2 , Figure 2 This is a schematic diagram of the data construction apparatus provided in an embodiment of this application. The data construction apparatus in this embodiment includes: Data processing module: used to collect multiple first test data from the production environment; Distribution fitting module: used to fit the Gaussian mixture model based on the multiple first test data to obtain the fitted Gaussian mixture model; Data generation module: used to generate keyword vectors that conform to the data distribution of the production environment through the fitted Gaussian mixture model, and generate second test data based on the keyword vectors.

[0058] In this embodiment of the application, the distribution fitting module is used to encode the multiple first test data based on the attribute information of the multiple first test data to obtain multiple first keyword vectors; Based on the multiple first keyword vectors, the Gaussian mixture model is fitted to obtain the fitted Gaussian mixture model.

[0059] In this embodiment of the application, the distribution fitting module is used to fit the Gaussian mixture model based on the plurality of first keyword vectors using the expectation-maximization algorithm to obtain the fitted Gaussian mixture model.

[0060] In this embodiment of the application, the data generation module is used to sample data based on the weight of each Gaussian component in the Gaussian mixture model to obtain the keyword vector.

[0061] In this embodiment of the application, the data generation module is used to generate the second test data based on the keyword vector using a large language model.

[0062] In this embodiment of the application, the large language model is a large language model that has been pre-trained and fine-tuned; the distribution fitting module is used to perform autoregressive training on the large language model based on unlabeled corpus data to obtain a pre-trained large language model; to perform sequence processing on the multiple first test data to obtain multiple sequence data; and to fine-tune the pre-trained large language model based on the multiple sequence data and the multiple first keyword vectors to obtain a fine-tuned large language model.

[0063] In this embodiment of the application, the data processing module is used to perform preprocessing operations on the plurality of first test data; the preprocessing operations include one or more of the following operations: removing irrelevant data, formatting processing, and length filtering.

[0064] Those skilled in the art should understand that Figure 2 The functions of each unit in the data construction device shown can be understood by referring to the relevant description of the aforementioned method. Figure 2 The functions of each unit in the data construction device shown can be implemented by a program running on a processor or by specific logic circuits.

[0065] Figure 3 This is a schematic structural diagram of an electronic device provided in an embodiment of this application. Figure 3The electronic device shown includes a processor 310, which can call and run computer programs from memory to implement the data construction method provided in the embodiments of this application.

[0066] Optionally, such as Figure 3 As shown, the electronic device may also include a memory 320. The processor 310 can retrieve and run computer programs from the memory 320 to implement the data construction method provided in the embodiments of this application.

[0067] The memory 320 can be a separate device independent of the processor 310, or it can be integrated into the processor 310.

[0068] Optionally, such as Figure 3 As shown, the electronic device may also include a transceiver 330, which the processor 310 can control to communicate with other devices. Specifically, it can send information or data to other devices or receive information or data sent by other devices.

[0069] The transceiver 330 may include a transmitter and a receiver. The transceiver 330 may further include an antenna, and the number of antennas may be one or more.

[0070] The electronic device can implement the corresponding processes implemented by the data construction device in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0071] For example, embodiments of this application also provide a computer program product, including a computer program that can be executed by a processor 310 of an electronic device to perform the steps described in any of the foregoing methods.

[0072] Figure 4 This is a schematic structural diagram of the chip according to an embodiment of this application. Figure 4 The chip shown includes a processor 410, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0073] Optionally, such as Figure 4 As shown, the chip may also include a memory 420. The processor 410 can retrieve and run computer programs from the memory 420 to implement the methods described in this embodiment.

[0074] The memory 420 can be a separate device independent of the processor 410, or it can be integrated into the processor 410.

[0075] Optionally, the chip may also include an input interface 430. The processor 410 can control the input interface 430 to communicate with other devices or chips; specifically, it can acquire information or data sent by other devices or chips.

[0076] Optionally, the chip may also include an output interface 440. The processor 410 can control the output interface 440 to communicate with other devices or chips, specifically, to output information or data to other devices or chips.

[0077] This chip can be applied to the electronic devices in the embodiments of this application, and the chip can implement the corresponding processes implemented by the electronic devices in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0078] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0079] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0080] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0081] It should be understood that the above-described memory is exemplary and not a limiting description. For example, the memory in the embodiments of this application may also be static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DR RAM), etc. That is to say, the memory in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0082] This application also provides a storage medium for storing a computer program. This storage medium can be applied to the electronic device in this application embodiment, and the computer program causes the computer to execute the corresponding processes implemented by the electronic device in the various methods of this application embodiment; for brevity, further details are omitted here.

[0083] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0084] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0085] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0086] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0087] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0088] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or electronic device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0089] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data construction method, characterized in that, include: Multiple sets of initial test data were collected from the production environment; Based on the multiple first test data, the Gaussian mixture model is fitted to obtain the fitted Gaussian mixture model. The fitted Gaussian mixture model is used to generate keyword vectors that conform to the data distribution of the production environment, and second test data is generated based on the keyword vectors.

2. The method according to claim 1, characterized in that, The process of fitting the Gaussian mixture model based on the plurality of first test data to obtain the fitted Gaussian mixture model includes: Based on the attribute information of the multiple first test data, the multiple first test data are encoded to obtain multiple first keyword vectors; Based on the multiple first keyword vectors, the Gaussian mixture model is fitted to obtain the fitted Gaussian mixture model.

3. The method according to claim 2, characterized in that, The process of fitting the Gaussian mixture model based on the plurality of first keyword vectors to obtain the fitted Gaussian mixture model includes: Based on the multiple first keyword vectors, the Gaussian mixture model is fitted using the expectation-maximization algorithm to obtain the fitted Gaussian mixture model.

4. The method according to claim 3, characterized in that, The step of generating a keyword vector that conforms to the data distribution of the production environment using the fitted Gaussian mixture model includes: The keyword vector is obtained by sampling data based on the weight of each Gaussian component in the Gaussian mixture model.

5. The method according to claim 2, characterized in that, The generation of second test data based on the keyword vector includes: Based on the keyword vectors, the second test data is generated using a large language model.

6. The method according to claim 5, characterized in that, The large language model is a pre-trained and fine-tuned large language model; among which... The pre-training process includes: Based on unlabeled corpus data, the large language model is trained by autoregression to obtain a pre-trained large language model. The fine-tuning process includes: The multiple first test data are processed into multiple sequence data; Based on the multiple sequence data and the multiple first keyword vectors, the pre-trained large language model is fine-tuned to obtain the fine-tuned large language model.

7. The method according to any one of claims 1 to 6, characterized in that, After obtaining multiple sets of first test data from the production environment, the method further includes: The plurality of first test data are preprocessed; the preprocessing operations include one or more of the following operations: removing irrelevant data, formatting, and length filtering.

8. A data construction apparatus, characterized in that, Data processing module: used to collect multiple first test data from the production environment; Distribution fitting module: used to fit the Gaussian mixture model based on the multiple first test data to obtain the fitted Gaussian mixture model; Data generation module: used to generate keyword vectors that conform to the data distribution of the production environment through the fitted Gaussian mixture model, and generate second test data based on the keyword vectors.

9. An electronic device, characterized in that, include: A processor and a memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the data construction method as described in any one of claims 5 to 7.

10. A storage medium, characterized in that, Used to store a computer program that causes a computer to perform the data construction method as described in any one of claims 1 to 7.

Images