Test data processing method, computer readable storage medium and electronic equipment
By obtaining multiple test data files, determining the number of concurrent threads, allocating memory for each thread, dividing the data set, using concurrent threads to write the data set to the database, and processing it in a pre-set standard data format, it solves the problem of inefficient efficiency when processing diversified test data provided by different test equipment in the prior art, and achieves efficient unified processing and resource conservation.
Patent Information
- Application Number
- CN202411646070.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-18
- Publication Date
- 2025-03-25
AI Technical Summary
The prior art is less efficient when processing diversified test data provided by different test equipment, resulting in waste of resources and high costs.
By obtaining multiple test data files, determining the number of concurrent threads, allocating memory for each thread, dividing the data set, using concurrent threads to write the data set to the database, and processing it in a pre-set standard data format.
It realizes efficient and unified processing of test data in different data formats, avoiding resource waste and effectively saving costs.
Smart Images
Figure CN119689307A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and specifically provides a test data processing method, a computer-readable storage medium and an electronic device. Background Art
[0002] In the field of energy storage system testing technology, the collection and analysis of battery cell test data is crucial for evaluating product performance and ensuring product quality. These test data are not only used for performance comparison research, but also play a key role in design links such as battery cell selection. However, with the expansion of laboratories and the diversification of testing needs, the sources of test equipment are becoming increasingly wide, and the test data formats provided by different test equipment are different, which brings challenges to data collection and processing.
[0003] Currently, facing the diverse sources of test data, a common solution is to rely on manual processing, which requires staff to classify and merge data according to specific needs and finally import them into a unified database. Another method is for each equipment manufacturer to provide unified and adapted test data. However, the above methods all have the problems of low efficiency and high cost. Summary of the invention
[0004] The present invention aims to solve the above technical problem, that is, to solve the problem of low efficiency in processing test data in different data formats.
[0005] In a first aspect, the present invention provides a method for processing test data, comprising:
[0006] Acquire multiple test data files; wherein the multiple test data files correspond to different test cycles, and each test data file contains at least one set of cyclic test data within the test cycle;
[0007] Obtaining free memory, and determining the number of concurrent threads based on the free memory and the cyclic test data of the multiple test data files;
[0008] Allocating a used memory to each thread based on the free memory and the number of concurrent threads, and dividing the allocated test data file based on the used memory allocated to each thread and the cyclic test data of the allocated test data file to obtain multiple data sets;
[0009] A concurrent thread is started to write the multiple data sets into the database respectively based on a preset standard data format.
[0010] In some embodiments, determining the number of concurrent threads based on the free memory and the cyclic test data of the plurality of test data files comprises:
[0011] respectively obtaining a single set of cycle test data volumes of the plurality of test data files;
[0012] Determine a maximum data volume from a single set of cycle test data volumes of the plurality of test data files;
[0013] The number of concurrent threads is determined based on the free memory and the maximum data amount.
[0014] In some embodiments, allocating memory for use to each thread based on the free memory and the number of concurrent threads includes:
[0015] The used memory is allocated to each thread according to the ratio of the free memory to the number of concurrent threads.
[0016] In some embodiments, dividing the allocated test data file based on the used memory allocated to each thread and the cyclic test data of the allocated test data file includes:
[0017] For each thread, obtain the single set of loop test data volume of the test data file assigned to the current thread;
[0018] Determine the batch processing amount of the cyclic test data based on the used memory allocated to the current thread and the amount of the single group of cyclic test data;
[0019] The test data file allocated to the current thread is divided based on the batch processing amount.
[0020] In some embodiments, dividing the test data file assigned to the current thread based on the batch processing amount includes:
[0021] Obtain the total data volume of the test data file allocated to the current thread;
[0022] Determining the number of divisions based on the total data volume and the batch processing volume;
[0023] The test data file allocated to the current thread is divided based on the division quantity.
[0024] In some embodiments, writing the plurality of data sets into a database based on a preset standard data format comprises:
[0025] For each of the multiple data sets, the current data set is parsed to obtain original data;
[0026] The original data is converted based on the standard data format and the converted data is written into the database.
[0027] In some embodiments, converting the original data based on the standard data format includes:
[0028] At least one of the unit, data type, data format, data naming, and data value of the original data is converted based on the standard data format.
[0029] In some embodiments, converting the data value of the original data based on the standard data format includes:
[0030] Performing validity judgment on the data value of the original data;
[0031] In response to the instruction being determined to be invalid, the data value is converted into a valid value.
[0032] In some embodiments, after converting the original data based on the standard data format, the method further includes:
[0033] Verifying the converted data based on the standard data format to determine abnormal data;
[0034] The conversion method is adjusted based on the abnormal data to convert the original data based on the adjusted conversion method.
[0035] In a second aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements any one of the above-mentioned test data processing methods.
[0036] In a third aspect, the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the test data processing method described in any one of the above is implemented.
[0037] In the case of adopting the above technical scheme, the present invention can provide a method for processing test data, by obtaining multiple test data files; multiple test data files correspond to different test cycles, and each test data file contains at least one cyclic test data within the test cycle; obtaining free memory, and determining the number of concurrent threads based on the free memory and the cyclic test data of multiple test data files; allocating used memory to each thread based on the free memory and the number of concurrent threads, and dividing the allocated test data files based on the used memory allocated to each thread and the cyclic test data of the allocated test data files to obtain multiple data sets; starting concurrent threads, and writing multiple data sets into a database based on a pre-set standard data format. The method can achieve efficient unified processing for test data of different data formats by dividing the test data files, starting concurrent threads, and processing the test data in combination with a pre-set standard data format, and can avoid waste of resources and effectively save costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings, in which:
[0039] Figure 1 It is a flowchart of a method for processing test data provided in an embodiment of the present application;
[0040] Figure 2 is a flowchart of the execution method of step S12 provided in an embodiment of the present application;
[0041] Figure 3 is a flowchart of the execution method of step S12 provided in an embodiment of the present application;
[0042] Figure 4 This is a schematic diagram of the structure of an electronic device provided by a specific example of this application. DETAILED DESCRIPTION
[0043] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the protection scope of the present application.
[0044] Based on the description of the background technology section, it can be seen that in the face of diverse sources of test data, the solution is usually to rely on manual processing or to have each equipment manufacturer provide uniformly adapted test data, but the above methods all have the problems of low efficiency and high cost. In view of this, the present application provides a method for processing test data. The method can achieve efficient and unified processing for test data in different data formats by dividing the test data file, starting concurrent threads, and processing the test data in combination with a pre-set standard data format, and can avoid waste of resources and effectively save costs.
[0045] See also Figure 1 As shown, Figure 1 It is a flowchart of a method for processing test data provided by an embodiment of the present application, which may include:
[0046] Step S11: obtaining a plurality of test data files; wherein the plurality of test data files correspond to different test cycles, and each test data file contains at least one set of cyclic test data within the test cycle;
[0047] Step S12: obtaining free memory, and determining the number of concurrent threads based on the free memory and the cyclic test data of the multiple test data files;
[0048] Step S13: allocating a used memory to each thread based on the free memory and the number of concurrent threads, and dividing the allocated test data file based on the used memory allocated to each thread and the cyclic test data of the allocated test data file to obtain multiple data sets;
[0049] Step S14: start concurrent threads and write multiple data sets into the database based on a preset standard data format.
[0050] In some embodiments, step S11 may specifically include: obtaining multiple test data files provided by different manufacturers, wherein the data formats of the test data files provided by different manufacturers may be different, and the same manufacturer may provide multiple test data files in different test cycles.
[0051] In other embodiments, step S11 may also be specifically as follows: directly read multiple pre-stored test data files. Specifically, according to the target directory path determined by the user, the file system API may be used to traverse all files under the target directory, and based on the screening conditions such as the file type suffix, identify whether the current file is a test data file that needs to be imported. The files that meet the screening conditions, i.e., the test data files that need to be imported, are recorded or the matching test data files are saved in a list. A list or set of matching files is returned so that they can be efficiently obtained later and used for subsequent processing or use. As an example, the set of returned files can be expressed as [file1, file2, file3...fileN], where fileN represents the Nth test data file, and N is a positive integer greater than or equal to 1.
[0052] The test cycle may include multiple cycle test phases, each of which generates a set of cycle test data. Each cycle test phase may correspond to a complete process of an energy storage device such as a battery from the beginning of charging to the end of discharging, and the cycle test data may include a set of performance parameters of the energy storage device during the process, such as temperature, voltage, and current.
[0053] In some embodiments, obtaining free memory in step S12 may specifically include: using a memory management interface provided by the operating system or a third-party library such as psutil to obtain the amount of available memory in the current system, that is, free memory.
[0054] In some embodiments, see Figure 2 As shown, in step S12, determining the number of concurrent threads based on free memory and cyclic test data of multiple test data files can be specifically as follows:
[0055] Step S121: respectively obtaining a single group of cycle test data volumes of a plurality of test data files;
[0056] Step S122: determining the maximum data volume from the single-group cycle test data volumes of the multiple test data files;
[0057] Step S123: Determine the number of concurrent threads based on the free memory and the maximum data volume.
[0058] As an example, when multiple test data files [file1, file2, file3...fileN] are obtained, for each test data file, the data amount of a group of cyclic test data in the test data file is obtained, wherein the data amounts of the multiple groups of cyclic test data contained in the test data file may be the same or different. When the data amounts of the multiple groups of cyclic test data are the same, one group may be randomly selected and the data amount may be obtained as the data amount of a single group of cyclic test data, such as obtaining the data amount of a group of cyclic test data in file3 as T3. When the data amounts of the multiple groups of cyclic test data are different, the data amount of each group may be obtained separately, such as obtaining the data amount of a group of cyclic test data in file4 as T41 or T42, both of which may be used as the data amount of a single group of cyclic test data.
[0059] In some embodiments, obtaining the amount of a single set of cycle test data can be specifically as follows: obtaining the data type in the cycle test data and the memory usage of the corresponding data type, and summing the memory usage of all data types to obtain the data amount of the cycle test data. Taking the test data of the battery cell as an example, its test data contains fields such as system time, number of cycles, voltage, current, temperature, etc., and the data types involved include text data, digital data, and date data. For each data type, the memory usage of the data type can be obtained according to the data length of the data type and the bytes occupied by the unit length. For example, for text data, it can be calculated based on the length of the text and the number of bytes occupied by the unit length, with each character occupying 2 bytes; for digital data, a 64-bit double-precision floating point number is used, which occupies 8 bytes; for date data, the date data is stored in the form of a date object, and usually occupies 8 bytes of memory space. Add the memory usage of text data, digital data, and date data to obtain the amount of a single set of cycle test data. Among them, determining the amount of a single set of cycle test data can be expressed by the following expression:
[0060] The amount of data for a single set of cyclic testing = text memory usage + digital memory usage + date memory usage, where text memory usage = text length × 2, digital memory usage = digital length × 8, and date memory usage = date length × 8.
[0061] As an example, when the data amounts of multiple groups of cyclic test data of each test data file are the same, each test data file corresponds to a single group of cyclic test data amount, such as the single group of cyclic test data amount corresponding to file3 is T3, the single group of cyclic test data amount corresponding to file2 is T2... The single group of cyclic test data amount corresponding to fileN is TN, then the maximum value can be determined from T2, T3...TN to obtain the maximum data amount. When the data amounts of multiple groups of cyclic test data of at least some test data files are different, that is, there are at least some test data files corresponding to multiple different single group of cyclic test data amounts, such as the single group of cyclic test data amount corresponding to file4 is T41 and T42, then the maximum value can be determined from T2, T3, T41, T42...TN to obtain the maximum data amount.
[0062] In some embodiments, step S123 may specifically be determining the number of concurrent threads according to the ratio of free memory to the maximum data volume.
[0063] In some embodiments, allocating memory usage to each thread based on free memory and the number of concurrent threads in step S13 may specifically include: allocating memory usage to each thread based on the ratio of free memory to the number of concurrent threads. The memory usage allocated to each thread is the ratio of free memory to the number of concurrent threads. In this method, by starting concurrent threads, system resources can be effectively utilized, resource waste or excessive occupation can be avoided, and data processing efficiency can be improved.
[0064] In some embodiments, see Figure 3 As shown, in step S13, dividing the allocated test data files based on the used memory allocated to each thread and the cyclic test data of the allocated test data files may include:
[0065] Step S131: for each thread, obtaining the single group of loop test data volume of the test data file allocated to the current thread;
[0066] Step S132: determining the batch processing amount of the cyclic test data based on the used memory allocated to the current thread and the amount of a single group of cyclic test data;
[0067] Step S133: Divide the test data files assigned to the current thread based on the batch processing amount.
[0068] In some embodiments, step S132 may specifically determine the batch processing amount of the cyclic test data according to the ratio of the used memory allocated to the current thread and the amount of the single group of cyclic test data. As an example, if the allocated used memory is 4G and the amount of the single group of cyclic test data is 2G, then the batch processing amount may be determined to be 2.
[0069] In some embodiments, step S133 may specifically include:
[0070] Get the total amount of data in the test data file allocated to the current thread;
[0071] Determine the number of partitions based on the total data volume and batch processing volume;
[0072] Divide the test data file assigned to the current thread based on the number of partitions.
[0073] The total data volume of the test data file can be obtained by obtaining the data volume of each cyclic test data in the test data file and summing the data volumes of all cyclic test data.
[0074] In some embodiments, the number of divisions can be determined according to the ratio of the total number to the batch processing amount, and the test data file assigned to the current thread can be divided based on the number of divisions. Based on this method, multiple test data files can be divided respectively, thereby obtaining multiple data sets, so as to facilitate the subsequent processing of each data set separately. This method can avoid reading multiple test data files at one time, avoid the risk of memory overflow and system crash caused by excessive file size, and is conducive to reducing memory pressure, improving the response speed of the system, and reducing data processing delays.
[0075] In some embodiments, step S14 may specifically be starting concurrent threads, and each thread may perform the following steps:
[0076] For each of the multiple data sets, the current data set is parsed to obtain original data;
[0077] The original data is converted based on the standard data format and the converted data is written into the database.
[0078] In some embodiments, the data set can be parsed using conventional parsing methods in the art, and the parsed data can be converted into a dictionary form. As an example, the dictionary form of a set of cycle test data in the data set can be expressed as dictA = {"cycle number": "N / A", "process number": 6,"record number": 18769,"step number": 2,"operating mode": "constant power charging", "total time": "5:12:46.053", "step time": "0:03:32.000", "voltage (V)": 3.27,"current (A)": 74.291,"charging capacity (Ah)": 16.912683,"charging energy (kWh) ":52.833172,"Discharge capacity (Ah)":0,"Discharge energy (kWh)":0,"Capacity difference (Ah)":16.912683,"Energy difference (kWh)":52.833172,"Power (W)":896.933655,"Impedance (Ω)":0.011922,"Internal resistance (mΩ)":0,"SOC (%)":5.724585,"Operating condition number":0,"Reference voltage (V)":3.6707,"System recording time":1999 / 01 / 02 23:15:18,"Recording position":4222456,"dQdV":57.011772,"Cell temperature 1 (℃)":52.2,"Cell temperature 2 (℃)":52.5,"Cell temperature 3 (℃)":48.2}.
[0079] In some embodiments, converting the original data based on the standard data format may include: converting at least one of the unit, data type, data format, data naming, and data value of the original data based on the standard data format.
[0080] As an example, the unit of charging energy can be converted to Wh (watt-hour), the data value can be converted from 52.833172 to 52833.172, and the data format can be unified, such as converting the system record time to 1999-01-02 23:15:18. You can also rename the key name of dictA according to the preset naming rules to ensure the consistency of naming of all data sets, such as expressing the data type in English and converting the cycle number to cycleNumber.
[0081] In some embodiments, converting the data value of the original data based on the standard data format includes:
[0082] Make validity judgment on the data value of the original data;
[0083] In response to the instruction being determined to be invalid, the data value is converted into a valid value.
[0084] As an example, the original value N / A of the cycle number may be converted to 0.
[0085] In some embodiments, in order to more effectively ensure the uniformity of data, the data arrangement order, the unit of the original data, the data type, the data format, the data naming and the data value can be converted based on the standard data format. Taking the dictA shown above as an example, after the conversion, dictA'={"cycleNumber":0,"stepNumber":2,"workMode":"constant power charging","stepTime":"0:03:32.000","totalTime":"5:12:46","vol":3.27,"current":74.291,"chargeCapacity":16.912683, "chargeEnergy":5283.3172,"disChargeCapacity":0, "disChargeEnergy":0,"power":96.933655,"systemTime":1999 / 01 / 02 23:15:18","cellTemperature1":52.2,"cellTemperature2":52.5,"cellTemperatur e3":48.2}. Among them, the data type name cycleNumber represents the cycle number, stepNumber represents the step number, workMode represents the work mode, stepTime represents the step time, totalTime represents the total time, vol represents the voltage, current represents the current, chargeCapacity represents the charging capacity, chargeEnergy represents the charging energy, disChargeCapacity represents the discharging capacity, disChargeEnergy: discharging energy, power represents the power, systemTime represents the system recording time, and cellTemperature1 / 2 / 3 represents the cell temperature 1 / 2 / 3.
[0086] In some embodiments, after converting the original data based on the standard data format, the method may further include:
[0087] Verify the converted data based on the standard data format and identify abnormal data;
[0088] The transformation method is adjusted based on the abnormal data to transform the original data based on the adjusted transformation method.
[0089] In some embodiments, the converted data type can be verified based on the standard data format to see whether it is correct, whether the data range is reasonable, whether there are missing values or abnormal values, etc. Data that does not conform to the standard data format can be determined as abnormal data.
[0090] In other embodiments, operations such as marking or deleting may also be performed on abnormal data.
[0091] It should be noted that the standard data format can be pre-set based on the storage requirements of the database and the multiple test data files obtained. For example, according to the data content of each set of cycle test data in multiple test data files, the data types, the arrangement order of the data types, the data naming of the data types, the data format and unit of the data values, etc. can be set. As an example, the data standard format can be set based on the following content: the first column cycleNumber is an integer, which indicates the cycle number corresponding to the current data; the second column stepNumber is an integer, which indicates the step number corresponding to the current data; the third column workMode is a string, which indicates the current working mode of the battery... The sixteenth column cellTemperature3 is a floating point number, which indicates the large surface temperature of the current battery in degrees Celsius.
[0092] The above is a method for processing test data provided by an embodiment of the present application, by obtaining multiple test data files; multiple test data files correspond to different test cycles, and each test data file contains at least one cyclic test data within the test cycle; obtaining free memory, and determining the number of concurrent threads based on the free memory and the cyclic test data of multiple test data files; allocating usable memory for each thread based on the free memory and the number of concurrent threads, and dividing the allocated test data files based on the usable memory allocated to each thread and the cyclic test data of the allocated test data files to obtain multiple data sets; starting concurrent threads, and writing multiple data sets into a database based on a pre-set standard data format. This method can achieve efficient unified processing for test data of different data formats by dividing the test data files, starting concurrent threads, and processing the test data in combination with a pre-set standard data format, and can avoid waste of resources and effectively save costs.
[0093] It is understood by those skilled in the art that all or part of the processes in the method for implementing the above-mentioned embodiments of the present invention can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium, etc. that can carry the computer program code.
[0094] Another aspect of the present invention further provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the test data processing method in any of the above embodiments can be implemented. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present invention is a non-transitory computer-readable storage medium.
[0095] Another aspect of the present application provides an electronic device, which may include at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores a computer program, and when the computer program is executed by the at least one processor, the method described in any of the above embodiments is implemented. Figure 4 As shown, for example, the memory 11 and the processor 12 may each be provided with one and may be communicatively connected via a bus.
[0096] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.
Claims
1. A method for processing test data, characterized in that: include: Acquire multiple test data files; wherein the multiple test data files correspond to different test cycles, and each test data file contains at least one set of cyclic test data within the test cycle; Obtaining free memory, and determining the number of concurrent threads based on the free memory and the cyclic test data of the multiple test data files; Allocating a used memory to each thread based on the free memory and the number of concurrent threads, and dividing the allocated test data file based on the used memory allocated to each thread and the cyclic test data of the allocated test data file to obtain multiple data sets; Concurrent threads are started to write the multiple data sets into the database respectively based on a preset standard data format.
2. The method according to claim 1, characterized in that Determining the number of concurrent threads based on the free memory and the cyclic test data of the plurality of test data files comprises: respectively obtaining a single set of cycle test data volumes of the plurality of test data files; Determine a maximum data volume from a single set of cycle test data volumes of the plurality of test data files; The number of concurrent threads is determined based on the free memory and the maximum data amount.
3. The method according to claim 1, characterized in that: Allocating memory for use to each thread based on the free memory and the number of concurrent threads includes: The used memory is allocated to each thread according to the ratio of the free memory to the number of concurrent threads.
4. The method according to claim 1, characterized in that: Dividing the allocated test data file based on the used memory allocated to each thread and the cyclic test data of the allocated test data file comprises: For each thread, obtain the single group of loop test data volume of the test data file assigned to the current thread; Determine the batch processing amount of the cyclic test data based on the used memory allocated to the current thread and the amount of the single group of cyclic test data; The test data file allocated to the current thread is divided based on the batch processing amount.
5. The method according to claim 4, characterized in that The dividing the test data file allocated to the current thread based on the batch processing amount includes: Obtain the total amount of data in the test data file allocated to the current thread; Determining the number of divisions based on the total data volume and the batch processing volume; The test data file allocated to the current thread is divided based on the division quantity.
6. The method according to any one of claims 1 to 5, characterized in that Writing the plurality of data sets into the database respectively based on a preset standard data format comprises: For each of the multiple data sets, the current data set is parsed to obtain original data; The original data is converted based on the standard data format and the converted data is written into the database.
7. The method according to claim 6, characterized in that Converting the original data based on the standard data format includes: At least one of the unit, data type, data format, data naming, and data value of the original data is converted based on the standard data format.
8. The method according to claim 7, characterized in that Converting the data value of the original data based on the standard data format includes: Performing validity judgment on the data value of the original data; In response to the instruction being determined to be invalid, the data value is converted into a valid value.
9. The method according to claim 6, characterized in that After converting the original data based on the standard data format, the method further includes: Verifying the converted data based on the standard data format to determine abnormal data; The conversion method is adjusted based on the abnormal data to convert the original data based on the adjusted conversion method.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the test data processing method according to any one of claims 1 to 9 is implemented.
11. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the test data processing method according to any one of claims 1 to 9 is implemented.