Radiation source signal data set processing method and system suitable for deep learning input
By converting the format of the radiation source signal dataset and adding labels, a dataset suitable for deep learning is generated. This solves the problem in the existing technology that the dataset format is not suitable for direct use in deep learning, realizes flexible segmentation and organization of the dataset, and improves research efficiency and experimental reproducibility.
Patent Information
- Application Number
- CN202510614556.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-14
- Publication Date
- 2025-09-19
AI Technical Summary
The existing radiation source signal dataset format is not suitable for direct use in deep learning model training, which makes data processing cumbersome and error-prone, experimental results difficult to reproduce, and lacks feature organization methods for deep learning models, making it difficult to efficiently process large-scale datasets.
By creating a file acquisition loop, randomly extracting target format files, extracting the real and imaginary parts and converting them into multidimensional arrays, adding labels and saving them into PKL format files, generating multiple small PK1 format files, and vertically stacking them into an overall PK1 format file, it is suitable for deep learning input.
It achieves flexible segmentation and organization of data sets to meet the needs of deep learning models, improves research efficiency, reduces data processing costs, and promotes the reproduction and comparison of experimental results.
Smart Images
Figure CN120670375A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method and system for processing a radiation source signal data set suitable for deep learning input, and belongs to the technical field of data processing. Background Art
[0002] With the widespread application of deep learning technology in indoor positioning, a large number of researchers have begun to explore the application of existing emitter signal datasets to deep learning model training. Existing emitter signal datasets are typically collected and publicly available by research institutions, companies, or open source communities. These datasets contain rich emitter signal feature information and corresponding location labels. However, the format, structure, and organization of these raw datasets are often not suitable for direct training of deep learning models.
[0003] Currently, when researchers use these existing datasets, they usually need to manually write data processing scripts or use general data processing tools to convert data formats. These processing methods mainly include simple file format conversion, data cleaning, and basic feature extraction. However, this approach has many shortcomings: First, the data processing process is cumbersome and error-prone, and researchers need to invest a lot of time to understand the data structure and write conversion code; second, the processing methods used by different researchers are inconsistent, making it difficult to reproduce and compare experimental results; third, existing processing methods often only focus on basic format conversion, ignoring the special requirements of deep learning models for data organization; there is a lack of feature organization methods for deep learning, such as the inability to directly generate input formats suitable for convolutional neural networks; data batch processing capabilities are limited, making it difficult to efficiently process large-scale datasets. Summary of the Invention
[0004] The present invention aims to solve the technical problems in the prior art that simple conversion formats cannot meet the data format requirements of deep learning models and are difficult to efficiently process large-scale data sets, and further proposes a radiation source signal data set processing method and system suitable for deep learning input.
[0005] The technical solution adopted by the present invention to solve the above problems is: the present invention includes a method for processing a radiation source signal dataset suitable for deep learning input, the method comprising:
[0006] Create a file collection loop based on a preset distance range and emitter name group;
[0007] randomly extracting two target format files from the radiation source signal data to be processed through the file acquisition cycle;
[0008] Extract the real and imaginary parts of the data in the two target format files, and perform format integerization to obtain the real and imaginary data;
[0009] Integering the real part data and the imaginary part data into a multidimensional array;
[0010] Add labels to the multidimensional array according to the distance and radiation source transmitter name selected during file collection, and save them into a pkl format file;
[0011] Repeat the above steps starting from the step of extracting two target format files until the radiation source signal data to be processed is traversed to obtain multiple pk1 format files;
[0012] The data in the multiple pk1 format files are vertically superimposed to obtain an integrated pk1 format file.
[0013] In some embodiments, vertically stacking the data in the plurality of pk1 format files to obtain a whole pk1 format file includes:
[0014] The data in the plurality of pk1 format files are vertically superimposed using a python file to obtain a whole pk1 format file.
[0015] In some embodiments, randomly extracting two target format files from the radiation source signal data to be processed through the file acquisition loop includes:
[0016] Randomly selecting to-be-processed radiation source signal data corresponding to a certain distance and a certain radiation source transmitter name from the preset distance range and radiation source transmitter name group, extracting the target format file, and obtaining a first target format file;
[0017] Again, the radiation source signal data to be processed corresponding to a certain distance and a certain radiation source transmitter name is randomly selected from the preset distance range and radiation source transmitter name group, and the target format file is extracted to obtain a second target format file.
[0018] In some embodiments, the process of obtaining the preset distance range and radiation source transmitter name group includes:
[0019] Confirm all distance data and radiation source transmitter names involved in the radiation source signal data to be processed, and establish the preset distance range and radiation source transmitter name group.
[0020] In some embodiments, the target format file is a run1 format file.
[0021] In some embodiments, the step of converting the real data and the imaginary data into a multidimensional array comprises:
[0022] Integering the real part data and the imaginary part data into a multidimensional array in the format of x*y*z;
[0023] Among them, x represents the number of data items, y is divided into two cases: 0 and 1, 0 stores real data, 1 stores imaginary data, and z represents the number of sampling points.
[0024] The present invention also includes a radiation source signal data set processing system suitable for deep learning input, the system comprising:
[0025] A creation unit, used for creating a file collection loop according to a preset distance range and radiation source transmitter name group;
[0026] An extraction unit, configured to randomly extract two target format files from the radiation source signal data to be processed through the file acquisition cycle;
[0027] The first integer unit is used to extract the real and imaginary parts of the data in the two target format files and perform format integeration to obtain real and imaginary data;
[0028] A second integer unit, configured to integer the real part data and the imaginary part data into a multi-dimensional array;
[0029] An adding unit, used for adding labels to the multidimensional array according to the distance and radiation source transmitter name selected during file collection, and saving the labels into a pkl format file;
[0030] A traversal unit, configured to repeat the above steps starting from the step of extracting two target format files, until the radiation source signal data to be processed is traversed to obtain a plurality of pk1 format files;
[0031] The processing unit is used to vertically stack the data in the multiple pk1 format files to obtain an integrated pk1 format file.
[0032] In some embodiments, the processing unit is configured to use a python file to vertically stack the data in the plurality of pk1 format files to obtain a whole pk1 format file.
[0033] In some embodiments, the extraction unit is used to randomly select the radiation source signal data to be processed corresponding to a certain distance and a certain radiation source transmitter name from the preset distance range and radiation source transmitter name group, extract the target format file, and obtain the first target format file; and again randomly select the radiation source signal data to be processed corresponding to a certain distance and a certain radiation source transmitter name from the preset distance range and radiation source transmitter name group, extract the target format file, and obtain the second target format file.
[0034] In some embodiments, the second integer unit is used to integer the real data and the imaginary data into a multi-dimensional array in the format of x*y*z;
[0035] Among them, x represents the number of data items, y is divided into two cases: 0 and 1, 0 stores real data, 1 stores imaginary data, and z represents the number of sampling points.
[0036] The beneficial effects of the present invention are:
[0037] 1. The present invention can split a large data set into required sizes according to the needs of the deep learning model.
[0038] 2. The present invention generates multiple small pkl files before generating an overall large pkl file dataset suitable for deep learning model input. The small pkl files are classified and named according to the names of different distances and radiation source transmitters. Researchers can select the desired distances and radiation sources as needed and then vertically stack them into a large dataset, which is more flexible and can meet more different data requirements. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0040] Figure 1 A flowchart of a method for processing a radiation source signal dataset suitable for deep learning input provided in this application;
[0041] Figure 2 This is a structural diagram of a radiation source signal dataset processing system suitable for deep learning input provided in this application. DETAILED DESCRIPTION Specific implementation method one:
[0043] Combine Figure 1 This embodiment provides a method for processing a radiation source signal dataset suitable for deep learning input, the method comprising:
[0044] S101, creating a file collection loop according to a preset distance range and radiation source transmitter name group;
[0045] S102, randomly extracting two target format files from the radiation source signal data to be processed through the file acquisition cycle;
[0046] S103, extracting the real and imaginary parts of the data in the two target format files, and performing formatting to obtain real and imaginary data;
[0047] S104, converting the real part data and the imaginary part data into a multi-dimensional array;
[0048] S105, adding labels to the multidimensional array according to the distance and radiation source transmitter name selected during file collection, and saving the labels into a pkl format file;
[0049] S106, starting from the step of extracting two target format files, repeating the above steps in sequence until the radiation source signal data to be processed is traversed to obtain multiple pk1 format files;
[0050] S107 , vertically superimposing the data in the multiple pk1 format files to obtain an integrated pk1 format file.
[0051] It should be noted that the file acquisition loop refers to obtaining files from a data set according to preset requirements or conditions. In this application, the radiation source signal data is collected under a certain distance from a certain radiation source transmitter, for example, the radiation source signal data is collected at a distance of 64 feet from the first radiation source transmitter. Specifically, the run1 format file of the data is obtained.
[0052] The present invention has developed a method specifically for organizing existing emitter signal datasets into datasets suitable for deep learning, which is of great significance. This method provides a standardized data conversion process, supports data formats for various deep learning frameworks, and exhibits good scalability and ease of use. This not only improves research efficiency and reduces data processing costs, but also facilitates the comparison and replication of different research results, promoting the development of emitter signal localization technology.
[0053] In some embodiments, vertically stacking the data in the plurality of pk1 format files to obtain a whole pk1 format file includes:
[0054] The data in the plurality of pk1 format files are vertically superimposed using a python file to obtain a whole pk1 format file.
[0055] It's important to note that in Python, there are multiple ways to stack two arrays, primarily using the NumPy library's numpy.vstack(), numpy.hstack(), and numpy.concatenate() functions. Each of these methods has its own purpose and suits different needs. Among them, numpy.vstack() is used for vertical stacking, numpy.hstack() for horizontal stacking, and numpy.concatenate() offers greater flexibility, allowing you to specify the axis along which to concatenate. The choice of method can be tailored to your needs and processing objectives.
[0056] In some embodiments, randomly extracting two target format files from the radiation source signal data to be processed through the file acquisition loop includes:
[0057] Randomly selecting to-be-processed radiation source signal data corresponding to a certain distance and a certain radiation source transmitter name from the preset distance range and radiation source transmitter name group, extracting the target format file, and obtaining a first target format file;
[0058] Again, the radiation source signal data to be processed corresponding to a certain distance and a certain radiation source transmitter name is randomly selected from the preset distance range and radiation source transmitter name group, and the target format file is extracted to obtain a second target format file.
[0059] It should be noted that, under normal circumstances, the radiation source signal data itself is limited according to the order of the radiation source transmitters and according to the different distances of different radiation source transmitters, so as to collect the radiation source signal data of the corresponding positions. For example, a certain radiation source signal data set contains signal data of 16 radiation source transmitters, which are 2 feet to 64 feet away from the first to the sixth radiation source transmitters, with a step of 6 feet. Therefore, two files are randomly collected each time, and the cycle is repeated until all files in the radiation source signal data are traversed and collected for subsequent format conversion.
[0060] In some embodiments, the process of obtaining the preset distance range and radiation source transmitter name group includes:
[0061] Confirm all distance data and radiation source transmitter names involved in the radiation source signal data to be processed, and establish the preset distance range and radiation source transmitter name group.
[0062] It should be noted that, according to the description in the previous part, under normal circumstances, the radiation source signal data itself is limited according to the order of the radiation source transmitters and according to the different distances of different radiation source transmitters to collect the radiation source signal data of the corresponding positions. Therefore, the preset distance range and radiation source transmitter name group as the traversal collection conditions include the corresponding content.
[0063] In some embodiments, the target format file is a run1 format file.
[0064] It should be noted that the run1 format file is an executable file, mainly used to execute a program or script to complete a specific task. In the present invention, it is used as a target format file for processing radiation source data.
[0065] In some embodiments, the step of converting the real data and the imaginary data into a multidimensional array comprises:
[0066] Integering the real part data and the imaginary part data into a multidimensional array in the format of x*y*z;
[0067] Among them, x represents the number of data items, y is divided into two cases: 0 and 1, 0 stores real data, 1 stores imaginary data, and z represents the number of sampling points.
[0068] It should be noted that before the data is reshaped into an array, Python can be used to extract the real and imaginary parts of the data in the target format file and define a simple function to implement it.
[0069] The following is a function extract_real_imag that takes a complex number and returns its real and imaginary parts.
[0070] def extract_real_imag(complex_number):
[0071] """Extract the real and imaginary parts of a complex number"""
[0072] real_part=complex_number.real
[0073] imag_part=complex_number.imag
[0074] return real_part,imag_part
[0075] Example usage:
[0076] if__name__=="__main__":
[0077] z=3+4j#Define complex number
[0078] real,imag=extract_real_imag(z)#Extract the real and imaginary parts
[0079] print(f"The real part of the complex number {z} is: {real}, the imaginary part is: {imag}")
[0080] Run results:
[0081] Running the above code will result in the following output:
[0082] The real part of the complex number (3+4j) is: 3.0, and the imaginary part is: 4.0.
[0083] After extracting the real data and imaginary data from the two target files respectively, reshape them and vertically stack them into two real and imaginary data, and then reshape the real data and imaginary data into a multidimensional array of x*y*z, where x represents the number of data items, y is divided into 0 and 1, 0 stores the real data, 1 stores the imaginary data, and z represents the number of sampling points.
[0084] Then, the multidimensional array is labeled according to the distance and radiation source transmitter name selected during file acquisition, and saved as a pkl format file. The file is re-acquired to generate a pk1 format file until all the radiation source signal data to be processed are traversed, and multiple small pk1 format files are obtained.
[0085] After obtaining small PKL files for each distance and source transmitter, researchers can extract the corresponding small PK1 files for the desired distance and source transmitter and organize them into a large PKL file based on their needs. The main process involves looping through the small PKL files, vertically stacking the data, and finally saving them into a large PKL file. The PK1 file itself is a format suitable for deep learning model input. Specific implementation method 2:
[0087] Based on the above embodiment of the present invention, a radiation source signal data set processing method suitable for deep learning input is disclosed. Figure 2 Specifically disclosed is a radiation source signal dataset processing system suitable for deep learning input that applies the radiation source signal dataset processing method suitable for deep learning input.
[0088] like Figure 2 As shown, an embodiment of the present invention discloses a radiation source signal dataset processing system suitable for deep learning input, the system comprising:
[0089] The creation unit 201 is used to create a file collection loop according to a preset distance range and radiation source transmitter name group;
[0090] An extraction unit 202 is configured to randomly extract two target format files from the radiation source signal data to be processed through the file acquisition loop;
[0091] The first integer unit 203 is used to extract the real and imaginary parts of the data in the two target format files and perform format integeration to obtain real and imaginary data;
[0092] A second integer unit 204 is used to integer the real part data and the imaginary part data into a multi-dimensional array;
[0093] An adding unit 205 is used to add labels to the multidimensional array according to the distance and radiation source transmitter name selected during file collection, and save the labels into a pkl format file;
[0094] A traversal unit 206 is configured to repeat the above steps starting from the step of extracting two target format files until the radiation source signal data to be processed is traversed to obtain a plurality of pk1 format files;
[0095] The processing unit 207 is configured to vertically stack the data in the multiple pk1 format files to obtain an integrated pk1 format file.
[0096] In some embodiments, the processing unit is configured to use a python file to vertically stack the data in the plurality of pk1 format files to obtain a whole pk1 format file.
[0097] In some embodiments, the extraction unit is used to randomly select the radiation source signal data to be processed corresponding to a certain distance and a certain radiation source transmitter name from the preset distance range and radiation source transmitter name group, extract the target format file, and obtain the first target format file; and again randomly select the radiation source signal data to be processed corresponding to a certain distance and a certain radiation source transmitter name from the preset distance range and radiation source transmitter name group, extract the target format file, and obtain the second target format file.
[0098] In some embodiments, the second integer unit is used to integer the real data and the imaginary data into a multi-dimensional array in the format of x*y*z;
[0099] Among them, x represents the number of data items, y is divided into two cases: 0 and 1, 0 stores real data, 1 stores imaginary data, and z represents the number of sampling points.
[0100] For the specific working process of the creation unit 201, the extraction unit 202, the first integer unit 203, the second integer unit 204, the addition unit 205, the traversal unit 206 and the processing unit 207 in the radiation source signal dataset processing system suitable for deep learning input disclosed in the above embodiment of the present invention, please refer to the corresponding content in the radiation source signal dataset processing method suitable for deep learning input disclosed in the above embodiment of the present invention, and will not be repeated here.
[0101] In summary, the embodiments of the present application provide a method and system for processing a radiation source signal data set suitable for deep learning input, including: creating a file acquisition loop according to a preset distance range and radiation source transmitter name group; randomly extracting two target format files from the radiation source signal data to be processed through the file acquisition loop; extracting the real and imaginary parts of the data in the two target format files, and performing format integerization to obtain real data and imaginary data; integerating the real data and the imaginary data into a multidimensional array; adding labels to the multidimensional array according to the distance and radiation source transmitter name selected during file acquisition, and saving them as a pkl format file; repeating the above steps in sequence starting from the step of extracting two target format files until the radiation source signal data to be processed is traversed to obtain multiple pk1 format files; vertically stacking the data in the multiple pk1 format files to obtain a whole pk1 format file.
[0102] The beneficial effects of the present invention are:
[0103] 1. The present invention can split a large data set into required sizes according to the needs of the deep learning model.
[0104] 2. The present invention generates multiple small pkl files before generating an overall large pkl file dataset suitable for deep learning model input. The small pkl files are classified and named according to the names of different distances and radiation source transmitters. Researchers can select the desired distances and radiation sources as needed and then vertically stack them into a large dataset, which is more flexible and can meet more different data requirements.
[0105] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the present profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical content disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement of the above embodiments made according to the technical essence of the present invention, within the spirit and principles of the present invention, without departing from the content of the technical solution of the present invention, shall still fall within the scope of protection of the technical solution of the present invention.
Claims
1. A method for processing radiation source signal datasets suitable for deep learning input, characterized in that: The method comprises: Create a file collection loop based on a preset distance range and emitter name group; randomly extracting two target format files from the radiation source signal data to be processed through the file acquisition cycle; Extract the real and imaginary parts of the data in the two target format files, and perform format integerization to obtain the real and imaginary data; Integering the real part data and the imaginary part data into a multidimensional array; Add labels to the multidimensional array according to the distance and radiation source transmitter name selected during file collection, and save them into a pkl format file; Repeat the above steps starting from the step of extracting two target format files until the radiation source signal data to be processed is traversed to obtain multiple pk1 format files; The data in the multiple pk1 format files are vertically superimposed to obtain an integrated pk1 format file.
2. The method according to claim 1, characterized in that The vertically stacking of the data in the plurality of pk1 format files to obtain a whole pk1 format file comprises: The data in the plurality of pk1 format files are vertically superimposed using a python file to obtain a whole pk1 format file.
3. The method according to claim 1, characterized in that The randomly extracting two target format files from the radiation source signal data to be processed through the file acquisition loop includes: Randomly selecting to-be-processed radiation source signal data corresponding to a certain distance and a certain radiation source transmitter name from the preset distance range and radiation source transmitter name group, extracting the target format file, and obtaining a first target format file; Again, the radiation source signal data to be processed corresponding to a certain distance and a certain radiation source transmitter name is randomly selected from the preset distance range and radiation source transmitter name group, and the target format file is extracted to obtain a second target format file.
4. The method according to claim 1, wherein The process of obtaining the preset distance range and radiation source transmitter name group includes: Confirm all distance data and radiation source transmitter names involved in the radiation source signal data to be processed, and establish the preset distance range and radiation source transmitter name group.
5. The method according to claim 1, wherein The target format file is a run1 format file.
6. The method according to claim 1, characterized in that The step of shaping the real part data and the imaginary part data into a multidimensional array comprises: Integering the real part data and the imaginary part data into a multidimensional array in the format of x*y*z; Among them, x represents the number of data items, y is divided into two cases: 0 and 1, 0 stores real data, 1 stores imaginary data, and z represents the number of sampling points.
7. A radiation source signal dataset processing system suitable for deep learning input, characterized in that: The system comprises: A creation unit, used for creating a file collection loop according to a preset distance range and radiation source transmitter name group; An extraction unit, configured to randomly extract two target format files from the radiation source signal data to be processed through the file acquisition cycle; The first integer unit is used to extract the real and imaginary parts of the data in the two target format files and perform format integeration to obtain real and imaginary data; A second integer unit, configured to integer the real part data and the imaginary part data into a multi-dimensional array; An adding unit is used to add labels to the multidimensional array according to the distance and radiation source transmitter name selected during file collection, and save the labels into a pkl format file; A traversal unit, configured to repeat the above steps starting from the step of extracting two target format files, until the radiation source signal data to be processed is traversed to obtain a plurality of pk1 format files; The processing unit is used to vertically stack the data in the multiple pk1 format files to obtain an integrated pk1 format file.
8. The system according to claim 7, characterized in that The processing unit is used to vertically stack the data in the multiple pk1 format files using a python file to obtain an integrated pk1 format file.
9. The system according to claim 7, wherein: The extraction unit is used to randomly select the radiation source signal data to be processed corresponding to a certain distance and a certain radiation source transmitter name from the preset distance range and radiation source transmitter name group, extract the target format file, and obtain a first target format file; and randomly select the radiation source signal data to be processed corresponding to a certain distance and a certain radiation source transmitter name from the preset distance range and radiation source transmitter name group again, extract the target format file, and obtain a second target format file.
10. The system according to claim 7, wherein: The second integer unit is used to integer the real part data and the imaginary part data into a multidimensional array in an x*y*z format; Among them, x represents the number of data items, y is divided into two cases: 0 and 1, 0 stores real data, 1 stores imaginary data, and z represents the number of sampling points.