Pulsar candidate recognition method based on multi-modal fusion and generative model
Through the combination of multimodal fusion technology and generative model, the problem of data imbalance in pulsar candidate recognition is solved, the recognition accuracy is improved, and the pulsar candidate data is enhanced through VQVAE and GPT-2 generation models, and the pulsar candidate data is enhanced in 2 The channel attention mechanism is introduced in the dimensional diagnostic diagram modeling, which enhances the model's attention to real pulsar signals and achieves higher recognition accuracy and reliability.
Patent Information
- Application Number
- CN202411084282.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-11-29
AI Technical Summary
The prior art has data imbalance in the recognition of pulsar candidates, which has affected the model performance and it is difficult to quickly and accurately identify real pulsar signals.
Using multimodal fusion technology and generative model, the pulsar candidate data is enhanced through VQVAE and GPT-2 generative models, and the channel attention mechanism is added during the modeling of the two-dimensional diagnostic diagram, and the original data and generated data are established. Deep learning model, identifying and feature fusion of different diagnostic maps, and improving the model's attention to real pulsar signals.
It effectively solves the problem of data imbalance, improves the accuracy of pulsar candidate recognition and the recognition ability of the model, and significantly improves the recognition accuracy and reliability.
Smart Images

Figure CN119046725A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of pulsar candidate identification, and in particular to a pulsar candidate identification method based on multimodal fusion and generation model. Background Art
[0002] Pulsars are highly magnetized rotating compact stars and are natural laboratories for studying strong gravitational fields. The discovery of the first binary pulsar, PSR B913+16, provides indirect evidence for the existence of gravitational waves. Its orbital period tends to decrease, and this orbital decay is an important verification of Einstein's general theory of relativity. In terms of applied research, due to the stability of the pulsar rotation period, it can be used to establish time standards and spacecraft navigation. Therefore, it is necessary to discover new pulsars to tap their huge scientific research potential.
[0003] Since the discovery of pulsars, astronomers have observed more than 3,300 pulsars through radio telescopes and predict that about 20,000 pulsars will be found. In order to find more pulsars, some advanced modern radio telescopes have been put into use, such as the Square Kilometer Array (SKA) and the Five-hundred-meter Aperture Spherical Radio Telescope (FAST). As for FAST, by using 19-beam receivers for drift, tracking, motion scanning, and compilation scanning, it has higher measurement accuracy and can detect weaker signals from celestial bodies and artificial radio equipment. More than 800 pulsars have been discovered so far, and more than 1 million pulsar candidates can be generated every night.
[0004] Traditionally, the signals used for pulsar candidate identification are top candidates that have been folded, de-dispersed, and fast Fourier transformed. They are four-dimensional data consisting of time, frequency, rotation phase, and signal power, and are visualized as diagnostic graph data. Pulsar astronomers use these diagnostic graphs to identify real pulsar signals, but during the search process, the astronomical information observed by the telescope not only contains a large amount of pulsar information, but also contains more non-pulsar information and a large amount of artificial radio frequency interference, making the search for pulsars very difficult. Relying solely on manual investigation and screening by astronomers is seriously inconsistent with such a large number of pulsar candidates. How to quickly and accurately identify real pulsar signals is an urgent problem that needs to be solved. In addition, there is a serious data imbalance problem in the pulsar dataset. The number of real pulsar signals in the dataset is far lower than that of non-pulsar signals. This imbalance problem will have a negative impact on the performance of the model. Summary of the invention
[0005] The present invention aims to solve the problem of data imbalance in pulsar candidate datasets and improve recognition accuracy. By adopting deep learning algorithms such as multimodal fusion technology and generative models, a new pulsar candidate recognition method is proposed, specifically a pulsar candidate recognition method based on multimodal fusion and generative models is proposed.
[0006] The present invention is implemented by the following technical scheme. The present invention proposes a pulsar candidate identification method based on multimodal fusion and generation model, and the method comprises:
[0007] S1: Preprocessing of pulsar candidate signals;
[0008] S2: The pre-processed raw pulsar candidate data is enhanced using VQVAE and GPT-2 generative models;
[0009] S3: Improve the loss function of the multimodal model using a multi-loss fusion method;
[0010] S4: In the process of modeling the two-dimensional diagnostic graph, a channel attention mechanism is added to make the model pay more attention to the characteristics of the real pulsar signal;
[0011] S5: Combine the original data and generated data to establish a deep learning model using multimodal fusion technology to identify different diagnostic graphs. Before the respective results are obtained, the diagnostic graph features extracted by the model are fused and classification is performed based on the fused features.
[0012] Furthermore, in S1, during the acquisition of pulsar candidate data, periodic radio frequency interference is eliminated by removing channels corresponding to known interference frequencies, de-dispersion processing is performed, and multiple dispersion DM values are tested to determine the optimal DM value. Periodic signals are searched using Fourier analysis, and signals related to strong spectral features are eliminated to generate candidate signals.
[0013] Furthermore, the diagnostic graphs of the candidate signals are extracted by PRESTO, including pulse profile curves, DM curves, subband graphs and subintegral graphs, and these diagnostic graphs are used as features to conduct classification research on pulsar candidates.
[0014] Furthermore, in S2, VQVAE reduces the dimension of the input image and decodes and reconstructs it through vector quantization to generate a clear image; GPT-2 fits the coding table to generate a new image with a randomly generated coding vector.
[0015] Furthermore, in S3, the models all use the cross entropy loss function and the Adam optimizer; in order to improve the rationality of feature extraction by a single model, a loss function is added after each feature to ensure that the fused model can consider the role of a single model during back propagation; the total loss is the weighted sum of the loss of each model and the overall model loss,
[0016]
[0017] Furthermore, in S5, a one-dimensional convolution module is used to model the one-dimensional data of the pulse profile curve and the DM curve, and a sub-band graph and a sub-integral Figure 2 Dimensional data is modeled using a two-dimensional convolutional module.
[0018] Furthermore, the multimodal fusion technology adopts an intermediate fusion method to perform fusion at the fully connected layer of each model, and classifies and identifies pulsar candidate signals based on the fused features.
[0019] The present invention proposes a pulsar candidate identification system based on multimodal fusion and generation model, the system comprising:
[0020] Preprocessing module: preprocess the pulsar candidate signals;
[0021] Enhancement module: VQVAE and GPT-2 generative model are used to enhance the pre-processed raw pulsar candidate data;
[0022] Improvement module: Improve the loss function of the multimodal model using a multi-loss fusion method;
[0023] Modeling module: In the modeling process of the two-dimensional diagnostic graph, a channel attention mechanism is added to make the model pay more attention to the characteristics of the real pulsar signal;
[0024] Classification module: Combine the original data and generated data to establish a deep learning model using multimodal fusion technology to identify different diagnostic images. Before the respective results are obtained, the diagnostic image features extracted by the model are fused and classification is performed based on the fused features.
[0025] The present invention also proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method for identifying pulsar candidates based on multimodal fusion and generation model are implemented.
[0026] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the method for identifying pulsar candidates based on multimodal fusion and generation model.
[0027] Beneficial effects of the present invention:
[0028] The present invention proposes a pulsar candidate identification method based on multimodal fusion and generative model, which is applied to the field of pulsar candidate signal identification. By combining VQVAE and GPT-2 for data generation, the data imbalance problem is effectively solved. The method uses multiple models to identify different diagnostic graphs, and before the final output result, the extracted diagnostic graph features are fused to make full use of the complementarity between different diagnostic graphs. When modeling the two-dimensional diagnostic graph, the present invention introduces a channel attention mechanism, which enhances the model's attention to the key features of the real pulsar signal and improves the accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0030] Figure 1 This is the structure diagram of the generative model combining VQVAE and GPT-2.
[0031] Figure 2 This is a diagram of the network structure for pulsar candidate identification based on multimodality. DETAILED DESCRIPTION
[0032] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0033] The present invention proposes a pulsar candidate identification method based on multimodal fusion and generative model. The imbalance problem of pulsar candidate data is a significant challenge in pulsar candidate data sets. For example, in the FAST data set, even if all signals are incorrectly classified as non-pulsar signals, an accuracy rate of 98% can still be obtained. This imbalance problem will cause the accuracy of the prediction model to be falsely overstated. In addition, pulsar signals are very weak and easily submerged in background noise, so how to accurately identify pulsar candidates has become a problem that has attracted much attention from researchers.
[0034] Specifically, the present invention proposes a pulsar candidate identification method based on multimodal fusion and generative model, the method comprising the following steps:
[0035] S1: Preprocessing of pulsar candidate signals;
[0036] S2: The pre-processed raw pulsar candidate data is enhanced using VQVAE and GPT-2 generative models;
[0037] S3: Improve the loss function of the multimodal model using a multi-loss fusion method;
[0038] S4: In the process of modeling the two-dimensional diagnostic graph, a channel attention mechanism is added to make the model pay more attention to the characteristics of the real pulsar signal;
[0039] S5: Combine the original data and generated data to establish a deep learning model using multimodal fusion technology to identify different diagnostic graphs. Before the respective results are obtained, the diagnostic graph features extracted by the model are fused and classification is performed based on the fused features.
[0040] In the S1, in the process of acquiring pulsar candidate data, the periodic radio frequency interference is eliminated by removing the channel corresponding to the known interference frequency, and the dispersion processing is performed, and multiple dispersion amount (DM) values are tested to determine the optimal DM value, and the periodic signal is searched by Fourier analysis, and the signal related to the strong spectral feature is eliminated to generate the candidate signal. The diagnostic graph of the candidate signal is extracted by PRESTO, including the pulse profile curve, DM curve, subband graph and subintegral graph, and these diagnostic graphs are used as features for the classification study of pulsar candidates.
[0041] In S2, VQVAE reduces the dimension of the input image and decodes and reconstructs it through vector quantization to generate a clear image; GPT-2 fits the coding table to generate a new image with a randomly generated coding vector.
[0042] In S3, all models use the cross entropy loss function and Adam optimizer. To improve the rationality of extracting features from a single model, a loss function is added after each feature to ensure that the fused model can consider the role of a single model during back propagation. The total loss is the weighted sum of the loss of each model and the overall model loss.
[0043]
[0044] In S5, the one-dimensional data of the pulse profile curve and the DM curve are modeled using a one-dimensional convolution module, and the sub-band graph and the sub-integral Figure 2 Dimensional data is modeled using a two-dimensional convolutional module.
[0045] The multimodal fusion technology adopts an intermediate fusion method to perform fusion at the fully connected layer of each model, and classifies and identifies pulsar candidate signals based on the fused features.
[0046] The present invention proposes a pulsar candidate identification method based on multimodal fusion and generative model, aiming to solve the problem of imbalance of pulsar candidate data set and improve the recognition accuracy. The method includes using VQVAE (vector quantized variational autoencoder) and GPT-2 generative model to enhance the data. VQVAE reduces the dimension of the input image and decodes and reconstructs it to generate a clear image, while GPT-2 randomly generates new images by fitting the coding table. Combining the original and generated data, a deep learning model is established using multimodal fusion technology to identify different diagnostic graphs, classify them after fusion features, and make full use of the complementarity of diagnostic graphs. In the process of two-dimensional diagnostic graph modeling, a channel attention mechanism is introduced to make the model pay more attention to the real pulsar signal characteristics and improve the accuracy. The present invention is applicable to the field of pulsar signal detection and significantly improves recognition accuracy and reliability.
[0047] The experimental results of the present invention on the pulsar candidate dataset are described in detail below.
[0048] Three datasets were constructed, and the model training was performed separately and the role of generated data in the training model was analyzed. In Table 1, dataset 1 uses the original dataset, with a total of 11,696 samples, including 696 pulsar samples and 11,000 non-pulsar samples, and the ratio of positive and negative samples is approximately 1:15. On the basis of dataset 1, generated samples were added to dataset 2, with a total of 21,696 samples, of which the number of non-pulsar samples remained unchanged and the number of pulsar samples increased to 10,696, so that the dataset achieved sample balance and the ratio of positive and negative samples was 1:1. On the basis of dataset 2, non-pulsar samples were added to dataset 3, with a total number of 60,696 samples, and the ratio of positive and negative samples was 1:5. The 696 real pulsar samples in the above datasets are consistent with the samples used in the VQVAE generation model. The test set selects 500 pulsar samples and 500 non-pulsar samples. To ensure the accuracy of the model results, the samples selected in the test set have never appeared in the VQVAE model or the subsequent training model. At the same time, due to the large number of non-pulsar samples, multiple test sets will be established to test the robustness of the constructed model to non-pulsar samples.
[0049] Table 1
[0050]
[0051] After the model training is completed, the same test set is used for all data sets, and evaluation indicators such as f1-score, recall and precision are used for evaluation. Due to the large number of non-pulsar samples, 20 groups of data are randomly selected from the remaining non-pulsar data, with 500 samples in each group participating in the test together with the pulsar samples. The average results are shown in Table 2. When calculating the recall value, the positive and negative samples are evaluated separately, which more intuitively reflects the fitting effect of the model for the candidate pulsar samples. It can be seen that the Set1 data set has the worst model fitting effect due to the serious imbalance of positive and negative samples, and non-pulsar samples occupy a large proportion. Only the recall rate of non-pulsar samples has achieved a better effect. However, since the Set2 data set has both balanced data and increased data volume, the model fitting effect is the best, with an f1-score of 99.59%, a precision of 99.60%, a recall rate of 99.20% for non-pulsar samples, and a recall rate of 100% for pulsar samples. The Set3 dataset increases the number of non-pulsar samples, making the model more accurate in judging non-pulsar samples. However, the overall number of samples is larger, and the final model fitting effect is better than that of Set1. For the Set2 dataset, the misclassified samples are mainly concentrated in non-pulsar samples. It is observed that the erroneous samples are greatly affected by RFI, and their shapes are similar to those of real pulsar samples, resulting in misclassification.
[0052] Table 2
[0053]
[0054] The present invention also proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method for identifying pulsar candidates based on multimodal fusion and generation model are implemented.
[0055] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the method for identifying pulsar candidates based on multimodal fusion and generation model.
[0056] The memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DRRAM). It should be noted that the memory of the method described in the present invention is intended to include, but is not limited to, these and any other suitable types of memory.
[0057] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disc (SSD)).
[0058] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.
[0059] It should be noted that the processor in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor can be combined and performed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0060] The above is a detailed introduction to the pulsar candidate identification method based on multimodal fusion and generative model proposed in the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation to the present invention.
Claims
1. A pulsar candidate identification method based on multimodal fusion and generative model, characterized in that: The method comprises: S1: Preprocessing of pulsar candidate signals; S2: The pre-processed raw pulsar candidate data is enhanced using VQVAE and GPT-2 generative models; S3: Improve the loss function of the multimodal model using a multi-loss fusion method; S4: In the process of modeling the two-dimensional diagnostic graph, a channel attention mechanism is added to make the model pay more attention to the characteristics of the real pulsar signal; S5: Combine the original data and generated data to establish a deep learning model using multimodal fusion technology to identify different diagnostic graphs. Before the respective results are obtained, the diagnostic graph features extracted by the model are fused and classification is performed based on the fused features.
2. The method according to claim 1, characterized in that In S1, during the acquisition of pulsar candidate data, periodic radio frequency interference is eliminated by removing channels corresponding to known interference frequencies, de-dispersion processing is performed, and multiple dispersion DM values are tested to determine the optimal DM value. Periodic signals are searched using Fourier analysis, and signals related to strong spectral features are eliminated to generate candidate signals.
3. The method according to claim 2, characterized in that The diagnostic graphs of the candidate signal are extracted by PRESTO, including pulse profile curves, DM curves, subband graphs and subintegral graphs, and these diagnostic graphs are used as features to conduct classification research on pulsar candidates.
4. The method according to claim 1, characterized in that: In S2, VQVAE reduces the dimension of the input image and decodes and reconstructs it through vector quantization to generate a clear image; GPT-2 fits the coding table to generate a new image with a randomly generated coding vector.
5. The method according to claim 1, characterized in that In S3, the models all use the cross entropy loss function and the Adam optimizer. To improve the rationality of feature extraction by a single model, a loss function is added after each feature to ensure that the fused model can consider the role of a single model during back propagation. The total loss is the weighted sum of the loss of each model and the overall model loss.
6. The method according to claim 3, characterized in that: In S5, a one-dimensional convolution module is used to model the one-dimensional data of the pulse profile curve and the DM curve, and a two-dimensional convolution module is used to model the two-dimensional data of the sub-band graph and the sub-integral graph.
7. The method according to claim 6, characterized in that The multimodal fusion technology adopts an intermediate fusion method to perform fusion at the fully connected layer of each model, and classifies and identifies pulsar candidate signals based on the fused features.
8. A pulsar candidate identification system based on multimodal fusion and generative model, characterized in that: The system comprises: Preprocessing module: preprocess the pulsar candidate signals; Enhancement module: VQVAE and GPT-2 generative model are used to enhance the pre-processed raw pulsar candidate data; Improvement module: Improve the loss function of the multimodal model using a multi-loss fusion method; Modeling module: In the modeling process of the two-dimensional diagnostic graph, a channel attention mechanism is added to make the model pay more attention to the characteristics of the real pulsar signal; Classification module: Combine the original data and generated data to establish a deep learning model using multimodal fusion technology to identify different diagnostic images. Before the respective results are obtained, the diagnostic image features extracted by the model are fused and classification is performed based on the fused features.
9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Multi-modal feature fusion method for pulsar candidate signals
CN116611025A
Intelligent diagnosis method, device and equipment for hip joint lesion based on GPT
CN116758341A
Cited By
End-to-end pulsar candidate body identification method and device, and computer equipment
CN120599387A
Pulsar candidate body identification method and system based on channel separation and text guidance
CN121412653A
Method and system for identifying pulsar candidates based on channel separation and text guidance
CN121412653B
Single pulse signal identification system and method based on SW architecture
CN121682469A
