Remote sensing data extraction method and device, equipment and storage medium

Abstract

The application relates to the technical field of remote sensing communication, and discloses a remote sensing data extraction method, device, equipment and storage medium, the method comprising the following steps: creating an index for received satellite remote sensing data to obtain inverted index data; performing data preprocessing on the satellite remote sensing data according to the inverted index data to obtain each frame of target remote sensing data; and performing data merging on the each frame of target remote sensing data to obtain available remote sensing data. Since the inverted index data is created for the satellite remote sensing data, the specific position of the satellite remote sensing data can be found through the inverted index data when the remote sensing data is extracted, the slow extraction condition of the traditional extraction according to a frame header and the sorting and splicing according to frame counting are avoided, the data extraction efficiency is improved, and the timeliness of the remote sensing data is improved.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing communication technology, and in particular to a remote sensing data extraction method, device, equipment and storage medium. Background Art

[0002] With the advent of the data age, the amount of information and data is rapidly increasing. In the field of remote sensing and communication technology, satellite remote sensing data is typically large in volume. Due to the continuous accumulation and extremely complex nature of this data, the entire dataset is becoming increasingly difficult to process. Furthermore, because remote sensing data is transmitted from satellites to the ground via multiple channels, the same mission may be split into multiple data transmissions at any given point. The data received on the ground may be a mix of data from multiple satellites and different missions. Accurately and quickly extracting remote sensing data from this massive amount of data and restoring it to the original mission data is a prerequisite for remote sensing data analysis and application.

[0003] However, most traditional data extraction methods use the method of extracting according to the frame header or frame synchronization word, and then sorting and splicing according to the frame count. Since the timeliness of remote sensing data is more important in application scenarios such as weather forecasting and disaster relief, this method is still slow for extracting remote sensing data, which reduces the timeliness of remote sensing data.

[0004] The above content is only used to assist in understanding the technical solution of the present invention and does not constitute an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of the present invention is to provide a remote sensing data extraction method, device, equipment and storage medium, aiming to solve the technical problem that the traditional method of extracting according to frame header or frame synchronization word and then sorting and splicing according to frame count is still slow for extracting remote sensing data, thereby reducing the timeliness of remote sensing data.

[0006] To achieve the above object, the present invention provides a remote sensing data extraction method, which comprises the following steps:

[0007] Create an index for the received satellite remote sensing data to obtain inverted index data;

[0008] Performing data preprocessing on the satellite remote sensing data according to the inverted index data to obtain target remote sensing data of each frame;

[0009] Merging the target remote sensing data of each frame to obtain usable remote sensing data.

[0010] Optionally, indexing the received satellite remote sensing data to obtain inverted index data includes:

[0011] The received satellite remote sensing data is shunted through a message queue to obtain shunted remote sensing data;

[0012] The shunted remote sensing data is processed by word segmentation based on a Lucene library to obtain word segmentation data corresponding to the shunted remote sensing data;

[0013] Index creation is performed on the satellite remote sensing data according to the word segmentation data to obtain inverted index data.

[0014] Optionally, the data preprocessing on the satellite remote sensing data according to the inverted index data to obtain each frame of target remote sensing data comprises:

[0015] Frame query is performed on the satellite remote sensing data according to the inverted index data to obtain a remote sensing frame header corresponding to the satellite remote sensing data;

[0016] Similarity matching is performed on the remote sensing frame header and the inverted index data by a similarity matching algorithm to obtain a similarity value between the remote sensing frame header and the inverted index data;

[0017] Frame data in the satellite remote sensing data is repaired by the similarity value to obtain each frame of target remote sensing data.

[0018] Optionally, the similarity matching on the remote sensing frame header and the inverted index data by a similarity matching algorithm to obtain a similarity value between the remote sensing frame header and the inverted index data comprises:

[0019] The remote sensing frame header and the inverted index data are mapped into vectors to obtain a frame header vector corresponding to the remote sensing frame header and an index vector corresponding to the inverted index data;

[0020] The frame header vector and the index vector are respectively processed by vector angle to obtain a frame vector angle between the frame header vectors and an index vector angle between the index vectors;

[0021] Similarity matching is performed on the frame vector angle and the index vector angle by a similarity matching algorithm to obtain a similarity value between the remote sensing frame header and the inverted index data.

[0022] Optionally, the frame data in the satellite remote sensing data is repaired by the similarity value to obtain each frame of target remote sensing data, comprising:

[0023] It is judged whether the similarity value is greater than a preset threshold value, and a remote sensing frame header corresponding to the similarity value greater than the preset threshold value is taken as a frame header preselected point;

[0024] It is judged whether a frame distance between adjacent frame header preselected points reaches a preset frame length;

[0025] In the case where the frame distance does not reach the preset frame length, performing data correction on the remote sensing frame header corresponding to the remote sensing frame length that does not reach the preset frame length according to a preset data format to obtain corrected satellite remote sensing data;

[0026] Data extraction is performed on the frame data in the corrected satellite remote sensing data to obtain target remote sensing data of each frame.

[0027] Optionally, merging the target remote sensing data of each frame to obtain usable remote sensing data includes:

[0028] The mission time of extracting the satellite remote sensing data;

[0029] Sorting the target remote sensing data of each frame according to the mission time to obtain sorted data;

[0030] Positioning each frame of data in the sorted data to obtain a byte end position of the frame of data in bytes;

[0031] The sorted data are merged according to the end position of the byte to obtain usable remote sensing data.

[0032] Optionally, merging the sorted data according to the byte end position to obtain usable remote sensing data includes:

[0033] Performing offset calculation on each frame of data according to the byte end position to obtain a data offset corresponding to each frame of data;

[0034] When the data offset reaches a preset offset, frame interception is performed on each frame data corresponding to the preset offset to obtain intercepted frame data;

[0035] The sorted data and the intercepted frame data are merged according to the byte end position to obtain usable remote sensing data.

[0036] In addition, to achieve the above-mentioned purpose, the present invention further proposes a remote sensing data extraction device, the device comprising:

[0037] The inverted index module is used to create an index for the received satellite remote sensing data and obtain inverted index data;

[0038] A data extraction module is used to perform data preprocessing on the satellite remote sensing data according to the inverted index data to obtain target remote sensing data of each frame;

[0039] The data merging module is used to merge the target remote sensing data of each frame to obtain usable remote sensing data.

[0040] In addition, to achieve the above-mentioned purpose, the present invention also proposes a remote sensing data extraction device, which includes: a memory, a processor, and a remote sensing data extraction program stored in the memory and executable on the processor, wherein the remote sensing data extraction program is configured to implement the steps of the remote sensing data extraction method described above.

[0041] In addition, to achieve the above-mentioned purpose, the present invention further proposes a storage medium, on which a remote sensing data extraction program is stored. When the remote sensing data extraction program is executed by a processor, the steps of the remote sensing data extraction method described above are implemented.

[0042] The present invention creates an index for received satellite remote sensing data to obtain inverted index data; then preprocesses the satellite remote sensing data based on the inverted index data to obtain target remote sensing data for each frame; and finally, merges the target remote sensing data for each frame to obtain usable remote sensing data. Because the present invention creates inverted index data for satellite remote sensing data, the specific location of the satellite remote sensing data can be found through the inverted index data during data extraction. This avoids the traditional slow extraction method of extracting data based on frame headers and sorting and splicing based on frame counts, improves data extraction efficiency, and thus enhances the timeliness of remote sensing data in application scenarios such as weather forecasting and disaster relief. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a structural diagram of a remote sensing data extraction device in a hardware operating environment involved in an embodiment of the present invention;

[0044] Figure 2 This is a flow chart of a first embodiment of a remote sensing data extraction method according to the present invention;

[0045] Figure 3 This is a flow chart of a second embodiment of the remote sensing data extraction method of the present invention;

[0046] Figure 4 Schematic diagram of the flow of the third embodiment of the remote sensing data extraction method of the present invention;

[0047] Figure 5 This is the overall workflow of the third embodiment of the remote sensing data extraction method of the present invention;

[0048] Figure 6 This is a structural block diagram of the first embodiment of the remote sensing data extraction device of the present invention.

[0049] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a remote sensing data extraction device in a hardware operating environment according to an embodiment of the present invention.

[0052] like Figure 1 As shown, the remote sensing data extraction device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display and an input unit such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a wireless fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk storage device. The memory 1005 may also be a storage device independent of the processor 1001.

[0053] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the remote sensing data extraction device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0054] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a remote sensing data extraction program.

[0055] exist Figure 1 In the remote sensing data extraction device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the remote sensing data extraction device of the present invention can be set in the remote sensing data extraction device, and the remote sensing data extraction device calls the remote sensing data extraction program stored in the memory 1005 through the processor 1001 and executes the remote sensing data extraction method provided by the embodiment of the present invention.

[0056] The embodiment of the present application provides a remote sensing data extraction method, referring to Figure 2 , Figure 2 The figure is a flowchart of the first embodiment of the remote sensing data extraction method.

[0057] It should be noted that the remote sensing data formed after the satellite performs remote sensing observation on the earth has the characteristics of unstructured, large data volume and strong timeliness. In the process that the satellite transmits the remote sensing data to the ground receiving system, due to the influence of the ionosphere and solar radiation, the transmission data may have error codes or missing codes; in addition, the failure of the satellite sending end and the ground receiving end will also cause the data to be abnormal. If the data is abnormal, the shooting task will fail, which seriously affects the business progress.

[0058] Therefore, when the ground extracts the original satellite remote sensing data, the above-mentioned data abnormal error code and missing code problem also needs to be solved. Generally, the traditional solutions are as follows: 1. Re-transmit the frame data with transmission abnormalities. However, this method will occupy additional channels, which will block the normal transmission and download of other shooting task data, and the time cost of this method is high, and the timeliness of the remote sensing data is affected. 2. Discard the abnormal frame data. However, directly discarding the abnormal frame will cause problems in remote sensing image decoding, and the image is incomplete, which limits the practicability of the remote sensing data. 3. Repair the damaged data, repair the error code frame data through frame characteristics, and recover the remote sensing data to the maximum extent with the minimum cost. However, in the data repair process, the calculation amount increases; the repair success rate is unstable, and the timeliness of the remote sensing data will be affected. 4. Multiple ground stations are used to receive the same data during the ground receiving data process, to improve the data reliability. However, this method has the problems of data receiving and storage redundancy and large data volume, and the timeliness of the remote sensing data will also be affected.

[0059] Therefore, how to correctly and quickly extract the remote sensing data from a large amount of data and recover the original task data is the premise of remote sensing data analysis and application. In order to improve the timeliness of the remote sensing data, the embodiment of the present application provides a remote sensing data extraction method to speed up the extraction speed of the remote sensing data.

[0060] In the embodiment, the remote sensing data extraction method comprises the following steps:

[0061] Step S10: Index creation is performed on the received satellite remote sensing data to obtain inverted index data.

[0062] It should be noted that the execution entity of the method of this embodiment can be a computing service device with index creation and data processing capabilities, such as a personal computer or server, or other electronic devices capable of performing the same or similar functions, such as the aforementioned remote sensing data extraction device, and this embodiment is not limited thereto. The aforementioned remote sensing data extraction device (hereinafter referred to as the extraction device) will be used to specifically describe this embodiment and the following embodiments.

[0063] Satellite remote sensing data is information about the Earth's surface and atmosphere acquired from space by satellites or spacecraft. This includes images in different wavelengths, such as visible light, infrared, and microwaves, as well as altitude data, temperature data, and vegetation indices. Because satellite remote sensing data is transmitted from satellites to the ground via multiple channels, the same mission may be split into multiple data transmissions at any given point. The data received on the ground may be a mix of data from multiple satellites and different missions. Therefore, it is necessary to accurately and quickly extract satellite remote sensing data from this large amount of data and restore it to the original mission data.

[0064] It should be understood that inverted index data is index data obtained by constructing an inverted index, which is used to quickly locate the structured data of satellite remote sensing data. Inverted indexes originate from the need to find records based on attribute values ​​in practical applications. Each item in this index table includes an attribute value and the address of each record with that attribute value. Since the attribute value is determined by the location of the record rather than the record, it is called an inverted index. Data constructed using the inverted index source can be used to process large-scale text data sets such as satellite remote sensing data.

[0065] Both data extraction and repair require access to the data. Without an index, searching for frame headers or frame synchronization characters is slow, impacting data processing efficiency. Therefore, an inverted index is created before data extraction and repair. This index allows the specific location of frame data to be found during the extraction and repair process, accelerating data processing.

[0066] In a specific implementation, after receiving the satellite remote sensing data, the extraction device creates an inverted index according to the attribute values ​​of the satellite remote sensing data and obtains inverted index data to improve the speed of data extraction.

[0067] Step S20: performing data preprocessing on the satellite remote sensing data according to the inverted index data to obtain target remote sensing data of each frame.

[0068] It should be noted that each frame of target remote sensing data is obtained by extracting the frame data from the original satellite remote sensing data, analyzing the frame data, and repairing the frame data with errors and missing data. Since the remote sensing data generated by the satellite after remote sensing observation of the earth has the characteristics of being unstructured, large in data volume, and highly time-sensitive, when the satellite transmits the remote sensing data to the ground receiving system, it is affected by the ionosphere and solar radiation, and the transmitted data may have errors or missing data. Therefore, when the satellite remote sensing data is processed according to the inverted index data, preprocessing operations such as repairing the frame data with errors and missing data are also required to obtain each frame of target remote sensing data.

[0069] Step S30: merging the target remote sensing data of each frame to obtain usable remote sensing data.

[0070] It should be noted that, given that satellite remote sensing data is downloaded in blocks, to avoid frame errors at the splicing point, it is necessary to merge the target remote sensing data frames of different blocks to restore the complete mission data. For example, the specific stop and start positions of each target remote sensing data frame can be recorded to ensure byte integrity during the merging process.

[0071] Furthermore, considering the importance of the timeliness of remote sensing data in application scenarios such as weather forecasting and disaster relief, in order to further improve the extraction speed of remote sensing data, step S10 in this embodiment includes: diverting the received satellite remote sensing data through a message queue to obtain diverted remote sensing data; performing word segmentation processing on the diverted remote sensing data based on the Lucene library to obtain word segmentation data corresponding to the diverted remote sensing data; and indexing the satellite remote sensing data according to the word segmentation data to obtain inverted index data.

[0072] It should be noted that a message queue is a communication model for passing messages between applications, such as RabbitMQ, Apache Kafka, and ActiveMQ. Message queues are used to distribute satellite remote sensing data acquired by different satellites to different topics. This allows data processing programs to consume the same satellite data from different topics, reducing data complexity.

[0073] It is understood that the Lucene library is an open-source text search engine library used to build applications such as full-text search and information retrieval. The Lucene library is used to segment the diverted remote sensing data, obtaining segmented data for use in creating an inverted index. For example, when creating an index using the Lucene library, segmentation is required. Since binary data is inconvenient to segment, fixed-length segmentation can be used, which is not a limitation in this embodiment.

[0074] In a specific implementation, the extraction device can first split the data acquired by different satellites into different topics through a message queue, consume different topics, process the same satellite data, and reduce the data complexity. In order to improve the data extraction speed, it is necessary to further create an index for the remote sensing data. The index creation can be based on a binary format to exclude bit missing or bit shift of the data. By using the Lucene open source library, the modification and optimization can be facilitated. When Lucene creates an index, it needs to be segmented. Since the binary data is not convenient for segmentation, the satellite remote sensing data is segmented according to the segmented data, and the index is created to obtain the inverted index data.

[0075] After receiving the satellite remote sensing data, the extraction device of the embodiment creates an inverted index according to the attribute value of the satellite remote sensing data to obtain inverted index data, so as to improve the data extraction speed. Since the remote sensing data formed after the satellite performs remote sensing observation on the earth has the characteristics of unstructured, large data volume, and strong timeliness, during the transmission of the remote sensing data from the satellite to the ground receiving system, the transmission data may have error codes or missing codes due to the influence of the ionosphere and solar radiation. Therefore, when the satellite remote sensing data is processed according to the inverted index data, the frame data with error codes and missing data needs to be repaired and other preprocessing operations are performed, so as to obtain each frame of target remote sensing data. Finally, considering that the satellite remote sensing data is downloaded in blocks, in order to avoid the error of the spliced frame, the data of each frame of target remote sensing data needs to be merged to restore a complete task data, and available remote sensing data is obtained. Since the satellite remote sensing data received on the ground is segmented in binary format by using the lucene library to create inverted index data, in the subsequent data extraction process, the created index information is used to quickly query the data. The slow extraction speed of the traditional extraction according to the frame header and the sorting and splicing according to the frame count is avoided, the data extraction efficiency is improved, and the timeliness of the remote sensing data in the application scenarios such as weather prediction and disaster rescue is improved.

[0076] Reference Figure 3 , Figure 3 The flowchart of the second embodiment of the remote sensing data extraction method of the application is shown.

[0077] Based on the first embodiment, in the embodiment, considering the error code problem of the satellite remote sensing data, in order to improve the completeness and availability of the remote sensing data, the step S20 includes:

[0078] Step S21: querying the satellite remote sensing data according to the inverted index data to obtain a remote sensing frame header corresponding to the satellite remote sensing data.

[0079] It should be noted that the remote sensing frame header is the metadata information located at the beginning of each data frame in the satellite remote sensing data. It contains a description and identification of the entire data frame. The remote sensing frame header can be used to quickly parse and process the remote sensing data.

[0080] Step S22: performing similarity matching on the remote sensing frame header and the inverted index data using a similarity matching algorithm to obtain a similarity value between the remote sensing frame header and the inverted index data.

[0081] It is understood that similarity matching algorithms are algorithms used to compare the degree of similarity between two objects, such as Euclidean distance and cosine similarity. Using similarity matching algorithms to match satellite remote sensing data can reduce false positives and missed positives, thereby improving the accuracy of data extraction.

[0082] It should be noted that the similarity matching algorithm allows for a certain bit error rate. The frame header and frame synchronization word in satellite remote sensing data are key to frame data extraction. If there are errors in the frame header or frame synchronization word, the frame data extraction may fail. Based on the inverted index data of the Lucene library, a similarity matching algorithm for frame headers and frame synchronization words was created. This algorithm allows for some byte errors in the frame header or frame synchronization word and calculates the similarity between the errored frame header or synchronization word and the true frame header or synchronization word. If the similarity reaches a preset threshold, it is considered a possible frame header or synchronization word. This algorithm avoids all data matching, which not only improves data extraction efficiency but also reduces false detections and missed detections, thereby improving data extraction accuracy.

[0083] In a specific implementation, the extraction device can perform a frame query on the satellite remote sensing data based on the inverted index data to obtain a remote sensing frame header. The remote sensing frame header can then be used to quickly parse and process the parsed remote sensing data. A similarity matching algorithm is then used to perform similarity matching on the remote sensing frame header and the inverted index data to obtain a similarity value between the remote sensing frame header and the inverted index data. Using the similarity matching algorithm to perform data matching on the satellite remote sensing data reduces false detections and missed detections, thereby improving the accuracy of data extraction.

[0084] Step S23: performing data repair on the frame data in the satellite remote sensing data using the similarity value to obtain target remote sensing data of each frame.

[0085] It should be noted that when satellites transmit remote sensing data to ground receiving systems, they are affected by the ionosphere and solar radiation, which may cause errors or missing bits in the transmitted data. Therefore, when performing data processing on the satellite remote sensing data based on the inverted index data, it is necessary to perform preprocessing operations such as repairing the frames with errors and missing data to obtain the target remote sensing data for each frame.

[0086] Furthermore, considering the accuracy of similarity matching, step S22 in this embodiment includes: performing vector mapping on the remote sensing frame header and the inverted index data to obtain a frame header vector corresponding to the remote sensing frame header and an index vector corresponding to the inverted index data; performing vector angle processing on the frame header vector and the index vector respectively to obtain a frame vector angle between the frame header vectors and an index vector angle between the index vectors; performing similarity matching on the frame vector angle and the index vector angle through a similarity matching algorithm to obtain a similarity value between the remote sensing frame header and the inverted index data.

[0087] It should be noted that vector mapping is the process of mapping the remote sensing frame header and the inverted index data from one vector space to another. This mapping can be achieved through linear transformation, nonlinear transformation, or other mathematical functions. By performing vector mapping on the remote sensing frame header and the inverted index data, the corresponding frame header vector and index vector can be obtained.

[0088] In a specific implementation, the creation of an index allows for rapid querying of remote sensing data. However, after retrieving the remote sensing data, the frame data must be analyzed for errors and missing data, for example. Using the eigenvector method, the index and the data to be queried can be mapped into vectors. The similarity between the two data sets can be analyzed by calculating the frame vector angle between the frame header vectors and the index vector angle between the index vectors, thereby improving the accuracy of similarity matching.

[0089] Furthermore, considering the data repair process, step S23 in this embodiment includes: judging whether the similarity value is greater than a preset threshold value, and using the remote sensing frame header corresponding to the value greater than the preset threshold value as a frame header preselected point; judging whether the frame distance between adjacent frame header preselected points reaches a preset frame length; in the case that the frame distance does not reach the preset frame length, performing data correction on the remote sensing frame header corresponding to the remote sensing frame header that does not reach the preset frame length according to a preset data format to obtain corrected satellite remote sensing data; performing data extraction on the frame data in the corrected satellite remote sensing data to obtain target remote sensing data of each frame.

[0090] It should be noted that the preset threshold is a pre-set threshold for determining similarity. For example, the threshold can be set to 90% for complete similarity. It is important to note that the similarity matching algorithm allows for a certain bit error rate. This algorithm allows for some byte errors in the frame header or frame synchronization word. The algorithm calculates the similarity between the errored frame header or synchronization word and the true frame header or synchronization word. If the similarity reaches the preset threshold, it is considered likely to be a frame header or synchronization word, thus avoiding matching all data and improving the efficiency of the data extraction process.

[0091] It is understood that the frame header preselection point is the remote sensing frame header corresponding to a value greater than the preset threshold. Using this preselection point allows matching and tracking of frame data with bit errors or missing bits, thereby improving the efficiency and accuracy of matching and tracking. By using this preselection point, the computational complexity of matching and tracking can be reduced, thereby increasing processing speed.

[0092] It should be understood that the preset frame length is a pre-set frame length for processing. In satellite remote sensing data, the preset frame length can determine the amount of data for data correction operations.

[0093] In a specific implementation, if the similarity value is greater than a preset threshold, the remote sensing frame header at that location is considered a preselected frame header point. The next preselected point is then calculated. If the distance between the two preselected points is greater than or equal to the preset frame length, the preselected point is considered correct. Otherwise, there is a problem with the preselected point, and the number of problem frames is recorded. Finally, the remote sensing frame headers that do not meet the preset frame length are corrected according to the preset data format to obtain corrected satellite remote sensing data; and the frame data in the corrected satellite remote sensing data is extracted to obtain the target remote sensing data of each frame. Because the similarity matching algorithm is insensitive to frame errors and missing codes, data can still be extracted even when there are problems with the data. It has strong robustness against data misalignment, errors, and missing data, which improves the efficiency of data extraction.

[0094] In this embodiment, the extraction device can perform frame queries on the satellite remote sensing data based on the inverted index data to obtain remote sensing frame headers. The remote sensing frame headers can then be used to quickly parse and process the parsed remote sensing data. A similarity matching algorithm is then used to perform similarity matching on the remote sensing frame headers and the inverted index data to obtain a similarity value between the remote sensing frame headers and the inverted index data. This similarity matching algorithm is used to match the satellite remote sensing data, reducing false detections and missed detections, thereby improving data extraction accuracy. Furthermore, considering that when satellites transmit remote sensing data to ground receiving systems, data may be affected by the ionosphere and solar radiation, resulting in errors or missing bits. Therefore, when performing data analysis on the satellite remote sensing data based on the inverted index data, preprocessing operations such as repairing frames with errors or missing data are required to obtain the target remote sensing data for each frame. Furthermore, due to the creation of an index, querying remote sensing data is very fast. After extracting the remote sensing data, the frame data must be analyzed, for example to determine whether there are errors or missing data. Based on the eigenvector method, the index and the data to be queried can be mapped into vectors. The similarity between the two data is analyzed by calculating the frame vector angle between the frame header vectors and the index vector angle between the index vectors, thereby improving the accuracy of the similarity matching. Furthermore, if the similarity value is greater than a preset threshold, the remote sensing frame header at that location is considered a preselected frame header point. The next preselected point is then calculated. If the distance between the two preselected points is greater than or equal to the preset frame length, the preselected point is considered correct. Otherwise, there is a problem with the preselected point, and the number of problem frames is recorded. Finally, the remote sensing frame headers that do not meet the preset frame length are corrected according to a preset data format to obtain corrected satellite remote sensing data. The frame data within the corrected satellite remote sensing data is then extracted to obtain the target remote sensing data for each frame. Because the similarity matching algorithm is insensitive to frame errors and missing bits, data can still be extracted even when there are problems with the data, thereby improving data extraction efficiency.

[0095] refer to Figure 4 , Figure 4 2 is a flow chart of the third embodiment of the remote sensing data extraction method of the present invention.

[0096] Based on the above embodiments, in this embodiment, considering the remote sensing data splicing problem, since the remote sensing data is downloaded in blocks, different blocks are restored to a complete task data, and frame errors may exist at the splicing location. Therefore, step S30 includes:

[0097] Step S31: extracting the mission time of the satellite remote sensing data.

[0098] Step S32: sorting the target remote sensing data of each frame according to the mission time to obtain sorted data.

[0099] Step S33: positioning each frame data in the sorted data to obtain the byte end position of the frame data in the byte.

[0100] It should be noted that the mission time is the time required for the entire process of satellite remote sensing data from the start of observation by the satellite sensor to data generation and transmission.

[0101] It is understood that the byte end position is the disconnection position of each frame data in the sorted data. It should be noted that the byte end position of the frame data in the byte depends on the specific format and encoding method of the frame. Different file formats and protocols define different frame structures, so the byte end position will vary.

[0102] Step S34: merging the sorted data according to the byte end position to obtain usable remote sensing data.

[0103] In a specific implementation, the different satellite data within each frame of target remote sensing data can be decoupled and processed separately through a message queue. Each satellite data is queued and processed through a separate topic, reducing the complexity caused by the large number of data types. During the data merging process, a data merging algorithm can be used to record the specific stop and start positions of each frame of data in the current frame. The sorted data can then be merged based on the byte end positions to obtain usable remote sensing data, ensuring byte integrity during the merging process.

[0104] Furthermore, considering the data merging process, step S34 in this embodiment includes: performing offset measurement on each frame data according to the byte end position to obtain the data offset corresponding to each frame data; when the data offset reaches the preset offset, performing frame interception on each frame data corresponding to the preset offset to obtain the intercepted frame data; performing data merging on the sorted data and the intercepted frame data according to the byte end position to obtain usable remote sensing data.

[0105] It should be noted that the data offset corresponding to the frame data is the offset value of the starting position of the frame data relative to the starting offset in the entire data stream. The data offset can be used to locate and extract data of a specific frame.

[0106] In a specific implementation, the satellite remote sensing data can be divided by the shooting task time of the same satellite. Since the same satellite only performs the same task at the same time, there is an interval between tasks, so the task time can be used to split the task and combine the data belonging to the same task. In the task merging process, first, the different block data can be sorted according to the time; then, the position of the last data and the next byte end is calculated during data merging, and the byte data of adjacent data blocks needs to be merged according to the offset. If the frame data length is greater than the regular frame length, it needs to be cut off; if the length is less than the regular frame length, the data is problematic and needs to be recorded. Finally, the data merging is completed to form the available remote sensing data. By reconstructing the time sequence between tasks and merging them, the integrity of the remote sensing data of different tasks is ensured.

[0107] For ease of understanding, the remote sensing data extraction process of each embodiment is described above, but the present scheme is not limited thereto. Referring to Figure 5 , Figure 5 is the overall workflow in the third embodiment of the remote sensing data extraction method of the present application. First, create an index for satellite remote sensing data based on Lucene to improve data extraction speed; then establish a similarity matching algorithm to repair data and reduce the sensitivity of errors and missing data; then merge the split tasks. Finally, analyze the data quality to provide priority setting reference for the next data processing. The specific process is as follows:

[0108] First, the data (such as figure topic-1) obtained by different satellites (such as satellite 1, satellite 2, satellite 3) is shunted to different topics through the message queue. The extraction device consumes different topic data, processes the same satellite data, and reduces the data complexity.

[0109] Then create an inverted index through Lucene. In order to improve the speed of data extraction and matching query, it is necessary to create an index for remote sensing data. The creation format of the index is binary index format to exclude bit missing or bit shift of the data. The index creation can use Lucene open source library, which is convenient for modification and optimization. When creating an index for the frame data of remote sensing data in the Lucene library, it can be segmented. Since binary data is not convenient for segmentation, fixed length segmentation can be used.

[0110] Next, the remote sensing data is extracted and repaired. Remote sensing frame headers are queried based on the index. Since the data is indexed, the query process is very fast. After extracting the remote sensing data, the frame data is analyzed for bit errors and missing data. A similarity matching algorithm is configured to calculate the similarity between the index data and the frame header. Based on the eigenvector method, the similarity matching algorithm maps the index and the data to be queried into vectors and analyzes the similarity between the two data by calculating the angle between the vectors. If the similarity exceeds a preset threshold, the location is considered a preselected frame header point. The next preselected point is then calculated. If the distance between the two preselected points is greater than or equal to the frame length, the preselected point is considered correct. Otherwise, the preselected point is faulty, and the number of faulty frames is recorded. The similarity query algorithm is insensitive to frame errors and missing bits, so even if the data is faulty, data can still be extracted. Incorrect frame headers or frame synchronization characters are corrected according to the data's specified format.

[0111] Finally, the data is merged to form usable data. Remote sensing data tasks are divided based on the same satellite's acquisition time. The same satellite performs only the same task at the same time, with intervals between tasks. Therefore, tasks can be split based on task time, allowing data belonging to the same task to be merged. During task merging, different blocks of data are first sorted by time. Then, when merging data, the byte positions of the previous and next data blocks must be calculated, and the bytes of adjacent data blocks must be merged based on the offset. If the frame length exceeds the standard frame length, it is truncated. If the length is less than the standard frame length, the data is considered problematic and recorded.

[0112] Furthermore, considering the varying qualities of remote sensing data, the lack of prioritization during processing can render some low-quality data unusable after processing, wasting computing resources. Therefore, data quality analysis can be performed during the data repair process. By counting frame errors, the quality of the currently received remote sensing data can be analyzed, providing a reference for prioritizing subsequent remote sensing data processing operations. This prioritizes processing high-quality data, improving resource utilization.

[0113] This embodiment divides satellite remote sensing data by the same satellite's mission time. Since the same satellite performs only the same mission at the same time, there are gaps between missions. Therefore, mission time can be used to split missions and merge data belonging to the same mission. During the mission merging process, different blocks of data are first sorted by time. Then, during data merging, the byte positions of the previous and next data blocks are calculated, and the byte data of adjacent data blocks are merged based on the offset. If the frame length exceeds the normal frame length, it is truncated; if it is less than the normal frame length, the data is problematic and needs to be recorded. Finally, the data is merged to form usable remote sensing data. By reconstructing the time sequence between missions and merging them, the integrity of remote sensing data from different missions is ensured. Furthermore, data quality analysis can be performed. During the data repair process, the quality of the currently received remote sensing data is analyzed by counting the number of frame errors, providing a reference for prioritizing other remote sensing data processing operations. Data with higher quality is prioritized, improving resource utilization.

[0114] In addition, an embodiment of the present invention further provides a storage medium, on which a remote sensing data extraction program is stored. When the remote sensing data extraction program is executed by a processor, the steps of the remote sensing data extraction method described above are implemented.

[0115] Reference Figure 6 , Figure 6 This is a structural block diagram of the first embodiment of the remote sensing data extraction device of the present invention.

[0116] like Figure 6 As shown, the remote sensing data extraction device proposed in the embodiment of the present invention includes:

[0117] The inverted index module 601 is used to create an index for the received satellite remote sensing data to obtain inverted index data;

[0118] A data extraction module 602 is configured to perform data preprocessing on the satellite remote sensing data according to the inverted index data to obtain target remote sensing data of each frame;

[0119] The data merging module 603 is used to merge the target remote sensing data of each frame to obtain usable remote sensing data.

[0120] The satellite remote sensing data extraction device in the embodiment creates an inverted index according to attribute values of satellite remote sensing data after receiving the satellite remote sensing data, obtains inverted index data, and improves the speed of data extraction. Since the remote sensing data formed after the satellite performs remote sensing observation on the earth has the characteristics of unstructured, large data volume, and strong timeliness, and is affected by the ionosphere and solar radiation during transmission of the remote sensing data from the satellite to the ground receiving system, the transmission data may have error codes or missing codes. Therefore, when the satellite remote sensing data is processed according to the inverted index data, frame data with error codes and missing data needs to be repaired and other preprocessing operations are performed, so as to obtain each frame of target remote sensing data. Finally, considering that the satellite remote sensing data is downloaded in blocks, to avoid frame errors at the splicing position, the data of the target remote sensing data needs to be merged to recover a complete task data and obtain available remote sensing data. Since the satellite remote sensing data received on the ground is segmented in binary format by using the lucene library to create inverted index data. During the subsequent data extraction process, the created index information is used to quickly query the data. The slow extraction speed of the traditional extraction according to the frame header and sorting and splicing according to the frame count is avoided, the efficiency of data extraction is improved, and the timeliness of remote sensing data in application scenarios such as weather prediction and disaster rescue is improved.

[0121] Based on the first embodiment of the remote sensing data extraction device, a second embodiment of the remote sensing data extraction device is provided.

[0122] In the embodiment, the inverted index module 601 is further configured to split the received satellite remote sensing data by using a message queue to obtain split remote sensing data, perform segmentation processing on the split remote sensing data based on a Lucene library to obtain segmentation data corresponding to the split remote sensing data, and create an index based on the segmentation data to obtain inverted index data.

[0123] Further, the data extraction module 602 is further configured to perform frame query on the satellite remote sensing data based on the inverted index data to obtain a remote sensing frame header corresponding to the satellite remote sensing data, perform similarity matching on the remote sensing frame header and the inverted index data by using a similarity matching algorithm to obtain a similarity value between the remote sensing frame header and the inverted index data, and perform data repair on frame data in the satellite remote sensing data by using the similarity value to obtain each frame of target remote sensing data.

[0124] Furthermore, the data extraction module 602 is also used to perform vector mapping on the remote sensing frame header and the inverted index data to obtain a frame header vector corresponding to the remote sensing frame header and an index vector corresponding to the inverted index data; perform vector angle processing on the frame header vector and the index vector respectively to obtain a frame vector angle between the frame header vectors and an index vector angle between the index vectors; perform similarity matching on the frame vector angle and the index vector angle through a similarity matching algorithm to obtain a similarity value between the remote sensing frame header and the inverted index data.

[0125] Furthermore, the data extraction module 602 is also used to determine whether the similarity value is greater than a preset threshold value, and use the remote sensing frame header corresponding to the value greater than the preset threshold value as a frame header pre-selected point; determine whether the frame distance between adjacent frame header pre-selected points reaches a preset frame length; if the frame distance does not reach the preset frame length, perform data correction on the remote sensing frame header corresponding to the remote sensing frame header that does not reach the preset frame length according to a preset data format to obtain corrected satellite remote sensing data; perform data extraction on the frame data in the corrected satellite remote sensing data to obtain target remote sensing data of each frame.

[0126] Furthermore, the data merging module 603 is also used to extract the mission time of the satellite remote sensing data; sort the target remote sensing data of each frame according to the mission time to obtain sorted data; locate each frame data in the sorted data to obtain the byte end position of the frame data in the byte; merge the sorted data according to the byte end position to obtain usable remote sensing data.

[0127] Furthermore, the data merging module 603 is also used to perform offset measurement on each frame data according to the byte end position to obtain the data offset corresponding to each frame data; when the data offset reaches the preset offset, perform frame interception on each frame data corresponding to the preset offset to obtain the intercepted frame data; and perform data merging on the sorted data and the intercepted frame data according to the byte end position to obtain usable remote sensing data.

[0128] Other embodiments or specific implementations of the remote sensing data extraction device of the present invention can refer to the above-mentioned method embodiments and will not be described in detail here.

[0129] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0130] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0131] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0132] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A remote sensing data extraction method, characterized in that: The remote sensing data extraction method comprises: Create an index for the received satellite remote sensing data to obtain inverted index data; Performing a frame query on the satellite remote sensing data according to the inverted index data to obtain a remote sensing frame header corresponding to the satellite remote sensing data; Performing similarity matching on the remote sensing frame header and the inverted index data using a similarity matching algorithm to obtain a similarity value between the remote sensing frame header and the inverted index data; Performing data repair on the frame data in the satellite remote sensing data using the similarity value to obtain target remote sensing data of each frame; Merging the target remote sensing data of each frame to obtain usable remote sensing data.

2. The remote sensing data extraction method according to claim 1, wherein: The step of indexing the received satellite remote sensing data to obtain inverted index data includes: Diverting the received satellite remote sensing data through the message queue to obtain diverted remote sensing data; Perform word segmentation processing on the diverted remote sensing data based on the Lucene library to obtain word segmentation data corresponding to the diverted remote sensing data; The satellite remote sensing data is indexed and created according to the word segmentation data to obtain inverted index data.

3. The remote sensing data extraction method according to claim 1, wherein: The performing similarity matching on the remote sensing frame header and the inverted index data by using a similarity matching algorithm to obtain a similarity value between the remote sensing frame header and the inverted index data includes: Performing vector mapping on the remote sensing frame header and the inverted index data to obtain a frame header vector corresponding to the remote sensing frame header and an index vector corresponding to the inverted index data; Performing vector angle processing on the frame header vector and the index vector respectively to obtain a frame vector angle between the frame header vectors and an index vector angle between the index vectors; The frame vector angle and the index vector angle are similarly matched by a similarity matching algorithm to obtain a similarity value between the remote sensing frame header and the inverted index data.

4. The remote sensing data extraction method according to claim 3, wherein: The step of performing data repair on the frame data in the satellite remote sensing data by using the similarity value to obtain target remote sensing data of each frame includes: Determine whether the similarity value is greater than a preset threshold, and use the remote sensing frame header corresponding to the value greater than the preset threshold as a frame header preselected point; Determine whether the frame distance between adjacent frame header preselected points reaches the preset frame length; In the case where the frame distance does not reach the preset frame length, performing data correction on the remote sensing frame header corresponding to the remote sensing frame length that does not reach the preset frame length according to a preset data format to obtain corrected satellite remote sensing data; Data extraction is performed on the frame data in the corrected satellite remote sensing data to obtain target remote sensing data of each frame.

5. The remote sensing data extraction method according to claim 1, wherein: The step of merging the target remote sensing data of each frame to obtain usable remote sensing data includes: The mission time of extracting the satellite remote sensing data; Sorting the target remote sensing data of each frame according to the mission time to obtain sorted data; Positioning each frame of data in the sorted data to obtain a byte end position of the frame of data in bytes; The sorted data are merged according to the end position of the bytes to obtain usable remote sensing data.

6. The remote sensing data extraction method according to claim 5, wherein: The step of merging the sorted data according to the byte end position to obtain usable remote sensing data includes: Performing offset measurement on each frame of data according to the byte end position to obtain a data offset corresponding to each frame of data; When the data offset reaches a preset offset, frame interception is performed on each frame data corresponding to the preset offset to obtain intercepted frame data; The sorted data and the intercepted frame data are merged according to the byte end position to obtain usable remote sensing data.

7. A remote sensing data extraction device, characterized in that: The device comprises: The inverted index module is used to create an index for the received satellite remote sensing data and obtain inverted index data; a data extraction module configured to perform frame query on the satellite remote sensing data according to the inverted index data to obtain a remote sensing frame header corresponding to the satellite remote sensing data; perform similarity matching on the remote sensing frame header and the inverted index data using a similarity matching algorithm to obtain a similarity value between the remote sensing frame header and the inverted index data; and perform data repair on the frame data in the satellite remote sensing data using the similarity value to obtain target remote sensing data of each frame; The data merging module is used to merge the target remote sensing data of each frame to obtain usable remote sensing data.

8. A remote sensing data extraction device, characterized in that: The device includes: a memory, a processor, and a remote sensing data extraction program stored in the memory and executable on the processor, wherein the remote sensing data extraction program is configured to implement the steps of the remote sensing data extraction method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium stores a remote sensing data extraction program, which, when executed by a processor, implements the steps of the remote sensing data extraction method according to any one of claims 1 to 6.

Images