Satellite big data distributed storage method and system

Abstract

The invention relates to the technical field of data storage, and particularly discloses a satellite big data distributed storage method and system.The method comprises the steps that a reference image set is obtained according to satellite images containing text description and shooting parameters; carrying out statistics on text description of each reference image in the reference image set, carrying out subject analysis and word analysis on the text description obtained through statistics to obtain subject distribution of each text description and word distribution of each subject, and determining keywords of each reference image according to the subject distribution and the word distribution; creating a distributed storage library by taking each reference image as a benchmark; and storing the new satellite image based on the distributed storage library. When the satellite image is received, the text description and the shooting parameters are synchronously acquired, the existing satellite image and the text description and the shooting parameters thereof are compared, the reference image is selected, the to-be-classified image is classified and stored according to the reference image, a distributed storage architecture is constructed, and the storage process is very clear.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a satellite big data distributed storage method and system. Background Art

[0002] "Satellite big data" refers to a large amount of data collected, processed, analyzed and used through satellite remote sensing technology. Satellite big data contains many types of information, such as images of the earth's surface, climate change, environmental monitoring, disaster warning, agricultural production, etc. Remote sensing data obtained by satellites, after complex data processing and analysis, can provide decision support, predictive analysis and real-time monitoring services for relevant management parties.

[0003] The existing storage process of satellite images is mostly a centralized storage architecture, and the data is not very "clean". During the processing and analysis stage, a screening process needs to be added, which affects the processing and analysis efficiency. How to improve the storage clarity of satellite big data is the technical problem that the technical solution of the present invention wants to solve. Summary of the invention

[0004] The object of the present invention is to provide a satellite big data distributed storage method and system to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A satellite big data distributed storage method, the method comprising:

[0007] receiving satellite images containing text descriptions and shooting parameters uploaded by the management party, and when the number of uploaded images reaches a preset first number threshold, comparing each satellite image to determine the selection probability of each satellite image;

[0008] Selecting a preset second quantity threshold of satellite images based on the selection probability to obtain a reference image set;

[0009] Counting the text descriptions of each reference image in the reference image set, performing topic analysis and word analysis on the text descriptions obtained by statistics, obtaining the topic distribution of each text description and the word distribution of each topic, and determining the keywords of each reference image according to the topic distribution and word distribution;

[0010] Using each reference image as a benchmark, a distributed repository is created;

[0011] A satellite image to be classified containing text description and shooting parameters is received, the text description of the image to be classified is matched with keywords of each reference image, and a distributed storage repository is selected for storage.

[0012] As a further solution of the present invention: the receiving management party uploads satellite images containing text descriptions and shooting parameters, and when the number of uploads reaches a preset first number threshold, the step of comparing each satellite image to determine the selection probability of each satellite image includes:

[0013] Receive satellite images uploaded by the management party containing text descriptions and shooting parameters, and record the number of uploads;

[0014] When the upload quantity reaches a preset first quantity threshold, extracting feature values ​​of the satellite image based on a preset feature value extraction model;

[0015] Compare any satellite image with all other satellite images to determine the probability of selection of each satellite image;

[0016] The comparison process is:

[0017] Where P i is the selection probability of the i-th satellite image; Z i is the value corresponding to the i-th satellite image, Z is the total value of the values ​​corresponding to all satellite images; N is the number of uploads; α1, α2 and α3 are the preset correction coefficients, Δ j1 , Δ j2 and Δ j3 They represent the text description difference, shooting parameter difference and feature value difference between the i-th satellite image and the j-th satellite image respectively.

[0018] As a further solution of the present invention: the step of performing topic analysis and word analysis on the statistically obtained text descriptions to obtain the topic distribution of each text description and the word distribution of each topic includes:

[0019] The number of topics entered by the receiving administrator;

[0020] For each text description, extract a topic distribution from a Richter distribution based on the number of topics;

[0021] For each topic, draw word distribution from the Richter distribution;

[0022] For each text description, iteratively updating the topic distribution and the word distribution based on Gibbs sampling;

[0023] After the iterative update is completed, the final topic distribution and final word distribution for each text description are output.

[0024] As a further solution of the present invention: the iterative update process is:

[0025] Generate document content based on the extracted topic distribution and word distribution of each topic;

[0026] For each word w in the document content dn , calculate the conditional probability that the current word belongs to each topic, and reallocate the topic of the current word according to the calculated conditional probability;

[0027] The loop is executed until the preset number of loops is reached;

[0028] The document content generation process is as follows:

[0029] From the topic distribution θ d Extract a topic z from dn , from the selected topic z dn Word distribution Extract a word w from

[0030] The calculation process of conditional probability is:

[0031]

[0032] In the formula, P(z dn = k | z -dn , w, α, β) represents the conditional probability, z dn represents the topic assignment of the nth word in document d, z -dn represents the distribution of all topics except the nth word in document d, w represents the actual vocabulary of all words; α and β are preset values, both of which are 0.1;

[0033] n dk,-dn is the number of words in document d that belong to topic k, excluding the current word; n d,-dn is the total number of words in document d, excluding the current word, n kw,-dn is the number of occurrences of word w in topic k, excluding the current word, n k,-dn is the total number of words in topic k, excluding the current word; K is the dimension of each element in the topic distribution; V is the dimension of each element in the word distribution.

[0034] As a further solution of the present invention: determining the content of the keywords of each reference image according to the topic distribution and the word distribution includes:

[0035] Get the final topic distribution of each text description and select the text topic;

[0036] Get the word distribution corresponding to the text topic and select the word as the keyword;

[0037] Among them, the selection rules of text topics and the selection rules of words are both that the probability is greater than a preset value.

[0038] As a further solution of the present invention: the method further comprises:

[0039] When storing an image, each distributed repository determines whether to perform detection based on the retrieval probability;

[0040] When no detection is performed, the storage process is executed to extract the feature value of the image based on the same feature extraction model, and the mean of the feature value of all images in the distributed storage library is updated;

[0041] When performing detection, the feature value of the image is extracted based on the same feature extraction model, and the difference between the feature value and the mean feature value of all images in the distributed repository is calculated. When the difference reaches a preset difference threshold, a warning message is generated.

[0042] The technical solution of the present invention also provides a satellite big data distributed storage system, the system comprising:

[0043] An image receiving module is used to receive satellite images containing text descriptions and shooting parameters uploaded by the management party, and when the number of uploaded images reaches a preset first number threshold, compare each satellite image to determine the selection probability of each satellite image;

[0044] An image set construction module, configured to select satellite images of a preset second quantity threshold based on the selection probability to obtain a reference image set;

[0045] A text analysis module is used to count the text descriptions of each reference image in the reference image set, perform topic analysis and word analysis on the text descriptions obtained by statistics, obtain the topic distribution of each text description and the word distribution of each topic, and determine the keywords of each reference image according to the topic distribution and word distribution;

[0046] A repository creation module, used to create a distributed repository based on each reference image;

[0047] The matching storage module is used to receive the satellite image to be classified containing text description and shooting parameters, match the text description of the image to be classified with the keywords of each reference image, and select a distributed storage repository for storage.

[0048] As a further solution of the present invention: the image receiving module includes:

[0049] An upload management unit, used to receive satellite images containing text descriptions and shooting parameters uploaded by the management party, and record the number of uploads;

[0050] A feature extraction unit, configured to extract feature values ​​of the satellite image based on a preset feature value extraction model when the upload quantity reaches a preset first quantity threshold;

[0051] An image comparison unit, used to compare any satellite image with all other satellite images to determine the selection probability of each satellite image;

[0052] The comparison process is:

[0053] Where P i is the selection probability of the i-th satellite image; Z i is the value corresponding to the i-th satellite image, Z is the total value of the values ​​corresponding to all satellite images; N is the number of uploads; α1, α2 and α3 are the preset correction coefficients, Δ j1 , Δ j2 and Δ j3 They represent the text description difference, shooting parameter difference and feature value difference between the i-th satellite image and the j-th satellite image respectively.

[0054] As a further solution of the present invention: the text analysis module includes:

[0055] A quantity receiving unit, used to receive the quantity of topics input by the management party;

[0056] a topic identification unit for extracting, for each text description, a topic distribution from a Richter distribution based on the number of topics;

[0057] A word recognition unit, for extracting a word distribution from the Lickley distribution for each topic;

[0058] A distribution updating unit, configured to iteratively update the topic distribution and the word distribution for each text description based on Gibbs sampling;

[0059] The iteration unit is used to output the final topic distribution and final word distribution of each text description after the iterative update is completed.

[0060] As a further solution of the present invention: the iterative update process is:

[0061] Generate document content based on the extracted topic distribution and word distribution of each topic;

[0062] For each word w in the document content dn , calculate the conditional probability that the current word belongs to each topic, and reallocate the topic of the current word according to the calculated conditional probability;

[0063] The loop is executed until the preset number of loops is reached;

[0064] The document content generation process is as follows:

[0065] From the topic distribution θ d Extract a topic z from dn , from the selected topic z dn Word distribution Extract a word w from

[0066] The calculation process of conditional probability is:

[0067]

[0068] In the formula, P(z dn = k | z -dn , w, α, β) represents the conditional probability, z dn represents the topic assignment of the nth word in document d, z -dn represents the distribution of all topics except the nth word in document d, w represents the actual vocabulary of all words; α and β are preset values, both of which are 0.1;

[0069] n dk,-dn is the number of words in document d that belong to topic k, excluding the current word; n d,-dn is the total number of words in document d, excluding the current word, n kw,-dn is the number of occurrences of word w in topic k, excluding the current word, n k,-dn is the total number of words in topic k, excluding the current word; K is the dimension of each element in the topic distribution; V is the dimension of each element in the word distribution.

[0070] Compared with the prior art, the beneficial effects of the present invention are: when receiving satellite images, the present invention synchronously obtains text descriptions and shooting parameters, compares existing satellite images and their text descriptions and shooting parameters, selects reference images, and classifies and stores images to be classified according to the reference images, thereby constructing a distributed storage architecture, and the storage process is very clear. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention.

[0072] Figure 1 This is a flowchart of the satellite big data distributed storage method.

[0073] Figure 2 This is a flowchart of the first sub-process of the satellite big data distributed storage method.

[0074] Figure 3 This is a second sub-process flowchart of the satellite big data distributed storage method.

[0075] Figure 4 This is a structural block diagram of the satellite big data distributed storage system. DETAILED DESCRIPTION

[0076] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0077] Figure 1 : is a flowchart of a satellite big data distributed storage method. In an embodiment of the present invention, a satellite big data distributed storage method includes:

[0078] Step S100: receiving satellite images containing text descriptions and shooting parameters uploaded by the management party, and when the number of uploaded images reaches a preset first number threshold, comparing each satellite image to determine the selection probability of each satellite image;

[0079] Step S100 is the satellite image receiving stage, in which the management party selects and uploads some satellite images containing text descriptions and shooting parameters. The text description is the description of the satellite image, and the shooting parameters are the shooting coordinates and clarity of the satellite image. During the work process, the management party will receive the satellite images uploaded by the satellite image shooter, count and select them, and upload them to the execution platform of this method.

[0080] It should be noted that the screening process can be manual screening or intelligent screening. The intelligent screening process only screens the data completeness, such as whether there are missing parts in the text description and shooting parameters, or whether there are missing parts in the satellite image. The screening plan can adopt a conventional comparison plan.

[0081] When receiving satellite images containing text descriptions and shooting parameters, the received satellite images are compared to determine the selection probability of each satellite image.

[0082] Step S200: selecting satellite images of a preset second quantity threshold based on the selection probability to obtain a reference image set;

[0083] Step S200 is a further screening limitation on step S100. The received satellite images are continuously screened based on the selection probability, and then a small database is constructed, which is called a reference image set. In layman's terms, step S100 continuously receives uploaded satellite images and stores them in a large database. In the large database, the selection probability of each satellite image is continuously calculated, and then some satellite images are randomly selected based on the selection probability to establish a small database, which is the reference image set.

[0084] Step S300: Counting the text descriptions of each reference image in the reference image set, performing topic analysis and word analysis on the text descriptions obtained by counting, obtaining the topic distribution of each text description and the word distribution of each topic, and determining the keywords of each reference image according to the topic distribution and word distribution;

[0085] The text description of each reference image in the reference image set is read to create a text library. The text library is subjected to topic analysis to obtain a topic distribution, that is, the distribution of each topic in the text library. A text may include multiple topics. For each topic, a word distribution can be obtained. This process is a specific application of the LDA algorithm, which is a topic model algorithm for text analysis. After the topic distribution and word distribution are determined, the keywords of each text description, that is, the keywords of each reference image (reference images and text descriptions correspond one to one) can be easily determined.

[0086] Step S400: creating a distributed storage repository based on each reference image;

[0087] Taking each reference image as a benchmark, a separate repository is created to store satellite images similar to the reference image.

[0088] Step S500: receiving a satellite image to be classified containing a text description and shooting parameters, matching the text description of the image to be classified with keywords of each reference image, and selecting a distributed repository for storage;

[0089] Step S500 is the application stage. When a satellite image to be classified is received, it is compared with an existing reference image and the matching degree is calculated. When the matching degree is high enough, the corresponding repository is located and the satellite image is stored in the corresponding repository, thereby realizing the distributed storage function.

[0090] It should be noted that the satellite images of this application come with text descriptions and shooting parameters. If the text descriptions and shooting parameters are missing, they are considered to have no storage value and will not be stored. The solution provided by this application is essentially to store satellite image big data. The text descriptions and shooting parameters are used to ensure that they have storage value. For example, in the field of agricultural analysis, the text description is the growth assessment report of crops in the satellite image. Only satellite images with growth assessment reports have storage value and are regarded as processing objects of this platform.

[0091] Figure 2 The first sub-flow chart of the satellite big data distributed storage method comprises the following steps: receiving satellite images uploaded by the management party containing text descriptions and shooting parameters, and when the number of uploaded images reaches a preset first number threshold, comparing each satellite image to determine the selection probability of each satellite image:

[0092] Step S101: receiving satellite images containing text descriptions and shooting parameters uploaded by the management party, and recording the number of uploads;

[0093] Step S102: when the number of uploads reaches a preset first number threshold, extracting feature values ​​of the satellite image based on a preset feature value extraction model;

[0094] Step S103: Compare any satellite image with all other satellite images to determine the selection probability of each satellite image.

[0095] In an example of the technical solution of the present invention, satellite images containing text descriptions and shooting parameters uploaded by a receiving management party are received, and the number of uploads is recorded in real time. When the number of uploads is large enough, feature values ​​of the satellite images are extracted based on a preset feature value extraction model, and each satellite image is compared based on the feature values, text descriptions and shooting parameters. The comparison results of any satellite image with all other satellite images are read, and the selection probability of each satellite image is determined.

[0096] The comparison process is:

[0097] Where P i is the selection probability of the i-th satellite image; Z i is the value corresponding to the i-th satellite image, Z is the total value of the values ​​corresponding to all satellite images; N is the number of uploads; α1, α2 and α3 are the preset correction coefficients, Δ j1 , Δ j2 and Δ j3 They represent the text description difference, shooting parameter difference and feature value difference between the i-th satellite image and the j-th satellite image respectively.

[0098] The calculation schemes for the text description differences, shooting parameter differences and eigenvalue differences in the above content can adopt the existing schemes. The difference in text descriptions generally adopts conventional text similarity, and the simplest way is to compare the number of words. The difference in shooting parameters generally adopts spatial distance. The shooting parameters include coordinates and clarity. The clarity is generally the same, and the spatial distance can be calculated by the coordinates; the difference in eigenvalues ​​is simpler, and the numerical difference can be calculated.

[0099] Figure 3 This is a second sub-flow diagram of the satellite big data distributed storage method, wherein the steps of performing topic analysis and word analysis on the statistically obtained text descriptions to obtain the topic distribution of each text description and the word distribution of each topic include:

[0100] Step S301: receiving the number of topics input by the management party;

[0101] Step S302: for each text description, extracting a topic distribution from the Richter distribution based on the number of topics;

[0102] Step S303: for each topic, extract word distribution from the Richter distribution;

[0103] Step S304: for each text description, iteratively updating the topic distribution and the word distribution based on Gibbs sampling;

[0104] Step S305: After the iterative update is completed, the final topic distribution and final word distribution for each text description are output.

[0105] The number of topics is pre-entered by the administrator, for example, ten; for each document in the text library, a topic distribution θ is extracted from the Dirichlet distribution d ,θ d ~Dir(α),θ d is a K-dimensional vector, where K is the number of topics; for each topic k, a word distribution is extracted from the Dirichlet distribution Dir(β) It is a V-dimensional vector, which indicates the probability of a word under the topic.

[0106] The topic distribution and word distribution obtained by the above extraction are a baseline model. For each text, the topic distribution and the word distribution are iteratively updated based on Gibbs sampling, and the baseline model is continuously updated to obtain the final topic distribution in each document and the word distribution under each topic.

[0107] It should be noted that Dirichlet Distribution is a continuous multivariate probability distribution and a commonly used prior distribution in Bayesian statistics. It is particularly suitable for modeling multinomial distribution parameters. The main feature of Dirichlet Distribution is that it can describe the probability distribution of multiple random variables, and the sum of these random variables is 1, which makes it particularly suitable for modeling scenarios such as probability vectors and weight vectors.

[0108] Specifically, the iterative update process is:

[0109] Generate document content based on the extracted topic distribution and word distribution of each topic;

[0110] For each word w in the document content dn , calculate the conditional probability that the current word belongs to each topic, and reallocate the topic of the current word according to the calculated conditional probability;

[0111] The loop is executed until the preset number of loops is reached;

[0112] The document content generation process is as follows:

[0113] From the topic distribution θ d Extract a topic z from dn , from the selected topic z dn Word distribution Extract a word w from

[0114] The calculation process of conditional probability is:

[0115]

[0116] In the formula, P(z dn = k | z -dn , w, α, β) represents the conditional probability, z dn represents the topic assignment of the nth word in document d, z -dn represents the distribution of all topics except the nth word in document d, w represents the actual vocabulary of all words; α and β are preset values, both of which are 0.1;

[0117] n dk,-dn is the number of words in document d that belong to topic k, excluding the current word; n d,-dn is the total number of words in document d, excluding the current word, n kw,-dn is the number of occurrences of word w in topic k, excluding the current word, n k,-dn is the total number of words in topic k, excluding the current word; K is the dimension of each element in the topic distribution; V is the dimension of each element in the word distribution.

[0118] The above content is a specific application of the Gibbs sampling process, which approximates the true topic distribution of the document and the true word distribution of each topic by iteratively updating the topic assignment of each word.

[0119] As a preferred embodiment of the technical solution of the present invention, determining the content of the keywords of each reference image according to the topic distribution and the word distribution includes:

[0120] Get the final topic distribution of each text description and select the text topic;

[0121] Get the word distribution corresponding to the text topic and select the word as the keyword;

[0122] Among them, the selection rules of text topics and the selection rules of words are both that the probability is greater than a preset value.

[0123] When the final topic distribution and word distribution of each text are determined, words with high probability are selected from these distributions as keywords. The selection process first sets a probability threshold. When the probability of the text topic reaches the probability threshold, selection is performed. When the selection probability of a word reaches the probability threshold, selection is performed.

[0124] As a preferred embodiment of the technical solution of the present invention, the method further includes:

[0125] When storing an image, each distributed repository determines whether to perform detection based on the retrieval probability;

[0126] When no detection is performed, the storage process is executed to extract the feature value of the image based on the same feature extraction model, and the mean of the feature value of all images in the distributed storage library is updated;

[0127] When performing detection, the feature value of the image is extracted based on the same feature extraction model, and the difference between the feature value and the mean feature value of all images in the distributed repository is calculated. When the difference reaches a preset difference threshold, a warning message is generated.

[0128] In an example of the technical solution of the present invention, a detection process in the repository entry stage is introduced. First, a retrieval probability is set, such as 10%. When a distributed repository stores an image, it determines whether to perform detection based on the retrieval probability, that is, there is a 10% probability that detection is required. When the detection is not triggered, normal warehousing is performed, and the feature values ​​of the image are extracted based on the same feature extraction model, and the mean feature values ​​of all images in the distributed repository are updated; when the detection is triggered, the feature values ​​of the image are extracted based on the same feature extraction model, and the difference between the feature values ​​and the mean feature values ​​of all images in the distributed repository is calculated. When the difference reaches the preset difference threshold, it indicates that an error has occurred in the matching process, and a warning message is generated at this time.

[0129] It should be noted that the specific model of the feature extraction model involved in the present invention is not limited, but the applied feature extraction model needs to be statistical, that is, the present invention involves and only involves a feature extraction model. The input of the feature extraction model is an image, and the output is a numerical value. The existing feature extraction model can be used. The simplest way is to convert the image into grayscale and then calculate the grayscale eigenvalues ​​(mean and standard deviation, etc.).

[0130] Figure 4 1 is a structural block diagram of a satellite big data distributed storage system. In an embodiment of the present invention, a satellite big data distributed storage system, the system 10 includes:

[0131] The image receiving module 11 is used to receive satellite images containing text descriptions and shooting parameters uploaded by the management party, and when the number of uploaded images reaches a preset first number threshold, compare each satellite image to determine the selection probability of each satellite image;

[0132] An image set construction module 12 is used to select satellite images of a preset second quantity threshold based on the selection probability to obtain a reference image set;

[0133] A text analysis module 13 is used to count the text descriptions of each reference image in the reference image set, perform topic analysis and word analysis on the text descriptions obtained by statistics, obtain the topic distribution of each text description and the word distribution of each topic, and determine the keywords of each reference image according to the topic distribution and word distribution;

[0134] A repository creation module 14, used to create a distributed repository based on each reference image;

[0135] The matching storage module 15 is used to receive the satellite image to be classified containing text description and shooting parameters, match the text description of the image to be classified with the keywords of each reference image, and select a distributed storage repository for storage.

[0136] Furthermore, the image receiving module 11 includes:

[0137] An upload management unit, used to receive satellite images containing text descriptions and shooting parameters uploaded by the management party, and record the number of uploads;

[0138] A feature extraction unit, configured to extract feature values ​​of the satellite image based on a preset feature value extraction model when the upload quantity reaches a preset first quantity threshold;

[0139] An image comparison unit, used to compare any satellite image with all other satellite images to determine the selection probability of each satellite image;

[0140] The comparison process is:

[0141] Specifically, the text analysis module 13 includes:

[0142] A quantity receiving unit, used to receive the quantity of topics input by the management party;

[0143] a topic identification unit for extracting, for each text description, a topic distribution from a Richter distribution based on the number of topics;

[0144] A word recognition unit, for extracting a word distribution from the Lickley distribution for each topic;

[0145] A distribution updating unit, configured to iteratively update the topic distribution and the word distribution for each text description based on Gibbs sampling;

[0146] The iteration unit is used to output the final topic distribution and final word distribution of each text description after the iterative update is completed.

[0147] The iterative update process is:

[0148] Generate document content based on the extracted topic distribution and word distribution of each topic;

[0149] For each word w in the document content dn , calculate the conditional probability that the current word belongs to each topic, and reallocate the topic of the current word according to the calculated conditional probability;

[0150] The loop is executed until the preset number of loops is reached;

[0151] The document content generation process is as follows:

[0152] From the topic distribution θ d Extract a topic z from dn , from the selected topic z dn Word distribution Extract a word w from

[0153] The calculation process of conditional probability is:

[0154]

[0155] In the formula, P(z dn = k | z -dn , w, α, β) represents the conditional probability, z dn represents the topic assignment of the nth word in document d, z -dn represents the distribution of all topics except the nth word in document d, w represents the actual vocabulary of all words; α and β are preset values, both of which are 0.1;

[0156] n dk,-dn is the number of words in document d that belong to topic k, excluding the current word; n d,-dn is the total number of words in document d, excluding the current word, n kw,-dn is the number of occurrences of word w in topic k, excluding the current word, n k,-dn is the total number of words in topic k, excluding the current word; K is the dimension of each element in the topic distribution; V is the dimension of each element in the word distribution.

[0157] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A satellite big data distributed storage method, characterized in that: The method comprises: receiving satellite images containing text descriptions and shooting parameters uploaded by the management party, and when the number of uploaded images reaches a preset first number threshold, comparing each satellite image to determine the selection probability of each satellite image; Selecting a preset second quantity threshold of satellite images based on the selection probability to obtain a reference image set; Counting the text descriptions of each reference image in the reference image set, performing topic analysis and word analysis on the text descriptions obtained by statistics, obtaining the topic distribution of each text description and the word distribution of each topic, and determining the keywords of each reference image according to the topic distribution and word distribution; Using each reference image as a benchmark, a distributed repository is created; A satellite image to be classified containing text description and shooting parameters is received, the text description of the image to be classified is matched with keywords of each reference image, and a distributed storage repository is selected for storage.

2. The satellite big data distributed storage method according to claim 1, characterized in that: The step of receiving satellite images uploaded by the management party containing text descriptions and shooting parameters, and comparing each satellite image when the number of uploads reaches a preset first number threshold, and determining the selection probability of each satellite image comprises: Receive satellite images uploaded by the management party containing text descriptions and shooting parameters, and record the number of uploads; When the upload quantity reaches a preset first quantity threshold, extracting feature values ​​of the satellite image based on a preset feature value extraction model; Compare any satellite image with all other satellite images to determine the probability of selection of each satellite image; The comparison process is: Where P i is the selection probability of the i-th satellite image; Z i is the value corresponding to the i-th satellite image, Z is the total value of the values ​​corresponding to all satellite images; N is the number of uploads; α1, α2 and α3 are the preset correction coefficients, Δ j1 , Δ j2 and Δ j3 They represent the text description difference, shooting parameter difference and feature value difference between the i-th satellite image and the j-th satellite image respectively.

3. The satellite big data distributed storage method according to claim 1, characterized in that: The step of performing topic analysis and word analysis on the statistically obtained text descriptions to obtain the topic distribution of each text description and the word distribution of each topic includes: The number of topics entered by the receiving administrator; For each text description, extract a topic distribution from a Richter distribution based on the number of topics; For each topic, draw word distribution from the Richter distribution; For each text description, iteratively updating the topic distribution and the word distribution based on Gibbs sampling; After the iterative update is completed, the final topic distribution and final word distribution for each text description are output.

4. The satellite big data distributed storage method according to claim 3, characterized in that: The iterative update process is: Generate document content based on the extracted topic distribution and word distribution of each topic; For each word w in the document content dn , calculate the conditional probability that the current word belongs to each topic, and reallocate the topic of the current word according to the calculated conditional probability; The loop is executed until the preset number of loops is reached; The document content generation process is as follows: From the topic distribution θ d Extract a topic z from dn , from the selected topic z dn Word distribution Extract a word w from The calculation process of conditional probability is: In the formula, P(z dn = k | z -dn , w, α, β) represents the conditional probability, z dn represents the topic assignment of the nth word in document d, z -dn represents the distribution of all topics except the nth word in document d, w represents the actual vocabulary of all words; α and β are preset values, both 0.1; n dk,-dn is the number of words in document d that belong to topic k, excluding the current word; n d,-dn is the total number of words in document d, excluding the current word, n kw,-dn is the number of occurrences of word w in topic k, excluding the current word, n k,-dn is the total number of words in topic k, excluding the current word; K is the dimension of each element in the topic distribution; V is the dimension of each element in the word distribution.

5. The satellite big data distributed storage method according to claim 4, characterized in that: The content of keywords for each reference image determined based on topic distribution and word distribution includes: Get the final topic distribution of each text description and select the text topic; Get the word distribution corresponding to the text topic and select the word as the keyword; Among them, the selection rules of text topics and the selection rules of words are both that the probability is greater than a preset value.

6. The satellite big data distributed storage method according to claim 1, characterized in that: The method further comprises: When storing an image, each distributed repository determines whether to perform detection based on the retrieval probability; When no detection is performed, the storage process is executed to extract the feature value of the image based on the same feature extraction model, and the mean of the feature value of all images in the distributed storage library is updated; When performing detection, the feature value of the image is extracted based on the same feature extraction model, and the difference between the feature value and the mean feature value of all images in the distributed repository is calculated. When the difference reaches a preset difference threshold, a warning message is generated.

7. A satellite big data distributed storage system, characterized in that: The system comprises: An image receiving module is used to receive satellite images containing text descriptions and shooting parameters uploaded by the management party, and when the number of uploaded images reaches a preset first number threshold, compare each satellite image to determine the selection probability of each satellite image; An image set construction module, configured to select satellite images of a preset second quantity threshold based on the selection probability to obtain a reference image set; A text analysis module is used to count the text descriptions of each reference image in the reference image set, perform topic analysis and word analysis on the text descriptions obtained by statistics, obtain the topic distribution of each text description and the word distribution of each topic, and determine the keywords of each reference image according to the topic distribution and word distribution; A repository creation module, used to create a distributed repository based on each reference image; The matching storage module is used to receive the satellite image to be classified containing text description and shooting parameters, match the text description of the image to be classified with the keywords of each reference image, and select a distributed storage repository for storage.

8. The satellite big data distributed storage system according to claim 7, characterized in that: The image receiving module comprises: An upload management unit, used to receive satellite images containing text descriptions and shooting parameters uploaded by the management party, and record the number of uploads; A feature extraction unit, configured to extract feature values ​​of the satellite image based on a preset feature value extraction model when the upload quantity reaches a preset first quantity threshold; An image comparison unit, used to compare any satellite image with all other satellite images to determine the selection probability of each satellite image; The comparison process is: Where P i is the selection probability of the i-th satellite image; Z i is the value corresponding to the i-th satellite image, Z is the total value of the values ​​corresponding to all satellite images; N is the number of uploads; α1, α2 and α3 are the preset correction coefficients, Δ j1 , Δ j2 and Δ j3 They represent the text description difference, shooting parameter difference and feature value difference between the i-th satellite image and the j-th satellite image respectively.

9. The satellite big data distributed storage system according to claim 7, characterized in that: The text analysis module includes: A quantity receiving unit, used to receive the quantity of topics input by the management party; a topic identification unit for extracting, for each text description, a topic distribution from a Richter distribution based on the number of topics; A word recognition unit, for extracting a word distribution from the Lickley distribution for each topic; A distribution updating unit, configured to iteratively update the topic distribution and the word distribution for each text description based on Gibbs sampling; The iteration unit is used to output the final topic distribution and final word distribution of each text description after the iterative update is completed.

10. The satellite big data distributed storage system according to claim 9, characterized in that: The iterative update process is: Generate document content based on the extracted topic distribution and word distribution of each topic; For each word w in the document content dn , calculate the conditional probability that the current word belongs to each topic, and reallocate the topic of the current word according to the calculated conditional probability; The loop is executed until the preset number of loops is reached; The document content generation process is as follows: From the topic distribution θ d Extract a topic z from dn , from the selected topic z dn Word distribution Extract a word w from The calculation process of conditional probability is: In the formula, P(z dn = k | z -dn , w, α, β) represents the conditional probability, z dn represents the topic assignment of the nth word in document d, z -dn represents the distribution of all topics except the nth word in document d, w represents the actual vocabulary of all words; α and β are preset values, both 0.1; n dk,-dn is the number of words in document d that belong to topic k, excluding the current word; n d,-dn is the total number of words in document d, excluding the current word, n kw,-dn is the number of occurrences of word w in topic k, excluding the current word, n k,-dn is the total number of words in topic k, excluding the current word; K is the dimension of each element in the topic distribution; V is the dimension of each element in the word distribution.

Images