Satellite big data distributed storage method and system

Abstract

The application relates to the technical field of data storage, and particularly discloses a satellite big data distributed storage method and system, which comprises the following steps: obtaining a reference image set according to satellite images containing text descriptions and shooting parameters; counting the text descriptions of each reference image in the reference image set, carrying out theme analysis and word analysis on the counted text descriptions, obtaining theme distribution of each text description and word distribution of each theme, and determining the keywords of each reference image according to the theme distribution and the word distribution; taking each reference image as a benchmark to create a distributed storage library; and storing new satellite images based on the distributed storage library. When receiving satellite images, the application synchronously acquires text descriptions and shooting parameters, compares the existing satellite images and the text descriptions and the shooting parameters of the existing satellite images, selects reference images, classifies and stores the to-be-classified images according to the reference images, constructs a distributed storage architecture, and the storage process is very clear.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, and particularly relates to a satellite big data distributed storage method and system. BACKGROUND

[0002] Satellite big data refers to a large amount of data collected, processed, analyzed and utilized through satellite remote sensing technology. Satellite big data contains various types of information, such as images of the earth's surface, climate change, environmental monitoring, disaster warning, agricultural production, etc. Remote sensing data obtained by satellites, after complex data processing and analysis, can provide decision support, prediction analysis and real-time monitoring services for relevant management parties.

[0003] The existing satellite image storage process is mostly centralized storage architecture, and the data is not very "clean". In the processing and analysis stage, a screening process needs to be added, which affects the processing and analysis efficiency. How to improve the storage clarity of satellite big data is the technical problem that the present application technical scheme wants to solve. SUMMARY

[0004] The present application aims to provide a satellite big data distributed storage method and system to solve the problems raised in the background.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical scheme:

[0006] A satellite big data distributed storage method, the method comprising:

[0007] Receiving satellite images uploaded by a management party, the satellite images containing text descriptions and shooting parameters, when the number of uploaded satellite images reaches a preset first number threshold, comparing each satellite image to determine the selection probability of each satellite image;

[0008] Selecting satellite images of a preset second number threshold based on the selection probability to obtain a reference image set;

[0009] Counting the text descriptions of each reference image in the reference image set, performing theme analysis and word analysis on the counted text descriptions to obtain the theme distribution of each text description and the word distribution of each theme, and determining the keywords of each reference image according to the theme distribution and the word distribution;

[0010] Creating a distributed storage library based on each reference image;

[0011] Receiving satellite images to be classified containing text descriptions and shooting parameters, matching the text descriptions of the satellite images to be classified with the keywords of each reference image, and selecting the distributed storage library for storage.

[0012] As a further scheme of the present application: the step of comparing each satellite image to determine the selection probability of each satellite image when the uploaded quantity reaches the preset first quantity threshold value comprises:

[0013] Receiving satellite images uploaded by the management party, the satellite images containing text descriptions and shooting parameters, and recording the uploaded quantity;

[0014] When the uploaded quantity reaches the preset first quantity threshold value, extracting feature values of the satellite images based on a preset feature value extraction model;

[0015] Comparing any satellite image with all other satellite images to determine the selection probability of each satellite image;

[0016] The comparison process is as follows:

[0017] In the formula, P i is the selection probability of the i-th satellite image; Z i is the value corresponding to the i-th satellite image, and Z is the total value of the values corresponding to all satellite images; N is the uploaded quantity; α1, α2 and α3 are preset correction coefficients, Δ j1 , Δ j2 and Δ j3 respectively represent the text description difference, the shooting parameter difference and the feature value difference between the i-th satellite image and the j-th satellite image.

[0018] As a further scheme of the present application: the step of performing theme analysis and word analysis on the statistically obtained text descriptions to obtain the theme distribution of each text description and the word distribution of each theme comprises:

[0019] Receiving the theme quantity input by the management party;

[0020] For each text description, extracting a theme distribution from a Gompertz distribution based on the theme quantity;

[0021] For each theme, extracting a word distribution from the Gompertz distribution;

[0022] For each text description, iteratively updating the theme distribution and the word distribution based on Gibbs sampling;

[0023] After the iterative updating is completed, outputting the final theme distribution and the final word distribution for each text description.

[0024] As a further scheme of the present application: the iterative updating process is as follows:

[0025] Generating document content based on the extracted theme distribution and the word distribution of each theme;

[0026] For each word w in the document content dn , the conditional probability of the current word belonging to each topic is calculated, and the topic of the current word is redistributed according to the calculated conditional probability;

[0027] The loop is executed until a preset number of loops is reached;

[0028] The generation process of the document content is as follows:

[0029] A topic z is extracted from the topic distribution θ d A word w is extracted from the word distribution dn of the selected topic z dn ;

[0030] The calculation process of the conditional probability is as follows:

[0031]

[0032] In the formula, P(z dn =k|z -dn , w, α, β) represents the conditional probability, z dn represents the topic allocation of the nth word in the document d, z -dn represents the allocation of all topics except the nth word in the document d, w represents the actual vocabulary of all words; α and β are preset values, both of which are 0.1;

[0033] n dk,-dn is the number of words in the document d belonging to topic k, excluding the current word; n d,-dn is the total number of words in the document d, excluding the current word, n kw,-dn is the number of times the word w appears in topic k, excluding the current word, n k,-dn is the total number of words in topic k, excluding the current word; K is the dimension of each element in the topic distribution; V is the dimension of each element in the word distribution.

[0034] As a further scheme of the present application: the content of determining the key words of each reference image according to the topic distribution and the word distribution comprises:

[0035] Obtaining the final topic distribution of each text description, selecting a text topic;

[0036] Obtaining the word distribution corresponding to the text topic, and selecting a word as a key word;

[0037] The selection rule of the text topic and the selection rule of the word are both that the probability is greater than a preset value.

[0038] As a further scheme of the present application: the method further comprises: ​

[0039] Each distributed repository determines whether to perform detection according to a search probability when storing images;

[0040] When detection is not performed, a storage process is performed, feature values of the images are extracted based on the same feature extraction model, and the mean of the feature values of all images in the distributed repository is updated;

[0041] When detection is performed, feature values of the images are extracted based on the same feature extraction model, a difference between the feature values and the mean of the feature values of all images in the distributed repository is calculated, and when the difference reaches a preset difference threshold, an alert information is generated.

[0042] The technical scheme of the present application also provides a satellite big data distributed storage system, which comprises:

[0043] An image receiving module is configured to receive satellite images uploaded by a management party, the satellite images containing text descriptions and shooting parameters, and to determine selection probabilities of the satellite images by comparing the satellite images when the number of uploaded satellite images reaches a preset first number threshold;

[0044] An image set construction module is configured to select satellite images of a preset second number threshold based on the selection probabilities to obtain a reference image set;

[0045] A text analysis module is configured to count text descriptions of the reference images in the reference image set, to perform theme analysis and word analysis on the counted text descriptions to obtain theme distributions of the text descriptions and word distributions of each theme, and to determine keywords of each reference image according to the theme distributions and the word distributions;

[0046] A repository creation module is configured to create a distributed repository based on each reference image;

[0047] A matching storage module is configured to receive satellite images to be classified, which contain text descriptions and shooting parameters, to match the text descriptions of the satellite images to be classified with the keywords of the reference images, and to select the distributed repository for storage.

[0048] As a further scheme of the present application, the image receiving module comprises:

[0049] An upload management unit is configured to receive satellite images uploaded by a management party, the satellite images containing text descriptions and shooting parameters, and to record the number of uploaded satellite images;

[0050] A feature extraction unit is configured to extract feature values of the satellite images based on a preset feature value extraction model when the number of uploaded satellite images reaches a preset first number threshold;

[0051] An image comparison unit is configured to compare any satellite image with all other satellite images to determine selection probabilities of the satellite images.

[0052] The comparison process is:

[0053] In the formula, P i is the selection probability of the i-th satellite image; Z i is the corresponding numerical value of the i-th satellite image, Z is the total value of the corresponding numerical values of all satellite images; N is the upload quantity; α1, α2 and α3 are preset correction coefficients, Δ j1 , Δ j2 and Δ j3 respectively represent the text description difference, the shooting parameter difference and the feature value difference of the i-th satellite image and the j-th satellite image.

[0054] As a further scheme of the present application, the text analysis module comprises:

[0055] A quantity receiving unit configured to receive a topic quantity input by a management party;

[0056] A topic identifying unit configured to, for each text description, extract a topic distribution from a stick-breaking process based on the topic quantity;

[0057] A word identifying unit configured to, for each topic, extract a word distribution from a stick-breaking process;

[0058] A distribution updating unit configured to, for each text description, iteratively update the topic distribution and the word distribution based on Gibbs sampling;

[0059] An iteration unit configured to, after the iteration update is completed, output a final topic distribution and a final word distribution for each text description.

[0060] As a further scheme of the present application, the iteration update process is:

[0061] Generating a document content based on the extracted topic distribution and the word distribution of each topic;

[0062] For each word w dn in the document content, calculating a conditional probability that the current word belongs to each topic, and reassigning the topic of the current word according to the calculated conditional probability;

[0063] Performing the loop until a preset loop number is reached;

[0064] Wherein, the generation process of the document content is:

[0065] Extracting a topic z d from the topic distribution θ dn , and extracting a word w from the word distribution dn of the selected topic z ;

[0066] The calculation process of conditional probability is as follows:

[0067]

[0068] In the formula, P(z dn |z -dn , w, a, b) represents conditional probability, z dn represents the theme distribution of the nth word in the document d, z -dn represents the theme distribution of all words except the nth word in the document d, w represents the actual vocabulary of all words; a and b are preset values, both of which are 0.1.

[0069] n dk,-dn is the number of words belonging to the theme k in the document d, excluding the current word; n d,-dn is the total number of words in the document d, excluding the current word, n kw,-dn is the number of times of the word w in the theme k, excluding the current word, n k,-dn is the total number of words in the theme k, excluding the current word; K is the dimension of each element in the theme distribution; V is the dimension of each element in the word distribution.

[0070] Compared with the prior art, the present application has the beneficial effects that: when receiving satellite images, the present application synchronously acquires text descriptions and shooting parameters, compares the existing satellite images and their text descriptions and shooting parameters, selects reference images, classifies and stores the to-be-classified images according to the reference images, and constructs a distributed storage architecture, so that the storage process is very clear. BRIEF DESCRIPTION OF DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application.

[0072] Figure 1 It is a flowchart of the satellite big data distributed storage method.

[0073] Figure 2 It is a first sub-flowchart of the satellite big data distributed storage method.

[0074] Figure 3 It is a second sub-flowchart of the satellite big data distributed storage method.

[0075] Figure 4 It is a composition structure diagram of the satellite big data distributed storage system. DETAILED DESCRIPTION

[0076] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clear and explicit, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0077] Figure 1 For the flowchart of the satellite big data distributed storage method, in the embodiment of the present application, a satellite big data distributed storage method, the method comprises:

[0078] Step S100: receiving satellite images uploaded by a management party and containing text descriptions and shooting parameters, comparing each satellite image when the number of uploads reaches a preset first number threshold, and determining the selection probability of each satellite image;

[0079] Step S100 is the receiving stage of the satellite image, which is selected by the management party, and some satellite images containing text descriptions and shooting parameters are uploaded, the text description is the description of the satellite image, and the shooting parameter is the shooting coordinate and definition of the satellite image. In the working process, the management party will receive the satellite images uploaded by the shooting party of the satellite image, count and select them, and upload them to the method execution platform.

[0080] It should be noted that the screening process can be manual screening or intelligent screening. The intelligent screening process only screens the data integrity, such as whether there is a missing part in the text description and the shooting parameter, or whether there is a missing part in the satellite image, and the screening scheme can adopt a conventional comparison scheme.

[0081] When receiving the satellite images containing the text descriptions and the shooting parameters, the received satellite images are compared, and then the selection probability of each satellite image is determined.

[0082] Step S200: selecting satellite images of a preset second number threshold based on the selection probability to obtain a reference image set;

[0083] Step S200 is a further screening limitation on step S100. Based on the selection probability, the received satellite images are continuously screened, and then a small database, referred to as a reference image set, is constructed. In simple terms, step S100 continuously receives the uploaded satellite images and stores them in a large database. In the large database, the selection probability of each satellite image is continuously calculated, and then some satellite images are randomly selected based on the selection probability to establish a small database, i.e. the reference image set.

[0084] Step S300: count the text descriptions of each reference image in the reference image set, perform topic analysis and word analysis on the counted text descriptions to obtain the topic distribution of each text description and the word distribution of each topic, and determine the keywords of each reference image according to the topic distribution and the word distribution;

[0085] The text descriptions of each reference image in the reference image set are read to create a text library. Topic analysis is performed on the text library to obtain a topic distribution, i.e., the distribution of each topic in the text library. A text can include multiple topics. For each topic, a word distribution can be obtained. This process is a specific application of the LDA algorithm, which is a topic model algorithm for text analysis. After the topic distribution and the word distribution are determined, the keywords of each text description, i.e., the keywords of each reference image (reference image and text description are one-to-one corresponding), can be easily determined.

[0086] Step S400: create a distributed storage library based on each reference image;

[0087] An independent storage library is created based on each reference image for storing satellite images similar to the reference image.

[0088] Step S500: receive a satellite image to be classified containing a text description and a shooting parameter, match the text description of the satellite image to be classified with the keywords of each reference image, and select a distributed storage library for storage;

[0089] Step S500 is the application stage. When a satellite image to be classified is received, it is compared with the existing reference images to calculate the matching degree. When the matching degree is high enough, the corresponding storage library is located, and the satellite image is stored in the corresponding storage library, thereby realizing the distributed storage function.

[0090] It should be noted that the satellite image of the present application has a text description and a shooting parameter. If the text description and the shooting parameter are missing, it is considered to have no storage value and is not stored. The scheme provided by the present application essentially stores satellite image big data. The text description and the shooting parameter are used to ensure that it has storage value. For example, in the field of agricultural analysis, the text description is a growth evaluation report of crops in the satellite image. Only the satellite image with the growth evaluation report has storage value and is considered as the processing object of the platform.

[0091] Figure 2 The first sub-flow chart of the satellite big data distributed storage method is shown in FIG. 1. The step of comparing each satellite image to determine the selection probability of each satellite image when the number of uploaded satellite images containing text descriptions and shooting parameters reaches a preset first number threshold value includes:

[0092] Step S101: receiving satellite images uploaded by the management party, the satellite images containing text descriptions and shooting parameters, and recording the number of uploads;

[0093] Step S102: when the number of uploads reaches a preset first number threshold, extracting feature values of the satellite images based on a preset feature value extraction model;

[0094] Step S103: comparing any satellite image with all other satellite images to determine the selection probability of each satellite image.

[0095] In one example of the technical scheme, satellite images uploaded by the management party are received, the satellite images containing text descriptions and shooting parameters, and the number of uploads is recorded in real time. When the number of uploads is sufficient, feature values of the satellite images are extracted based on a preset feature value extraction model, and each satellite image is compared based on the feature values, the text descriptions and the shooting parameters. The comparison results of any satellite image with all other satellite images are read to determine the selection probability of each satellite image.

[0096] The comparison process is as follows:

[0097] In the formula, P i is the selection probability of the i-th satellite image; Z i is the value corresponding to the i-th satellite image, and Z is the total value of the values corresponding to all satellite images; N is the number of uploads; and α1, α2 and α3 are preset correction coefficients, Δ j1 , Δ j2 and Δ j3 respectively represent the text description difference, the shooting parameter difference and the feature value difference between the i-th satellite image and the j-th satellite image.

[0098] The calculation schemes of the text description difference, the shooting parameter difference and the feature value difference in the above content can use existing schemes. The difference in the text description generally uses a conventional text similarity. The simplest way is to compare the number of words. The difference in the shooting parameter generally uses a spatial distance. The shooting parameters include coordinates and definition. The definition is generally the same, and the spatial distance can be calculated from the coordinates. The difference in the feature value is simpler, and the numerical value difference can be calculated.

[0099] Figure 3 FIG. 2 is a second sub-flow block diagram of the satellite big data distributed storage method. The step of performing theme analysis and word analysis on the statistically obtained text descriptions to obtain the theme distribution of each text description and the word distribution of each theme includes:

[0100] Step S301: receiving the number of themes input by the management party;

[0101] Step S302: for each text description, a topic distribution is drawn from Dirichlet distribution based on the number of topics;

[0102] Step S303: for each topic, a word distribution is drawn from Dirichlet distribution;

[0103] Step S304: for each text description, the topic distribution and the word distribution are iteratively updated based on Gibbs sampling;

[0104] Step S305: after the iterative updating is completed, the final topic distribution and the final word distribution for each text description are output.

[0105] The number of topics is input in advance by an administrator, such as ten; for each document in the text library, a topic distribution θ d , θ d ~ Dir(α), θ d is a K-dimensional vector, and K is the number of topics; for each topic k, a word distribution φ is a V-dimensional vector, representing the probability of a word under the topic.

[0106] The topic distribution and the word distribution drawn above are a baseline model, and for each text, the topic distribution and the word distribution are iteratively updated based on Gibbs sampling, the baseline model is constantly updated, and the final topic distribution in each document and the word distribution under each topic are obtained.

[0107] It should be noted that Dirichlet distribution is a continuous multivariate probability distribution, which is a prior distribution commonly used in Bayesian statistics, and is particularly suitable for modeling the parameters of multinomial distribution. The main feature of Dirichlet distribution is that it can describe the probability distribution of multiple random variables, and the sum of these random variables is 1, which makes it particularly suitable for modeling situations such as probability vectors and weight vectors.

[0108] Specifically, the iterative updating process is as follows:

[0109] Generate document content based on the drawn topic distribution and the word distribution of each topic;

[0110] For each word w dn in the document content, calculate the conditional probability of the current word belonging to each topic, and reassign the topic of the current word according to the calculated conditional probability;

[0111] Loop until a preset number of loops is reached;

[0112] The generation process of the document content is as follows:

[0113] extract a topic z from the topic distribution θ d extract a word w from the word distribution of the selected topic z dn dn

[0114] The calculation process of conditional probability is as follows:

[0115]

[0116] In the formula, P(z dn =k|z -dn , w, a, b) represents conditional probability, z dn represents the topic assignment of the nth word in the document d, z -dn represents the assignment of all topics except the nth word in the document d, w represents the actual vocabulary of all words; a and b are preset values, both of which are 0.1;

[0117] n dk,-dn is the number of words belonging to topic k in the document d, excluding the current word; n d,-dn is the total number of words in the document d, excluding the current word, n kw,-dn is the number of times the word w appears in topic k, excluding the current word, n k,-dn is the total number of words in topic k, excluding the current word; K is the dimension of each element in the topic distribution; V is the dimension of each element in the word distribution.

[0118] The above is a specific application of the Gibbs sampling process, which approaches the true topic distribution of the document and the true word distribution of each topic by iteratively updating the topic assignment of each word.

[0119] As a preferred embodiment of the technical scheme of the present application, the content of determining the key words of each reference image according to the topic distribution and the word distribution comprises:

[0120] obtaining the final topic distribution of each text description, and selecting a text topic;

[0121] obtaining the word distribution corresponding to the text topic, and selecting a word as a key word;

[0122] The selection rules of the text topic and the selection rules of the word are both that the probability is greater than a preset value.

[0123] When the final topic distribution of each text and its word distribution are determined, high-probability words in these distributions can be selected as key words, wherein the selection process is to set a probability threshold, and when the probability of the text topic reaches the probability threshold, selection is performed, and when the selection probability of the word reaches the probability threshold, selection is performed.​​​

[0124] As a preferred embodiment of the technical scheme of the present application, the method further comprises:

[0125] When storing the image, each distributed repository determines whether to perform detection according to a search probability;

[0126] When not performing detection, a storage process is performed, feature values of the image are extracted based on the same feature extraction model, and the mean of the feature values of all images in the distributed repository is updated;

[0127] When performing detection, feature values of the image are extracted based on the same feature extraction model, a difference between the feature values and the mean of the feature values of all images in the distributed repository is calculated, and when the difference reaches a preset difference threshold, an alert information is generated.

[0128] In an example of the technical scheme of the present application, a detection process in the storage repository warehousing stage is introduced. First, a search probability, for example, 10%, is set. When storing the image, each distributed repository determines whether to perform detection based on the search probability, that is, there is a 10% probability of needing detection. When detection is not triggered, normal warehousing is performed, feature values of the image are extracted based on the same feature extraction model, and the mean of the feature values of all images in the distributed repository is updated. When detection is triggered, feature values of the image are extracted based on the same feature extraction model, a difference between the feature values and the mean of the feature values of all images in the distributed repository is calculated, and when the difference reaches a preset difference threshold, it is indicated that an error occurs in the matching process. At this time, an alert information is generated.

[0129] It should be noted that the specific model of the feature extraction model involved in the present application is not limited, but the applied feature extraction model needs to be counted, that is, the present application involves only one kind of feature extraction model. The input of the feature extraction model is an image, and the output is a numerical value. An existing feature extraction model can be used. The simplest way is to perform gray scale conversion on the image and then calculate the gray scale feature values (mean and standard deviation, etc.).

[0130] Figure 4 The satellite big data distributed storage system is a component structure diagram. In the embodiment of the present application, a satellite big data distributed storage system, the system 10 comprises:

[0131] An image receiving module 11 is configured to receive satellite images uploaded by a management party, the satellite images containing text descriptions and shooting parameters, and compare the satellite images when the number of uploaded satellite images reaches a preset first number threshold to determine selection probabilities of the satellite images.

[0132] An image set construction module 12 is configured to select satellite images of a preset second number threshold based on the selection probabilities to obtain a reference image set.

[0133] The text analysis module 13 is configured to count the text descriptions of the reference images in the reference image set, perform topic analysis and word analysis on the counted text descriptions, obtain topic distribution of each text description and word distribution of each topic, and determine the keywords of each reference image according to the topic distribution and the word distribution;

[0134] The storage library creation module 14 is configured to create a distributed storage library based on each reference image;

[0135] The matching storage module 15 is configured to receive a satellite image to be classified containing a text description and shooting parameters, match the text description of the satellite image to be classified with the keywords of each reference image, and select the distributed storage library for storage.

[0136] Further, the image receiving module 11 comprises:

[0137] The upload management unit is configured to receive satellite images uploaded by the management party and containing text descriptions and shooting parameters, and record the number of uploads;

[0138] The feature extraction unit is configured to extract feature values of the satellite images based on a preset feature value extraction model when the number of uploads reaches a preset first number threshold;

[0139] The image comparison unit is configured to compare any satellite image with all other satellite images to determine the selection probability of each satellite image;

[0140] The comparison process is as follows:

[0141] Specifically, the text analysis module 13 comprises:

[0142] The number receiving unit is configured to receive the number of topics input by the management party;

[0143] The topic identification unit is configured to extract a topic distribution from a Dirichlet distribution based on the number of topics for each text description;

[0144] The word identification unit is configured to extract a word distribution from a Dirichlet distribution for each topic;

[0145] The distribution updating unit is configured to update the topic distribution and the word distribution based on Gibbs sampling iteration for each text description;

[0146] The iteration unit is configured to output the final topic distribution and the final word distribution for each text description after the iteration update is completed.

[0147] The iteration update process is as follows:

[0148] The document content is generated based on the extracted topic distribution and the word distribution of each topic.

[0149] For each word w in the document content dn , the conditional probability of the current word belonging to each topic is calculated, and the topic of the current word is reassigned according to the calculated conditional probability;

[0150] The loop is executed until a preset number of loops is reached;

[0151] wherein the generation process of the document content is:

[0152] a topic z is extracted from the topic distribution θ d ; a word w is extracted from the word distribution dn of the selected topic z dn ;

[0153] The calculation process of the conditional probability is:

[0154]

[0155] In the formula, P(z dn =k|z -dn , w, α, β) represents the conditional probability, z dn represents the topic assignment of the nth word in the document d, z -dn represents the assignment of all topics except the nth word in the document d, and w represents the actual vocabulary of all words; α and β are preset values, both of which are 0.1;

[0156] n dk,-dn is the number of words belonging to the topic k in the document d, excluding the current word; n d,-dn is the total number of words in the document d, excluding the current word, n kw,-dn is the number of times the word w appears in the topic k, excluding the current word, n k,-dn is the total number of words in the topic k, excluding the current word; K is the dimension of each element in the topic distribution; and V is the dimension of each element in the word distribution.

[0157] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, and improvement within the spirit and principle of the present application shall be included in the protection scope of the present application.​

Claims

1. A satellite big data distributed storage method, characterized in that, The method comprises: receiving satellite images uploaded by a management party, the satellite images containing text descriptions and shooting parameters, comparing each satellite image when the number of uploaded satellite images reaches a preset first number threshold, and determining a selection probability of each satellite image; selecting satellite images of a preset second number threshold based on the selection probability to obtain a reference image set; counting the text descriptions of each reference image in the reference image set, performing theme analysis and word analysis on the counted text descriptions, obtaining theme distribution of each text description and word distribution of each theme, and determining keywords of each reference image according to the theme distribution and the word distribution; creating a distributed storage library based on each reference image; receiving a satellite image to be classified containing a text description and shooting parameters, matching the text description of the satellite image to be classified with the keywords of each reference image, and selecting the distributed storage library for storage; the step of receiving satellite images uploaded by a management party, the satellite images containing text descriptions and shooting parameters, comparing each satellite image when the number of uploaded satellite images reaches a preset first number threshold, and determining a selection probability of each satellite image comprises: receiving satellite images uploaded by a management party, the satellite images containing text descriptions and shooting parameters, and recording the number of uploaded satellite images; when the number of uploaded satellite images reaches a preset first number threshold, extracting feature values of the satellite images based on a preset feature value extraction model; comparing any satellite image with all other satellite images to determine the selection probability of each satellite image; the comparison process is: wherein P i is the selection probability of the i-th satellite image; Z i is the corresponding numerical value of the i-th satellite image, Z is the total value of the corresponding numerical values of all satellite images; N is the upload quantity; α1, α2 and α3 are preset correction coefficients, Δ j1 , Δ j2 and Δ j3 respectively represent the text description difference, the shooting parameter difference and the feature value difference of the i-th satellite image and the j-th satellite image. the step of performing theme analysis and word analysis on the counted text descriptions, obtaining theme distribution of each text description and word distribution of each theme, comprises: receiving a theme number input by the management party; for each text description, extracting a theme distribution from a Gompertz distribution based on the theme number; for each theme, extracting a word distribution from the Gompertz distribution; for each text description, iteratively updating the theme distribution and the word distribution based on Gibbs sampling; after the iterative updating is completed, outputting the final theme distribution and the final word distribution of each text description. 2.The satellite big data distributed storage method according to claim 1, characterized in that, the iterative updating process is: generating document content based on the extracted theme distribution and the word distribution of each theme; For each word w in the content of the document dn , compute a conditional probability that the current word belongs to each topic, reassign the topic of the current word based on the computed conditional probabilities; performing a loop until a preset loop number is reached; wherein the generation process of the document content is: From topic distribution θ d Extract a topic z from dn From the selected topic z dn Word distribution Extract a word 'w' from the text. the calculation process of the conditional probability is: where P(z dn = k | z -dn , w, a, β) denotes conditional probability, z dn denotes the topic assignment of the nth word in the document d, z -dn denotes the assignment of all topics except the nth word in the document d, w denotes the actual vocabulary of all words; a and β are preset values, both of which are 0.

1. n dk,-dn is the number of words in document d that belong to topic k, excluding the current word; n d,-dn is the total number of words in document d, excluding the current word, n kw,-dn is the number of occurrences of word w in topic k, excluding the current word, n k,-dn is the total number of words in topic k, excluding the current word; K is the dimension of the topic distribution; V is the dimension of the word distribution. 3.The satellite big data distributed storage method of claim 1, wherein, the content of determining keywords of each reference image according to the theme distribution and the word distribution comprises: obtaining the final theme distribution of each text description and selecting a text theme; obtaining the word distribution corresponding to the text theme and selecting a word as a keyword; wherein the selection rule of the text theme and the selection rule of the word are both that the probability is greater than a preset value. 4.The satellite big data distributed storage method of claim 1, wherein, The method further comprises: when storing an image, each distributed storage library determines whether to perform detection according to a detection probability; when not performing detection, performing a storage process, extracting feature values of the image based on the same feature extraction model, and updating the mean value of the feature values of all images in the distributed storage library; When detection is performed, feature values of the image are extracted based on the same feature extraction model, a difference between the feature values and a mean value of feature values of all images in a distributed repository is calculated, and when the difference reaches a preset difference threshold, alert information is generated.

5. A satellite big data distributed storage system, characterized in that, The system comprises: An image receiving module configured to receive satellite images uploaded by a management party, the satellite images containing text descriptions and shooting parameters, and compare each satellite image when the number of uploaded satellite images reaches a preset first number threshold to determine a selection probability of each satellite image. An image set construction module configured to select a preset second number threshold of satellite images based on the selection probability to obtain a reference image set. A text analysis module configured to count text descriptions of each reference image in the reference image set, perform theme analysis and word analysis on the counted text descriptions to obtain theme distribution of each text description and word distribution of each theme, and determine a keyword of each reference image based on the theme distribution and the word distribution. A repository creation module configured to create a distributed repository based on each reference image. A matching storage module configured to receive a satellite image to be classified containing a text description and shooting parameters, match the text description of the satellite image to be classified with the keyword of each reference image, and select the distributed repository for storage. The image receiving module comprises: An upload management unit configured to receive satellite images uploaded by the management party, the satellite images containing text descriptions and shooting parameters, and record the number of uploaded satellite images. A feature extraction unit configured to extract feature values of the satellite images based on a preset feature value extraction model when the number of uploaded satellite images reaches a preset first number threshold. An image comparison unit configured to compare any satellite image with all other satellite images to determine a selection probability of each satellite image. The comparison process is as follows: wherein P i is the selection probability of the i-th satellite image; Z i is the corresponding numerical value of the i-th satellite image, Z is the total value of the corresponding numerical values of all satellite images; N is the upload quantity; α1, α2 and α3 are preset correction coefficients, Δ j1 , Δ j2 and Δ j3 respectively represent the text description difference, the shooting parameter difference and the feature value difference of the i-th satellite image and the j-th satellite image. The text analysis module comprises: A number receiving unit configured to receive a theme number input by the management party. A theme identification unit configured to extract theme distribution from a Gompertz distribution based on the theme number for each text description. A word identification unit configured to extract word distribution from a Gompertz distribution for each theme. A distribution updating unit configured to update the theme distribution and the word distribution based on Gibbs sampling iteration for each text description. An iteration unit configured to output a final theme distribution and a final word distribution for each text description after the iteration updating is completed.

Images