Method and system for supporting space science and application data quality analysis and evaluation

Abstract

The invention provides a method and system for supporting space science and application data quality analysis and evaluation, and the method comprises the steps: determining a target evaluation index type based on the data type of data included in any file; and determining an evaluation index corresponding to the target evaluation index type as a target evaluation index, and obtaining an actual evaluation range of data included in any file under each target evaluation index. And on the basis that the actual evaluation range of any file under any target evaluation index does not completely fall into the preset evaluation range of any target evaluation index, determining that any target evaluation index corresponding to any file is abnormal. And determining that any file is abnormal based on the fact that the ratio of the number of the abnormal target evaluation indexes corresponding to any file to the total number of the target evaluation indexes is greater than an abnormal proportion threshold. According to the invention, quality analysis and evaluation of space science and application data can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of manned space engineering data processing, and in particular to a method and system for supporting space science and application data quality analysis and evaluation. Background Art

[0002] Space science and application data refers to data related to space science experiments, outer space, the Earth, etc. obtained by scientific experiments conducted on orbit by scientific experiment payloads of space stations. It includes information of multiple types (such as images, videos, parameters, etc.) and multiple disciplines (such as materials, biology, astrophysics, astronomy, etc.). It is a very important information resource that can be used for scientific research, resource development, environmental monitoring, etc. It plays an important role in the application of life sciences, materials sciences, microgravity physics and other fields, and has important significance and value.

[0003] Data quality analysis and evaluation refers to the systematic analysis and evaluation of data, including a series of steps such as data collection, cleaning, format interpretation, integration, and management. It aims to ensure the accuracy, completeness, consistency, and reliability of the data. It is an important part of data processing production, management, and application.

[0004] Due to the large scale of space science and application data and the complexity of related fields, extremely high requirements are placed on the security and reliability of downlink data transmission, reception, interpretation, storage and other stages. Therefore, it is necessary to conduct quality analysis and evaluation of the data involved to understand the quality of the data, discover problems in a timely manner and effectively solve them, and ensure the accuracy and integrity of the data.

[0005] It can be seen that how to conduct quality analysis and evaluation of space science and application data is an urgent problem that needs to be solved. Summary of the invention

[0006] The present invention provides a method and system for supporting space science and application data quality analysis and evaluation, aiming to solve the technical problems described in the above background technology.

[0007] The technical solution of the present invention to solve the above technical problems is as follows:

[0008] In a first aspect, the present invention provides a method for supporting data quality analysis and evaluation in space science and applications, in which, for any file in at least one file to be analyzed, based on the data type of the data included in any file, a target evaluation index type is determined in at least one evaluation index type. Different evaluation index types correspond to different evaluation indicators, and the evaluation index type is used to characterize the preset evaluation range of data of the corresponding data type. The evaluation index corresponding to the target evaluation index type is determined as the target evaluation index, and the actual evaluation range of the data included in any file under each target evaluation index is obtained. For any target evaluation index, based on the fact that the actual evaluation range of any file under any target evaluation index does not completely fall into the preset evaluation range of any target evaluation index, it is determined that any target evaluation index corresponding to any file is abnormal. Based on the ratio of the number of abnormal target evaluation indicators corresponding to any file to the total number of target evaluation indicators being greater than the abnormal ratio threshold, it is determined that any file is abnormal.

[0009] Based on the above technical solution, the present invention can also be improved as follows.

[0010] Further, at least one evaluation index type includes a parameter data evaluation index type, an image data evaluation index type, and a video data evaluation index type. In the case where any file includes data of a parameter data type or a binary data type, the parameter data evaluation index type is determined as the target evaluation index type. In the case where any file includes data of an image data type or a spectral data type, the image data evaluation index type is determined as the target evaluation index type. In the case where any file includes data of a video data type, the video data evaluation index type is determined as the target evaluation index type.

[0011] Further, the evaluation index corresponding to the parameter data evaluation index type includes at least one of the data mean, data median, data standard deviation and reasonable data ratio. Wherein, the reasonable data ratio is the ratio of the first value to the second value. The first value is the number of sampled data that falls within the preset minimum reasonable value range and the preset maximum reasonable value range, and the second value is the total number of all sampled data. The evaluation index corresponding to the image data evaluation index type includes at least one of the image signal-to-noise ratio, image clarity, image contrast, image information entropy, the range of grayscale values ​​of pixels included in the image, image brightness, image resolution, image pixels, and the occlusion rate of the experimental target in the image. The evaluation index corresponding to the video data evaluation index type includes at least one of the video signal-to-noise ratio, video clarity, video contrast, video information entropy, the range of grayscale values ​​of pixels included in the video, video brightness, video resolution, video pixels, video frame rate, video continuity, and video distortion. Wherein, video continuity is associated with the similarity between adjacent frame images in the video.

[0012] Furthermore, at every preset period, any file in an unselected state in at least one file to be analyzed is determined as any file, and any file is marked as a selected state.

[0013] Further, after determining that any file is abnormal, any file is marked as unselected.

[0014] Further, a data analysis instruction input by a user is received. In response to the data analysis instruction, any file is determined in at least one file to be analyzed.

[0015] Further, information associated with the abnormality of any target evaluation index corresponding to any file is displayed in the early warning interface. After determining that any file is abnormal, information associated with the abnormality of any file is displayed in the early warning interface. The information associated with the abnormality of any target evaluation index corresponding to any file and the information associated with the abnormality of any file are displayed in the early warning interface in at least one form of text, table, and image.

[0016] Further, a configuration page is displayed, and the configuration page is used to support the user to set at least one evaluation indicator type. In response to the user inputting a selection operation of a first evaluation indicator type among the at least one evaluation indicator type, a list of evaluation indicators corresponding to the first evaluation indicator type is displayed. In response to the user inputting a selection operation of a first evaluation indicator among the evaluation indicators included in the list, an input box for adjusting a preset evaluation range of the first evaluation indicator is displayed. In response to the user inputting a target preset evaluation range in the input box, the preset evaluation range of the first evaluation indicator is adjusted to the target preset evaluation range.

[0017] In a second aspect, the present invention provides a system for supporting quality analysis and evaluation of space science and application data, the system comprising a quality analysis and evaluation management module. The quality analysis and evaluation management module is used to:

[0018] For any file in at least one file to be analyzed, based on the data type of the data included in any file, determine the target evaluation indicator type in at least one evaluation indicator type. Wherein, different evaluation indicator types correspond to different evaluation indicators, and the evaluation indicator type is used to characterize the preset evaluation range of data of the corresponding data type. The evaluation indicator corresponding to the target evaluation indicator type is determined as the target evaluation indicator, and the actual evaluation range of the data included in any file under each target evaluation indicator is obtained. For any target evaluation indicator, based on the fact that the actual evaluation range of any file under any target evaluation indicator does not completely fall into the preset evaluation range of any target evaluation indicator, it is determined that any target evaluation indicator corresponding to any file is abnormal. Based on the fact that the ratio of the number of abnormal target evaluation indicators corresponding to any file to the total number of target evaluation indicators is greater than the abnormal ratio threshold, it is determined that any file is abnormal.

[0019] Based on the above technical solution, the present invention can also be improved as follows.

[0020] Furthermore, the system provided by the present invention also includes a quality assessment anomaly and analysis module and a result and report management module. The quality assessment anomaly and analysis module is used to:

[0021] A configuration page is displayed, and the configuration page is used to support the user to set at least one evaluation indicator type. In response to the user inputting a selection operation of a first evaluation indicator type among the at least one evaluation indicator type, a list of evaluation indicators corresponding to the first evaluation indicator type is displayed. In response to the user inputting a selection operation of a first evaluation indicator among the evaluation indicators included in the list, an input box for adjusting a preset evaluation range of the first evaluation indicator is displayed. In response to the user inputting a target preset evaluation range in the input box, the preset evaluation range of the first evaluation indicator is adjusted to the target preset evaluation range.

[0022] The Results and Report Management module is used to:

[0023] Displaying an early warning interface, the early warning interface including information associated with any abnormality of any target evaluation indicator corresponding to any file, and information associated with any abnormality of the file. The information associated with any abnormality of any target evaluation indicator corresponding to any file and the information associated with any abnormality of the file are displayed in the early warning interface in at least one form of text, table, and image.

[0024] In a third aspect, the present invention provides an electronic device comprising: a memory, one or more processors; the memory and the processor are coupled; wherein the memory stores computer program code, the computer program code comprises computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the method for supporting space science and application data quality analysis and evaluation as described in any one of the first aspects above.

[0025] In a fourth aspect, a computer-readable storage medium is provided, comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method for supporting space science and application data quality analysis and evaluation as described in any one of the first aspects.

[0026] In a fifth aspect, a computer program product is provided. When the computer program product is run on a computer, the computer is enabled to execute the method for supporting space science and application data quality analysis and evaluation as described in any one of the first aspects above.

[0027] The beneficial effects of the present invention are as follows: based on different data types, different evaluation index types and evaluation indexes corresponding to each evaluation index type are designed to ensure the accuracy of the evaluation of different types of data products. By constructing a data product evaluation process, the quality evaluation task is performed in two ways: scheduled tasks and manual initiation, which can ensure the integrity of the data product quality evaluation. By detecting and tracking data products with abnormal quality in real time, the changes in downlink data and product quality can be grasped in a timely manner, and the downlink data processing model and algorithm can be corrected and optimized. Displaying the evaluation results can assist scientific management and decision-making, and enhance the operation, maintenance and decision-making capabilities of the space science and application data platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic flow chart of a method for supporting space science and application data quality analysis and evaluation provided by the present invention;

[0029] Figure 2 A schematic diagram of the flow chart of the manual working mode provided by the present invention;

[0030] Figure 3 A schematic diagram of the process flow of the automatic working mode provided by the present invention;

[0031] Figure 4 A schematic diagram of the level 0 data customization task flow provided by the present invention;

[0032] Figure 5 A schematic diagram of the data quality analysis task flow provided by the present invention;

[0033] Figure 6 A schematic diagram of a quality evaluation process monitoring page provided by the present invention;

[0034] Figure 7 A schematic diagram of a judgment process provided by the present invention;

[0035] Figure 8 A schematic diagram of abnormal result query provided by the present invention;

[0036] Fig. 9 A schematic diagram of an early warning interface provided by the present invention;

[0037] Fig.10 A schematic diagram of a report query interface provided by the present invention;

[0038] Fig.11 A schematic diagram of a report query preview provided by the present invention;

[0039] Fig.12 A schematic diagram of the structure of the system for supporting space science and application data quality analysis and evaluation provided by the present invention;

[0040] Fig.13 A schematic diagram of the flow of the configuration algorithm program provided by the present invention;

[0041] Fig.14 A schematic diagram of a data quality analysis task configuration page provided by the present invention;

[0042] Fig.15 This is a schematic diagram of the abnormal indicator configuration page provided by the present invention. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the present application, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship, for example, A / B can represent A or B; "and / or" in the present application is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. And, in the description of the present application, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish between the same items or similar items with substantially the same functions and effects. Those skilled in the art will understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not necessarily limit the differences. At the same time, in the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design.

[0044] The present invention provides a method and system for supporting quality analysis and evaluation of space science and application data, which can be applied to any scenario of quality analysis and evaluation of space science and application data. Through the method of the present invention, different evaluation index types and evaluation parameters corresponding to each evaluation index type are designed based on different data types, so as to ensure the accuracy of the evaluation of different types of data products; by constructing a data product evaluation process, the quality evaluation task is executed in two ways: scheduled task and manual initiation, so as to ensure the integrity of the data product quality evaluation; by detecting and tracking data products with abnormal quality in real time, the changes in downlink data and product quality can be grasped in time, and the downlink data processing model and algorithm can be corrected and optimized; the statistical charts and analysis reports of the results can assist scientific management and decision-making, and enhance the operation and maintenance and decision-making capabilities of the space science and application data platform.

[0045] The solution of the present invention is further described below in conjunction with specific embodiments.

[0046] See also Figure 1, the method for supporting space science and application data quality analysis and evaluation provided by the present invention includes the following steps S101-S104:

[0047] S101: For any file among at least one file to be analyzed, based on the data type of data included in any file, determine a target evaluation index type in at least one evaluation index type.

[0048] Among them, different evaluation indicator types correspond to different evaluation indicators, and the evaluation indicators are used to characterize a preset evaluation range of data of the corresponding data type.

[0049] In some embodiments, the data format type of the space science and application data quality analysis and evaluation system can be divided into binary data, parameter data, image data, video data and spectral data. Among them, binary data can be regarded as a special parameter data, and spectral data can be regarded as a special image data. Based on this, for different data types, the present invention can set different evaluation index types, and evaluate data of different data types based on the evaluation indexes corresponding to different evaluation index types.

[0050] Specifically, at least one evaluation index type in the present invention includes a parameter data evaluation index type, an image data evaluation index type, and a video data evaluation index type. In the case where any file includes data of a parameter data type or a binary data type, the parameter data evaluation index type can be determined as the target evaluation index type. In the case where any file includes data of an image data type or a spectral data type, the image data evaluation index type can be determined as the target evaluation index type. In the case where any file includes data of a video data type, the video data evaluation index type can be determined as the target evaluation index type.

[0051] In some embodiments, the evaluation index corresponding to the parameter data evaluation index type includes at least one of the data mean, data median, data maximum, data minimum, data standard deviation, and reasonable data ratio. Wherein, the reasonable data ratio is the ratio of the first value to the second value. The first value is the number of sampled data falling within the preset minimum reasonable value range and the preset maximum reasonable value range, and the second value is the total number of all sampled data.

[0052] In some embodiments, the evaluation indicators corresponding to the image data evaluation indicator type include image signal-to-noise ratio, image clarity, image contrast, image information entropy, the range of grayscale values ​​of pixels included in the image, image brightness, image resolution, image pixels, and at least one of the occlusion rate of the experimental target in the image.

[0053] Specifically, the image signal-to-noise ratio is an objective standard for evaluating image quality based on the error between corresponding pixels. It is the logarithm of the ratio of the image mean to the image standard deviation, and the unit is dB. When the input is an RGB three-channel image, Gray=0.1140*R+0.5870*G+0.2989*B is used to convert it into a grayscale image. When the input is a grayscale image, the specific calculation formula is:

[0054]

[0055]

[0056] Among them, height and width represent the height and width of the original image respectively, and the image is evenly divided into several sub-block areas with a side length of blk. According to experience, blk is generally taken as 4. For each sub-block area with a side length of blk, its pixel mean mean(i,j) and pixel standard deviation std(i,j) are calculated. After traversing the entire image, the mean and standard deviation of all sub-block areas are averaged to finally obtain the mean Mean(img) and standard deviation STD(img) of the entire image. The corresponding image signal-to-noise ratio PSNR is the logarithm of the ratio of the image mean to the image standard deviation.

[0057] Image clarity is an indicator that describes the clarity of an image. The clearer the image, the higher the quality and the greater the clarity. The less clear (blurrier) the image, the lower the quality and the smaller the clarity. The specific algorithm is as follows:

[0058]

[0059] Where img(i,j) represents the image pixel value at the coordinate (i,j). For each pixel, the ratio of the pixel difference and distance between it and the surrounding 8 pixels is calculated. The entire image is traversed, and the clarity(i,j) at each pixel is summed and averaged to finally obtain the clarity description of the entire image, Clarity(img).

[0060] Image clarity can also be determined based on the average gradient. The average gradient refers to the average value of the grayscale change rate. It is used to indicate image clarity and reflects the rate of change in the contrast of tiny details in the image, that is, the rate of change in the density of the image in multiple dimensions. It represents the relative clarity of the image. The average gradient is the image clarity, which reflects the image's ability to express detail contrast. When the input is an RGB three-channel image, Gray = 0.1140*R+0.5870*G+0.2989*B is used to convert it to a grayscale image. When the input is a grayscale image, the calculation formula is expressed as:

[0061]

[0062] Among them, M×N represents the size of the image, Represents the horizontal gradient, Represents the vertical gradient.

[0063] Image contrast refers to the measurement of different brightness levels between the brightest white and the darkest black in the light and dark areas of an image, that is, the size of the grayscale contrast of an image. The larger the difference range, the greater the contrast, and the smaller the difference range, the smaller the contrast. The grayscale co-occurrence matrix is ​​usually used to describe the contrast.

[0064] The size of image information entropy reflects the amount of information carried by the image. Generally speaking, the larger the information entropy of an image, the richer its information and the better its quality. Information entropy can be used to compare the difference in information volume between different images, but it cannot be used as the only criterion for measuring image quality. When the input is an RGB three-channel image, Gray=0.1140*R+0.5870*G+0.2989*B is used to convert it into a grayscale image. When the input is a grayscale image, the specific algorithm process of information entropy is as follows:

[0065] Statistical grayscale probability: Assume that the maximum grayscale value of the image is M (8-bit image M = 255, 16-bit image M = 65535, etc.), then first count the number of pixels corresponding to each pixel value i from 0 to M histogram[i], and initialize the information entropy H = 0;

[0066] Calculate the ratio of the number of pixels corresponding to each pixel value to the total number of pixels, rat io(i) = histogram[i] / (H×W);

[0067] Information Entropy:

[0068] The range of the grayscale values ​​of the pixels included in the image is to count the grayscale values ​​of each pixel, and see the range of grayscale levels of the pixels in this image. If an image has a wide dynamic range, then its contrast will be higher and the visual effect will be clearer. If two adjacent grayscales are very close, it is not easy for the human eye to distinguish them clearly. In other words, the larger the dynamic range of the image, the larger the grayscale span, and the clearer the image resolution. When the input is an RGB three-channel image, Gray=0.1140*R+0.5870*G+0.2989*B is used to convert it to a grayscale image. When the input is a grayscale image, the specific algorithm calculation process is as follows:

[0069] Count the grayscale value of each pixel: find the maximum pixel value max and the minimum pixel value min;

[0070] The range of grayscale values ​​of pixels in the image is |max-min|.

[0071] The image brightness is determined based on the mean of the image. The larger the mean, the brighter the image, and vice versa. When the input is an RGB three-channel image, Gray=0.1140*R+0.5870*G+0.2989*B is used to convert it to a grayscale image. When the input is a grayscale image, the specific algorithm definition is: Input an image img with a height of H and a width of W, and the image mean is:

[0072]

[0073] Image pixels are determined based on the image standard deviation. The image standard deviation reflects the degree of dispersion of the image pixel value and the mean. The larger the standard deviation, the higher the image pixel. When the input is an RGB three-channel image, Gray=0.1140*R+0.5870*G+0.2989*B is used to convert it to a grayscale image. When the input is a grayscale image, the specific algorithm is defined as follows:

[0074] Input an image img with a height of H and a width of W, and calculate the image mean:

[0075]

[0076] The image standard deviation is expressed as:

[0077] Image resolution is the image size (or image dimension) set by a digital camera when shooting. Generally, a digital image of size is called image resolution. For the same actual physical scene, the larger the image size is set, the more pixels the image will have, and the clearer the image will be. Therefore, this indicator can measure the quality of the image.

[0078] The experimental target in the image is blocked, which means that the front-end camera is blocked by foreign objects due to external factors, causing the video or image main area to be partially or completely blocked, resulting in serious loss of image monitoring information. When the input is an RGB three-channel image, Gray=0.1140*R+0.5870*G+0.2989*B is used to convert it into a grayscale image. When the input is a grayscale image, the specific algorithm process is defined as follows:

[0079] Binarize the image img with an input size of H×W. The darker part is the background and the other part is the foreground. First, calculate the grayscale average of the entire image. Traverse the entire image, if the pixel value is greater than the average value, set it to 255; if the pixel value is less than the average value, set it to 0;

[0080] Using the binarized image, traverse each pixel i and perform connected area detection on the background area. Determine whether there is a point at the far left or the top of the four neighborhoods of pixel i. If there is no point at all, it indicates the beginning of a new area. If there is a point at the far left and no point at the top of the four neighborhoods of this point, mark this point with the value of the far left point; if there is no point at the far left and a point at the top of the four neighborhoods of this point, mark this point with the value of the top point. If there is a point at the far left and a point at the top of the four neighborhoods of this point, mark this point as the smallest marked point between the two and modify the large mark to a small mark;

[0081] Filter out the connected areas that meet this condition: the area contains at least 2 or more pixel coordinates at the upper, lower, left, and right boundaries of the image. If not, the experimental target is not occluded, the occlusion rate is 0, and the algorithm ends;

[0082] Find the largest connected area C among the selected connected areas. Determine the mean pixel value of area C in the original image. If it is 0 (completely black), it is an occluded area. Otherwise, the experimental target is not occluded, the occlusion rate is 0, and the algorithm ends.

[0083] The total number of pixels M in the maximum connected area C is obtained, and the ratio of M to the number of pixels H×W of the entire image, that is, M / (H×W), is the occlusion rate;

[0084] It should be noted that it is assumed that the area of ​​the largest connected area among all the connected areas found in the second step is S. Generally speaking, when the occluded area is large, the largest connected area should be the occluded area. Therefore, when M ≥ S, we believe that the image is likely to be occluded and further tracking and analysis is needed.

[0085] In some embodiments, the evaluation indicators corresponding to the video data evaluation indicator type include at least one of video signal-to-noise ratio, video clarity, video contrast, video information entropy, the range of grayscale values ​​of pixels included in the video, video brightness, video resolution, video pixels, video frame rate, video continuity, and video distortion.

[0086] Specifically, the video frame rate is defined as the number of frames displayed per second in the dynamic picture, which is used to measure the frequency of continuous appearance of the video signal. The unit is frame / second (fps). The corresponding frame rate of the video is finally read using the opencv library. First, the total number of video frames N and the total length of the video T are read, and the frame rate is defined as: N / T (fps).

[0087] Video continuity refers to the similarity between all two adjacent frames of a video. The higher the similarity, the smoother the video and the better the continuity. From the perspective of content, a video may contain multiple video clips with different contents. Therefore, we first find the changing boundaries of the video scene. For the images X and Y of the previous and next frames of the video, we take the area R of the center area of ​​the current frame image Y with a size of n×n (n=16) as the reference block, and perform a traversal search on the previous frame image X to find the most similar block area and calculate the similarity with the area.

[0088] The video distortion evaluation modeling statistics for the spatiotemporal natural scenes of the video, and comprehensively evaluates the degree of distortion in time and space. The specific algorithm process is as follows:

[0089] For a video input sequence containing M frames, the two adjacent frames are subtracted to obtain M-1 difference frames. Each difference frame is divided into multiple n×n image blocks. The two-dimensional DCT is applied to each n×n image block. The DCT coefficients of each image block from each difference frame are modeled as obeying the generalized Gaussian probability distribution. Assuming that the spatial size of each video frame is H×W, then each frame will have (H×W) / (n×n) DCT blocks, each DCT block contains n×n frequency coefficients, each of which will appear (H×W) / (n×n) times in each difference frame. Here n=5;

[0090] Next, the parameter density function is used to fit the histogram of each frequency coefficient in each differential frame.

[0091]

[0092] Where μ is the mean, γ is the shape parameter, and α and β are normalization and scale parameters:

[0093]

[0094] Where σ is the standard deviation and Γ is the original γ function

[0095]

[0096] Next, the DCT coefficients are fitted on each differential frame to obtain the 5×5 γ coefficient matrix corresponding to each differential frame.

[0097]

[0098] Determine an estimate of the corrected mean for the absolute values:

[0099]

[0100] Calculate the ratio:

[0101]

[0102] The optimal shape parameters obtained by the final fitting are:

[0103]

[0104] At this point, a 5×5 γ matrix corresponding to each differential frame is obtained through parameter fitting. The γ matrix is ​​divided into shape parameters of three frequency bands: low, medium, and high. The geometric mean of the shape parameters of each frequency band is calculated as:

[0105]

[0106] Where f∈{low,mid,high}. In each 5×5 size γ matrix,

[0107] {γ 12 , γ 13 , γ 21 , γ 22 , γ 23 , γ 31 , γ 32 , γ 33} is low frequency,

[0108] {γ 14 , γ 15 , γ 24 , γ 33 , γ 42 , γ 43 , γ 51 , γ 52} is the intermediate frequency,

[0109] {γ 25 , γ 34 , γ 35 , γ 44 , γ 45 , γ 53 , γ 54 , γ 55} is high frequency;

[0110] Finally the following spectral ratios can be calculated:

[0111]

[0112] Finally, the shape parameter spectral ratio obtained above is input into the SVR model trained on the video dataset to predict the quality score.

[0113] In some embodiments, based on the evaluation indicators corresponding to each evaluation indicator type, an evaluation algorithm corresponding to each evaluation indicator type can be generated, and each evaluation algorithm can be packaged and integrated so that the user can call it at any time. Specifically, the evaluation algorithms corresponding to the parameter data evaluation indicator type, the image data evaluation indicator type, and the video data evaluation indicator type can be written, and the different evaluation algorithms can be converted into Single ity images through the Docker image repository, and the administrator is provided with permission to import them into the corresponding database for storage. For example, the quality analysis and evaluation system algorithm is imported and stored in the image library.

[0114] It can be seen that the present invention can evaluate the quality of data more finely and accurately. By integrating the evaluation index types of multiple dimensions and the evaluation indexes corresponding to each evaluation index type, the product quality can be evaluated more comprehensively, and the credibility and reliability of data quality evaluation can be improved. The evaluation algorithm corresponding to each evaluation index type is encapsulated as a Docker image, and converted and uploaded through a Single ity image, which realizes the standardization and modularization of the evaluation algorithm corresponding to each evaluation index type, and improves the maintainability and scalability of the evaluation algorithm corresponding to each evaluation index type.

[0115] In some embodiments, the data quality assessment task may be performed in a manual / automatic scheduled manner.

[0116] Specifically, if the data quality evaluation task is performed in an automatic timing manner, a preset period can be set, and at each preset period, any file in the unselected state in at least one file to be analyzed is determined as any file, and any file is marked as selected. After that, steps S101-S104 can be continued. After determining that any file is abnormal, any file can be marked as unselected again.

[0117] If the data quality assessment task is performed manually, a data analysis instruction input by a user may be received. In response to the data analysis instruction, any file is determined in at least one file to be analyzed. Thereafter, steps S101-S104 may be continued.

[0118] The following describes in detail the execution of data quality assessment tasks in a manual / automatic timing manner with specific examples.

[0119] S11, according to different data quality analysis and evaluation needs, different working modes are adopted, including automatic timed tasks and manual tasks. Administrators can configure and start automatic tasks, and administrators and operators can configure and start manual tasks, which are divided into automatic working mode and manual working mode.

[0120] Among them, the quality analysis and evaluation modes corresponding to different data types are shown in Table 1 below:

[0121] Table 1

[0122]

[0123]

[0124] It should be noted that the manual working mode is also called the manual task-driven mode. According to the spacecraft identification, payload type, data level, data name (Chinese description, non-data file name), data acquisition start and end time, etc., select the data set to be evaluated, select the data quality evaluation process and start the quality evaluation workflow. The quality analysis and evaluation system collects the process information and algorithm results during the execution of the workflow and displays them in the user interface. The process of the manual working mode is as follows: Figure 2 shown.

[0125] The automatic working mode is also called automatic task drive. The process of the automatic working mode is as follows: Figure 3 As shown in the figure, there are two modes, data quality extraction mode and data quality analysis mode, which correspond to the extraction and calculation of science and application level 0 data quality information and the sampling and analysis of science and application advanced data products through algorithms. Among them, level 0 data is to unify the format of all downstream raw data and generate level 0 data products of various scientific data and engineering data.

[0126] S12, for all level 0 scientific and application data products, configure different types of level 0 data quality information extraction and calculation based on the automatic mode, set the execution time and execution cycle (by day, week, month, year), regularly parse the quality evaluation information in the file, save it in the data quality evaluation result table, and display it in the user interface. The level 0 data custom task process is as follows: Figure 4 shown.

[0127] S13, for all level 1 and level 2 science and application data products and level 0 science and application data products of the multifunctional optical cabin, configure the quality analysis and evaluation process of different levels and types of data, select the type and level of data products that need to be sampled, set the sampling ratio, execution time and execution cycle (daily, weekly, monthly, and annually), select the data quality evaluation process, and trigger the quality evaluation process regularly according to user settings. Collect data information (data list, data file storage address) within the execution cycle according to the sampling ratio, and start the quality evaluation process regularly. After the process is executed, save the quality evaluation results in the data quality evaluation result table and display them in the user interface. Among them, the data quality analysis task process is as follows: Figure 5 shown.

[0128] It should be noted that Level 1 data is based on each Level 0 data product. It further extracts payload observation / measurement data according to the data protocol format of each payload, extracts metadata based on the payload's scientific objectives, the types and characteristics of the observations / measurements performed, the configured observation / measurement instruments, and the association and continuity between the observations / measurements performed, and encapsulates the data and metadata into a file in a standard format for output, providing users with high-quality Level 1 data products for each payload. Level 2 data is data obtained by further producing and encapsulating the Level 1 data.

[0129] In some embodiments, it is also possible to provide monitoring functions for all automatic and manual task-driven quality evaluation processes, view the algorithm progress and status of the quality evaluation process through a visual interface, and display spacecraft identification, payload type, data level, data type, data acquisition start and end time, process name, algorithm index, process status (not started, ended), process start time, process end time and other information in a list form. Among them, the schematic diagram of the quality evaluation process monitoring page is as follows Figure 6 shown.

[0130] The beneficial effect of adopting the above further solution is that it realizes the automated execution of tasks and ensures the efficiency of quality evaluation tasks; the administrator can configure the specific parameters of the task as needed, such as execution cycle, sampling ratio, etc., to achieve refined management and control of data quality analysis and evaluation work; for data products that have not yet been evaluated or have already been evaluated, users can quickly locate the data set that needs to be evaluated and initiate the process by manual initiation, ensuring the flexibility of the quality evaluation work.

[0131] S102: Determine the evaluation indicator corresponding to the target evaluation indicator type as the target evaluation indicator, and obtain the actual evaluation range of the data included in any file under each target evaluation indicator.

[0132] Wherein, the target evaluation indicator type corresponds to multiple evaluation indicators, and each evaluation indicator corresponding to the target evaluation indicator type can be determined as the target evaluation indicator. Afterwards, based on the data included in any one of the files, the actual evaluation range of the data included in any one of the files under each target evaluation indicator can be calculated.

[0133] S103: For any target evaluation indicator, based on the fact that the actual evaluation range of any file under any target evaluation indicator does not completely fall within the preset evaluation range of any target evaluation indicator, determining that any target evaluation indicator corresponding to any file is abnormal.

[0134] In some embodiments, information associated with the abnormality of any target evaluation index corresponding to any file is displayed in the early warning interface. The information associated with the abnormality of any target evaluation index corresponding to any file can be displayed in the early warning interface in at least one form of text, table, and image.

[0135] S104: Determine that any file is abnormal based on the ratio of the number of abnormal target evaluation indicators corresponding to any file to the total number of target evaluation indicators being greater than an abnormal ratio threshold.

[0136] In combination with S103 and S104, it can be known that the present application can configure abnormal indicators of data products and monitor and track data with abnormal corresponding evaluation indicators. For example, a configuration page can be displayed, and the configuration page is used to support the user to set at least one evaluation indicator type. In response to the user input of the selection operation of the first evaluation indicator type in at least one evaluation indicator type, a list of evaluation indicators corresponding to the first evaluation indicator type is displayed. In response to the user input of the selection operation of the first evaluation indicator among the evaluation indicators included in the list, an input box for adjusting the preset evaluation range of the first evaluation indicator is displayed. In response to the user entering the target preset evaluation range in the input box, the preset evaluation range of the first evaluation indicator is adjusted to the target preset evaluation range.

[0137] Furthermore, the abnormal indicators of the configuration data product in the present invention and the monitoring and tracking of the abnormal data of the corresponding evaluation indicators are described in detail below with reference to specific examples.

[0138] The present invention can judge whether the current data product file is abnormal based on the evaluation result, and needs to configure the data product abnormality index. The data product abnormality index is divided into index abnormality configuration and file abnormality configuration, and supports automatic configuration and manual configuration.

[0139] Among them, the abnormal indicator configuration is the configuration to determine whether a certain indicator of a data product is abnormal. The content that needs to be configured includes data products, file types, indicator classifications, indicator names, and normal value ranges of indicators. When the indicator value calculated by the algorithm exceeds the normal value range, the current indicator of the data product is considered abnormal. The automatic configuration of abnormal indicators is a scientific calculation method to obtain the normal value range of indicators. Through the method of scheduled tasks, the historical quality analysis and evaluation results of the current data product are regularly scanned, and the normal value range is calculated. This method can effectively improve the configuration efficiency. At the same time, a visual interface is provided to support manual adjustment of abnormal configuration of indicators.

[0140] The automatic calculation method of index abnormality is to take the clarity index of image type files as an example, first calculate the N clarity parameters x of image type files under the current product. i Perform sum calculation and find the average value of clarity According to the average and the clarity parameter x i Calculate the standard deviation s of the image file clarity of the current product N ; According to the three standard deviations of the normal distribution, 99% of the clarity parameters are normal, and the rest are abnormal parameters. The calculation formula is as follows:

[0141]

[0142] File abnormality configuration is used to determine whether the current data product file is abnormal. A single indicator abnormality cannot be considered as a file abnormality. The content that needs to be configured includes data products, file types, and abnormal indicator ratios. When the number of abnormal indicators / total number of indicators is greater than the value of the abnormal indicator ratio, the current data product file is considered abnormal. For newly added data product types, the abnormal ratio defaults to 1 / 3, and a visual interface is provided to support the adjustment of file abnormality configuration.

[0143] In some embodiments, users can configure the file anomaly index of data products through a visual interface. They need to add product name, file classification, anomaly index ratio, etc., and save them in the anomaly configuration table for anomaly analysis of data quality evaluation results. It supports the configuration of different file types for a type of product. The specific configuration steps are as follows:

[0144] 1. First, select data products and product categories;

[0145] 2. Enter the value of the abnormal indicator ratio;

[0146] File configuration is performed for all products under the selected compartment, where the indicator classification includes data verification, parameters, images, videos, spectra and other evaluation types, and the numerical range of abnormal indicator proportion is between 0-100%.

[0147] Users can also configure according to the indicator abnormalities of product files. The content that needs to be added includes product name, file type, indicator classification, indicator name, normal range of indicator, etc. The data is saved in the indicator abnormality configuration table.

[0148] Supports configuring different parameters for the same type of files. The specific steps are as follows:

[0149] 1. First, select the data product;

[0150] 2. For parameter data, you need to select parameter indicators. You can perform fuzzy search on parameter indicator names and support selecting multiple items at the same time.

[0151] 3. Click the data type drop-down box and select the corresponding data type;

[0152] 4. Click the indicator category drop-down box and select the indicator category (including image evaluation indicator type, video evaluation indicator type, spectral evaluation indicator type, etc.);

[0153] 5. Click the drop-down box of the evaluation indicator type classification, select the indicator name (supports selecting multiple indicators at the same time), and set the preset evaluation range of each indicator selected;

[0154] like Figure 7 As shown, the results of the data product after the algorithm process will be compared with the normal range of the indicators. The results within the normal indicator range are normal results, and those outside the normal indicator range are abnormal results. Then, according to the abnormal indicator ratio of the product file = the number of abnormal indicators / the total number of indicators, if the abnormal indicator ratio exceeds the abnormal indicator ratio data in the abnormal configuration table, it is judged as an abnormal file.

[0155] It can be seen that the present invention can perform detailed exception configuration for multiple indicators of data products, thereby improving the accuracy and comprehensiveness of exception analysis of data quality evaluation results; users can quickly understand the overall status and specific problems of data quality based on key information such as abnormal data type, exception generation time, evaluation indicators, etc., so as to make timely and effective response measures.

[0156] In some embodiments, information associated with any file anomaly may be displayed in the early warning interface, wherein the information associated with any file anomaly may be displayed in the early warning interface in at least one form of text, table, or image.

[0157] Specifically, according to the data quality evaluation analysis results, a quality analysis evaluation statistical chart and an analysis report can be generated, and the quality analysis evaluation statistical chart and the analysis report can be displayed in the early warning interface.

[0158] Among them, the early warning interface will collect the abnormal information of each analysis result involved in the quality analysis and evaluation by default, and display it to the user in the form of a list. The displayed data information includes abnormal data type, abnormal generation time, evaluation index type (parameter data evaluation index type, image data evaluation index type, video data evaluation index type, etc.), abnormal description, etc., and supports conditional filtering of abnormal information files, data types, completion time, etc.; it also provides abnormal information statistical bar charts at the product file level and indicator level, which can be filtered according to time period, product, file type, etc., to quickly locate abnormal files and abnormal indicators. The abnormal result query diagram is shown as follows: Figure 8 shown.

[0159] See also Fig. 9The early warning interface can also collect the results of indicator analysis and evaluation of all data. The result query statistics are in the form of tables, line graphs, bar graphs, pie charts, radar charts, funnel charts and other advanced two-dimensional data visualization forms. The results of quality analysis and evaluation of various scientific experimental data are statistically displayed according to experimental load, data type, data result status (completed, in progress, not started and failed), indicator classification, result statistics, and evaluation indicators.

[0160] In some embodiments, quality anomaly tracking analysis, quality statistical analysis and quality evaluation task summary analysis can also be performed based on the quality evaluation results to form the following three types of reports: quality analysis and evaluation anomaly analysis report (abnormal report for short), quality analysis and evaluation statistical report (statistical report for short), quality analysis and evaluation summary report (summary report for short). The present invention can also provide Fig.10 The page shown can be used to search and retrieve quality analysis and evaluation reports, and can also provide Fig.11 The preview of the supported quality analysis and evaluation reports and the page where the selected quality analysis and evaluation reports can be downloaded to the local computer are shown.

[0161] It can be seen that the present invention can intuitively visualize data, making complex data quality analysis intuitive and easy to understand; assisting scientific management and decision-making, and improving the platform operation and maintenance capabilities.

[0162] The present invention also provides a system for supporting the analysis and evaluation of space science and application data quality, see Fig.12 ,The system includes a user authentication and log management module, a ,quality analysis and evaluation management module, a quality evaluation exception and ,analysis module, and a result and report management module.

[0163] Among them, the user authentication and log management module is used for user authority authentication of the quality analysis and evaluation system, and for providing users with functions such as log query and log export.

[0164] The data quality analysis and evaluation module is used to encapsulate and integrate the aforementioned evaluation algorithms, build and start the quality evaluation process, analyze abnormal data, generate result statistics and reports. It is the main module supporting the space science and application data quality analysis and evaluation system.

[0165] The quality analysis and evaluation management module is used to encapsulate and integrate the aforementioned evaluation algorithms, build quality evaluation processes, automatically configure tasks, manually initiate quality evaluation processes, and monitor and query quality evaluation processes.

[0166] Specifically, the quality analysis and evaluation management module is used to determine the target evaluation indicator type in at least one evaluation indicator type for any file in at least one file to be analyzed, based on the data type of the data included in any file. Among them, different evaluation indicator types correspond to different evaluation indicators, and the evaluation indicator type is used to characterize the preset evaluation range of data of the corresponding data type. The evaluation indicator corresponding to the target evaluation indicator type is determined as the target evaluation indicator, and the actual evaluation range of the data included in any file under each target evaluation indicator is obtained. For any target evaluation indicator, based on the fact that the actual evaluation range of any file under any target evaluation indicator does not completely fall into the preset evaluation range of any target evaluation indicator, it is determined that any target evaluation indicator corresponding to any file is abnormal. Based on the ratio of the number of abnormal target evaluation indicators corresponding to any file to the total number of target evaluation indicators being greater than the abnormal ratio threshold, it is determined that any file is abnormal.

[0167] The quality evaluation anomaly and analysis module is used to configure data product anomaly indicators, query the details of data products with abnormal indicators, and track and trace the source.

[0168] Specifically, the quality evaluation anomaly and analysis module is used to display a configuration page, and the configuration page is used to support the user to set at least one evaluation indicator type. In response to the user inputting a selection operation of a first evaluation indicator type in at least one evaluation indicator type, a list of evaluation indicators corresponding to the first evaluation indicator type is displayed. In response to the user inputting a selection operation of a first evaluation indicator in each evaluation indicator included in the list, an input box for adjusting a preset evaluation range of the first evaluation indicator is displayed. In response to the user inputting a target preset evaluation range in the input box, the preset evaluation range of the first evaluation indicator is adjusted to the target preset evaluation range.

[0169] The result and report management module is used for quality evaluation result query and statistics, report configuration generation and report query.

[0170] Specifically, the result and report management module is used to display an early warning interface, which includes information associated with any abnormality of any target evaluation indicator corresponding to any file, and information associated with any abnormality of any file. The information associated with any abnormality of any target evaluation indicator corresponding to any file and the information associated with any abnormality of any file are displayed in the early warning interface in at least one form of text, table, and image.

[0171] In some embodiments, data information (data list, data file storage address) within the execution cycle can also be collected according to the sampling task, and the quality evaluation process can be started regularly. Specifically, by reading various types of space science experiment payload data, the data information (data list, data file storage address) and the process template can be sent to the Kafka message queue. The system provided by the present invention can monitor Kafka messages at the same time, obtain data product information to assign different quality analysis and evaluation processes, and start the process. After the quality evaluation process is executed, the algorithm program needs to send the calculation results to the Kafka message queue. The system provided by the present invention can obtain the information in the message queue, parse the data to obtain the quality evaluation results and store them in the warehouse. For data files that fail the quality evaluation in the process, the failed data file information and its result status are recorded in the database, and the failed data can be selected again to re-initiate the process.

[0172] The system provided in the embodiment of the present application can also be connected to a space science experiment payload data archive management module. The space science experiment payload data archive management module is used to store and manage scientific and application data. The system provided in the embodiment of the present application can obtain data directories, data addresses, data details, etc. in the space science experiment payload data archive management module.

[0173] It can be seen that the present invention improves the stability of the quality analysis and evaluation system; records the data files that failed the quality evaluation in the process, allowing the user or administrator to select these failed data again in subsequent operations to re-initiate the process, thereby improving the reliability and integrity of the data quality evaluation.

[0174] In order to better illustrate and understand the principle of the method provided by the present invention, the scheme of the present invention is described below in conjunction with an optional specific embodiment. It should be noted that the specific implementation of each step in the specific embodiment should not be understood as a limitation to the scheme of the present invention. On the basis of the principle of the scheme provided by the present invention, other implementations that can be thought of by those skilled in the art should also be considered within the scope of protection of the present invention.

[0175] In this embodiment, the scenario of analysis and evaluation of space science and application level 1 parameter type data products is taken as a specific embodiment. In this embodiment, the operation is implemented according to the following steps:

[0176] S1, parameter evaluation algorithm program is written and packaged for integration.

[0177] The parameter evaluation algorithm receives the data file (csv file) in the data archiving software, acquires data, and extracts all different types of parameters recorded in the data file, including temperature, humidity, liquid / gas composition, speed, microgravity, light intensity, etc. For each type of parameter, the reasonable value ratio, maximum and minimum values, mean, median, standard deviation, etc. are calculated, and the calculation results are saved in the quality evaluation result table.

[0178] The parameter evaluation algorithm is a Python program, which needs to be packaged in a Docker image. The algorithm program needs to be configured, and the configuration file and the algorithm image file need to be packaged and uploaded. Fig.13 As shown. The algorithm program needs to send three types of messages, such as control, progress, and details, to the Kafka message of the quality analysis and evaluation system to remind the start of the next step. After the algorithm program is executed, the algorithm result is placed in the detailed message and sent to the Kafka message. The quality analysis and evaluation system obtains the algorithm result and saves it in the database.

[0179] S2, configure the data quality analysis task of level 1 parameter type data product.

[0180] In the quality analysis and evaluation system, users select the level 1 parameter type data product that needs to be sampled, set the sampling ratio, execution time and execution cycle (daily, weekly, monthly, and annually), and select the data quality evaluation process. The diagram of the data quality analysis task configuration interface is as follows: Fig.14 shown.

[0181] Data quality analysis task extraction rules: The time period set by the automatic task is used as the starting point for data extraction, and the time period is used as the data base for random average extraction for different data types. For example, by reading the space science experiment payload data, there are 100 data in the three-day period from January 26, 2022 to January 29, 2022 under the Level 1 parameter type data product. At a ratio of 20%, 20 data are randomly extracted from these 100 data without distinction. The number of random inspections = extraction percentage * the total number of data in the data directory of this type.

[0182] Based on the sampling task configuration, the quality analysis and evaluation system will automatically create corresponding scheduled tasks. The system will regularly trigger the quality evaluation process of the first-level parameter data product according to the user settings, collect data information (data list, data file storage address) within the execution cycle according to the sampling ratio, and regularly send data information and process templates to the Kafka message queue. The quality analysis and evaluation system listens to Kafka messages and starts the quality evaluation process.

[0183] S3, configure the abnormal indicator range of level 1 parameter type data products.

[0184] Parameter type data products (csv files) may contain different types of parameters, such as temperature, humidity, liquid / gas composition, speed, microgravity, light intensity, etc. The calculation of each type of parameter includes six types of indicator data: reasonable value proportion, maximum and minimum values, mean, median, and standard deviation. For parameter type data, the main focus is on the reasonable value proportion indicator. Users select the level 1 parameter type data product that needs to be sampled in the quality analysis and evaluation system, and configure the normal value range of the reasonable value proportion indicator of each parameter. The abnormal indicator configuration diagram is as follows: Fig.15 At the same time, in order to improve the operational efficiency of administrators, the quality analysis and evaluation system also supports batch exception setting function.

[0185] S4, quality evaluation process monitoring.

[0186] Through the quality evaluation process monitoring, you can view the algorithm progress and status of the quality evaluation process, quality evaluation process name, algorithm indicators, process status (not started, ended) and other information. The quality evaluation process monitoring page is as follows: Figure 7 shown.

[0187] S5, quality evaluation results are generated.

[0188] After the quality evaluation process is completed, the algorithm results will be placed in a detailed message and sent to the Kafka message. The quality analysis and evaluation system obtains the algorithm results by listening to the Kafka message and parses the parameter index value. Obtain the reasonable value ratio index value and compare it with the normal range of the index. If the index value exceeds the normal range of the index, it proves that the current parameter is abnormal, and if the index value is within the normal range, it proves that the current parameter is normal. Calculate the parameter abnormality index ratio of the product: parameter abnormality index ratio = number of abnormal parameters / total number of parameters, compare the calculated parameter abnormality ratio with the corresponding abnormal index ratio in the index abnormality configuration table, if it exceeds, the current file is abnormal.

[0189] After obtaining the quality evaluation results of the product files, they are stored in the database based on MySQL.

[0190] S6, quality evaluation result statistics and report generation.

[0191] The quality evaluation result statistics interface uses line graphs, bar graphs, pie charts, etc. to display the number of quality evaluations of the current level 1 parameter type data products, the completion rate, and the normal, abnormal, and evaluation failure status of product quality over time.

[0192] It supports the generation of exception analysis reports and quality analysis and evaluation statistical reports for the current data products. There are two types of reports: one is the incremental report, which generates a statistical report within a period of time; the other is the cumulative mode, which generates a statistical report by summarizing all quality evaluation results.

[0193] Among them, the above-mentioned system for supporting space science and application data quality analysis and evaluation can be a computer program (including program code) running in a computer device. For example, the system for supporting space science and application data quality analysis and evaluation is an application software; the system can be used to execute the corresponding steps in the method provided in the embodiment of the present invention.

[0194] In some schemes, multiple embodiments of the present application can be combined, and the combined scheme can be implemented. Optionally, some operations in the process of each method embodiment are optionally combined, and / or the order of some operations is optionally changed. In addition, the execution order between the steps of each process is only exemplary and does not constitute a restriction on the execution order between the steps. There can also be other execution orders between the steps. It is not intended to indicate that the execution order is the only order in which these operations can be performed. A person of ordinary skill in the art will think of a variety of ways to reorder the operations described herein. In addition, it should be noted that the process details involved in a certain embodiment of this article are also applicable to other embodiments in a similar manner, or different embodiments can be used in combination.

[0195] In addition, some steps in the method embodiment may be equivalently replaced by other possible steps. Alternatively, some steps in the method embodiment may be optional and may be deleted in certain usage scenarios. Alternatively, other possible steps may be added to the method embodiment. Furthermore, the method embodiments may be implemented separately or in combination.

[0196] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0197] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0198] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0199] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0200] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for supporting data quality analysis and evaluation for space science and applications, characterized in that: include: For any file in at least one file to be analyzed, based on the data type of data included in the any file, determine a target evaluation indicator type in at least one evaluation indicator type; wherein different evaluation indicator types correspond to different evaluation indicators; and the evaluation indicators are used to characterize a preset evaluation range of data of the corresponding data type; Determine the evaluation indicator corresponding to the target evaluation indicator type as the target evaluation indicator, and obtain the actual evaluation range of the data included in any one of the files under each of the target evaluation indicators; For any target evaluation indicator, based on the fact that the actual evaluation range of any file under any target evaluation indicator does not completely fall within the preset evaluation range of any target evaluation indicator, determining that any target evaluation indicator corresponding to any file is abnormal; Based on the fact that the ratio of the number of abnormal target evaluation indicators corresponding to the any file to the total number of the target evaluation indicators is greater than an abnormal proportion threshold, it is determined that the any file is abnormal.

2. The method according to claim 1, characterized in that The at least one evaluation index type includes a parameter data evaluation index type, an image data evaluation index type, and a video data evaluation index type; the determining of a target evaluation index type in at least one evaluation index type based on the data type of the data included in any one of the files includes: In the case where any of the files includes data of a parameter type or a binary type, determining the parameter type of data evaluation index type as the target evaluation index type; In the case that any of the files includes data of an image data type or a spectral data type, determining the image data evaluation index type as the target evaluation index type; In the case that any of the files includes data of a video data type, the video data evaluation index type is determined as the target evaluation index type.

3. The method according to claim 2, characterized in that The evaluation index corresponding to the parameter data evaluation index type includes at least one of a data mean, a data median, a data standard deviation, and a reasonable data ratio; wherein the reasonable data ratio is a ratio of a first value to a second value; the first value is the number of sampled data falling within a preset minimum reasonable value range and a preset maximum reasonable value range; the second value is the total number of all sampled data; The evaluation index corresponding to the image data evaluation index type includes at least one of image signal-to-noise ratio, image clarity, image contrast, image information entropy, a range of grayscale values ​​of pixels included in the image, image brightness, image resolution, image pixels, and an occlusion rate of an experimental target in the image; The evaluation indicators corresponding to the video data evaluation indicator type include at least one of video signal-to-noise ratio, video clarity, video contrast, video information entropy, the range of grayscale values ​​of pixels included in the video, video brightness, video resolution, video pixels, video frame rate, video continuity, and video distortion; wherein the video continuity is associated with the similarity between adjacent frame images in the video.

4. The method according to claim 2, characterized in that: Based on the data type of the data included in any one of the files, before determining the target evaluation indicator type in at least one evaluation indicator type, the method further includes: At every preset period, any file in the at least one file to be analyzed that is in an unselected state is determined as the any file, and the any file is marked as being in a selected state.

5. The method according to claim 4, characterized in that After determining that any of the files is abnormal, the method further includes: Mark any of the files as unselected.

6. The method according to claim 2, characterized in that Based on the data type of the data included in any one of the files, before determining the target evaluation indicator type in at least one evaluation indicator type, the method further includes: Receiving data analysis instructions input by a user; In response to the data analysis instruction, the any one file is determined in the at least one file to be analyzed.

7. The method according to claim 1, characterized in that After determining that any target evaluation indicator corresponding to any file is abnormal, the method further includes: Displaying information associated with the abnormality of any target evaluation indicator corresponding to any file in the early warning interface; After determining that any of the files is abnormal, the method further includes: Displaying information associated with any of the file anomalies in the warning interface; Among them, the information associated with the abnormality of any target evaluation indicator corresponding to any file and the information associated with the abnormality of any file are displayed in the early warning interface in at least one form of text, table, and image.

8. The method according to claim 1, characterized in that Also includes: Displaying a configuration page; the configuration page is used to support the user to set the at least one evaluation indicator type; In response to a user input selecting a first evaluation indicator type among the at least one evaluation indicator type, displaying a list of evaluation indicators corresponding to the first evaluation indicator type; In response to a user input of selecting a first evaluation indicator among the evaluation indicators included in the list, an input box for adjusting a preset evaluation range of the first evaluation indicator is displayed; In response to the user inputting a target preset evaluation range in the input box, the preset evaluation range of the first evaluation indicator is adjusted to the target preset evaluation range.

9. A system for supporting data quality analysis and evaluation for space science and applications, characterized in that: include: Quality analysis and evaluation management module; the quality analysis and evaluation management module is used to: For any file in at least one file to be analyzed, based on the data type of data included in the any file, determine a target evaluation indicator type in at least one evaluation indicator type; wherein different evaluation indicator types correspond to different evaluation indicators; and the evaluation indicators are used to characterize a preset evaluation range of data of the corresponding data type; Determine the evaluation indicator corresponding to the target evaluation indicator type as the target evaluation indicator, and obtain the actual evaluation range of the data included in any one of the files under each of the target evaluation indicators; For any target evaluation indicator, based on the fact that the actual evaluation range of any file under any target evaluation indicator does not completely fall within the preset evaluation range of any target evaluation indicator, determining that any target evaluation indicator corresponding to any file is abnormal; Based on the fact that the ratio of the number of abnormal target evaluation indicators corresponding to the any file to the total number of the target evaluation indicators is greater than an abnormal proportion threshold, it is determined that the any file is abnormal.

10. The system according to claim 9, characterized in that Also includes: Quality assessment anomaly and analysis module, result and report management module; among them, The quality assessment anomaly and analysis module is used to: Displaying a configuration page; the configuration page is used to support the user to set the at least one evaluation indicator type; In response to a user input selecting a first evaluation indicator type among the at least one evaluation indicator type, displaying a list of evaluation indicators corresponding to the first evaluation indicator type; In response to a user input of selecting a first evaluation indicator among the evaluation indicators included in the list, an input box for adjusting a preset evaluation range of the first evaluation indicator is displayed; In response to a user inputting a target preset evaluation range in the input box, adjusting the preset evaluation range of the first evaluation indicator to the target preset evaluation range; The result and report management module is used to: Display a warning interface; the warning interface includes information associated with the abnormality of any target evaluation indicator corresponding to any file, and information associated with the abnormality of any file; wherein the information associated with the abnormality of any target evaluation indicator corresponding to any file and the information associated with the abnormality of any file are displayed in the warning interface in at least one form of text, table, and image.

Images