Middle platform data index positioning and retrieval method

Abstract

The present invention relates to the field of data processing, and specifically to a method for indexing, locating, and retrieving data on a middle platform. Based on the characteristics of data uploaded to the middle platform by all connected data source ends, the storage status of the uploaded data on the middle platform is adjusted, and based on the data inventory status of all storage intervals in the middle platform, the retrieval priority is determined to achieve orderly retrieval of the middle platform; based on the target data attribute information and retrieval priority that the user end needs to query, the query path is determined, so that the storage interval is marked for data browsing, thereby improving the traceability of data retrieval on the middle platform; when the target data is successfully retrieved, the retrieval result data packet is returned to the user end based on the distribution status of the target data in the middle platform, ensuring that the data in the middle platform is fully acquired; when the target data is not successfully retrieved, the data storage change attribute information of the middle platform is predicted, and data retrieval is performed again, which can avoid repeated and invalid retrieval of the middle platform and improve the data retrieval efficiency of the middle platform.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for indexing, locating and retrieving data from a middleware. Background Art

[0002] With the development of the industry, industrial data has continued to increase and expand, thus forming industrial big data. Industrial big data plays a vital role in industrial research. In order to achieve the comprehensive and effective utilization of industrial big data, it is necessary to save industrial big data through the middle-office system. This not only allows for the classification and storage of industrial big data, but also facilitates the sharing and transmission of industrial big data between the front-end and back-end systems connected to the middle-office system, thereby improving the utilization efficiency of industrial big data. The middle-office system has the advantages of flexibility and efficiency in batch data query. However, when individual users initiate data query requests to the middle-office system, the middle-office system usually needs to search the massive internal industrial big data. This not only consumes the computing power resources of the middle-office system, but also reduces the efficiency of data query. It is prone to problems such as data omission and duplicate queries. It cannot conduct orderly and accurate retrieval of the big data within the middle-office system, which reduces the data retrieval efficiency of the middle-office system. Summary of the Invention

[0003] In response to the defects of the existing technology, the present invention provides a data index positioning and retrieval method for the middle platform. Based on the uploaded data characteristics of all connected data source ends to the middle platform, the storage status of the uploaded data by the middle platform is adjusted, and based on the data inventory status of all storage intervals in the middle platform, the retrieval priority is determined to achieve orderly retrieval of the middle platform; based on the target data attribute information and retrieval priority that the user end needs to query, the query path is determined, so as to mark the storage interval for data browsing and improve the traceability of the data retrieval of the middle platform; when the target data is successfully retrieved, the retrieval result data packet is returned to the user end based on the distribution status of the target data in the middle platform to ensure that the data in the middle platform is fully acquired; when the target data is not successfully retrieved, the data storage change attribute information of the middle platform is predicted, and the data is retrieved again, which can avoid repeated and invalid retrieval of the middle platform, and perform orderly and accurate retrieval and search of the big data in the middle platform, thereby improving the data retrieval efficiency of the middle platform.

[0004] The present invention provides a method for indexing, locating, and retrieving data from a central platform, comprising the following steps:

[0005] Step S1: Based on the communication log of the middle station, determine the characteristics of the uploaded data of all connected data source terminals to the middle station; based on the uploaded data characteristics, adjust the storage status of the uploaded data by the middle station; based on the data inventory status of all storage intervals in the middle station, determine the retrieval priority of all storage intervals;

[0006] Step S2: determining target data attribute information that the user needs to query based on the query request of the user terminal; determining a query path for a corresponding storage interval based on the target data attribute information and the search priority, thereby marking the storage interval for data browsing;

[0007] Step S3: Based on the data browsing mark, determine whether the target data that the user terminal needs to query is successfully retrieved; if the retrieval is successful, generate a retrieval result data packet based on the distribution status of the target data in the middle platform and return it to the user terminal;

[0008] Step S4: When the retrieval fails, the data storage change attribute information of the middle station is predicted based on the uploaded data characteristics; based on the data storage change attribute information and the target data attribute information, data retrieval is performed again to return the retrieval results to the user end.

[0009] In one embodiment disclosed in the present application, in step S1, based on the communication log of the middle station, the characteristics of the uploaded data of all connected data source terminals to the middle station are determined, including:

[0010] Based on the gateway traffic change information in the network where the middle station is located, determine the active gateway in the network; monitor the active gateway to determine the identity information of all data source terminals connected to the active gateway;

[0011] Based on the identity information, the data upload records of the data source end to the middle station are filtered from the communication log of the middle station; based on the data upload records, the type and quantity of upload data from the data source end to the middle station are determined, which are used as the uploaded data features.

[0012] In one embodiment disclosed in the present application, in step S1, adjusting the storage status of the uploaded data by the middle platform based on the uploaded data characteristics includes:

[0013] Obtaining data storage history records of all storage intervals in the middle platform, analyzing the data storage history records, and obtaining the type and quantity of data stored in each of the storage intervals in the middle platform;

[0014] Compare the data type and quantity uploaded by the data source end to the middle platform with the data type and quantity stored in each of all storage intervals to determine the storage interval that matches the uploaded data of the data source end;

[0015] Based on the data upload rate of the data source, the data storage transmission bandwidth allocated by the middle platform to the matching storage interval is adjusted.

[0016] In one embodiment disclosed in the present application, in step S1, based on the data inventory status of all storage intervals in the middle platform, determining the search priority of all storage intervals includes:

[0017] Obtaining data storage capacity and data semantic information for each of the storage intervals in the middle platform, and determining content duplication characteristics of the data stored in each storage interval based on the data storage capacity and the data semantic information; wherein the content duplication characteristics refer to the proportion of data with identical or similar content in each storage interval;

[0018] Based on the content repetition feature, a search priority for all storage intervals is determined; wherein the search priority includes a search order for all storage intervals and a search duration for each storage interval.

[0019] In one embodiment disclosed in the present application, in step S2, based on the query request of the user terminal, determining the target data attribute information that the user terminal needs to query includes:

[0020] Based on the identity information of the user terminal requesting access to the middle station, determine the data search history of the user terminal on the middle station; based on the data search history, determine the search time information of the user terminal on the middle station, so as to judge whether the user terminal has the search permission;

[0021] When the user terminal has the retrieval authority, the query request of the user terminal is parsed to obtain the target data index information that the user terminal needs to query, which is used as the target data attribute information; wherein, the target data index information includes the keyword semantic information of the target data that the user terminal needs to query.

[0022] In one embodiment disclosed in the present application, in step S2, based on the target data attribute information and the search priority, a query path for a corresponding storage interval is determined, thereby marking the storage interval for data browsing, including:

[0023] Comparing the target data index information with the storage directories of all storage intervals in the middle platform to screen out several target storage intervals; determining the search order of all target storage intervals and the search time for each target storage interval based on the search priority;

[0024] Based on the search order of all target storage intervals and the search time of each target storage interval, a query path for all target storage intervals is determined; based on the query path, the browsed data in the target storage interval is identified, so as to add an identification code to the data that matches the keyword semantic information.

[0025] In one embodiment disclosed in the present application, in step S3, judging whether the target data to be queried by the user terminal is successfully retrieved based on the data browsing mark includes:

[0026] The creation time of the data with the identification code added in the storage interval is compared with the preset time range. If the creation time is within the preset time range, it is determined that the target data that the user terminal needs to query is successfully retrieved; otherwise, it is determined that the target data that the user terminal needs to query is not successfully retrieved.

[0027] In one embodiment disclosed in the present application, in step S3, when the search is successful, a search result data packet is generated based on the distribution status of the target data in the middle platform and returned to the user terminal, including:

[0028] When the retrieval is successful, the data of the corresponding storage intervals is copied, extracted, integrated and compressed based on the distribution position and distribution quantity of the target data in all storage intervals in the middle platform, a retrieval result data packet is generated, and the retrieval result data packet is returned to the user end.

[0029] In one embodiment disclosed in the present application, in step S4, when the retrieval fails, the data storage change attribute information of the middle platform is predicted based on the uploaded data characteristics, including:

[0030] When the retrieval fails, the type and amount of data that the data source will upload to the middle platform in the future are predicted based on the characteristics of the uploaded data; based on the type and amount of future uploaded data, the data storage change attribute information of all storage intervals in the middle platform is predicted; wherein, the data storage change attribute information includes the increased data volume of corresponding type data of all storage intervals in the middle platform in the future time period.

[0031] In one embodiment disclosed in the present application, in step S4, data retrieval is performed again based on the data storage change attribute information and the target data attribute information, thereby returning the retrieval result to the user terminal, including:

[0032] Comparing the increased data volume with a preset data volume threshold, and if the increased data volume exceeds the preset data volume threshold, determining that data retrieval is allowed again; otherwise, determining that data retrieval is not allowed again;

[0033] When data retrieval is allowed again, the newly added data in all storage intervals in the middle platform are retrieved based on the target data attribute information, and the retrieval results are returned to the user end.

[0034] Compared with the existing technology, the data index positioning and retrieval method of the middle platform is based on the uploaded data characteristics of all connected data source ends to the middle platform, adjusts the storage status of the uploaded data by the middle platform, and determines the retrieval priority based on the data inventory status of all storage intervals in the middle platform to achieve orderly retrieval of the middle platform; based on the target data attribute information and retrieval priority that the user end needs to query, the query path is determined to mark the storage interval for data browsing, thereby improving the traceability of the data retrieval of the middle platform; when the target data is successfully retrieved, the retrieval result data packet is returned to the user end based on the distribution status of the target data in the middle platform to ensure comprehensive acquisition of the data in the middle platform; when the target data is not successfully retrieved, the data storage change attribute information of the middle platform is predicted, and data retrieval is performed again, which can avoid repeated and invalid retrieval of the middle platform, and perform orderly and accurate retrieval and search of the big data in the middle platform, thereby improving the data retrieval efficiency of the middle platform.

[0035] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0036] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 A flow chart of the middle platform data index positioning and retrieval method provided by the present invention. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0040] See Figure 1 , is a flow chart of a method for indexing, locating, and retrieving data from a middle platform provided in an embodiment of the present invention. The method for indexing, locating, and retrieving data from a middle platform includes:

[0041] Step S1: Based on the communication log of the middle station, determine the characteristics of the uploaded data from all connected data source terminals to the middle station; based on the uploaded data characteristics, adjust the storage status of the uploaded data by the middle station; based on the data inventory status of all storage intervals in the middle station, determine the retrieval priority of all storage intervals;

[0042] Step S2: Based on the query request of the user terminal, determine the target data attribute information that the user terminal needs to query; based on the target data attribute information and the search priority, determine the query path for the corresponding storage interval, and mark the storage interval for data browsing;

[0043] Step S3: Based on the data browsing mark, determine whether the target data that the user needs to query is successfully retrieved; if the retrieval is successful, generate a retrieval result data packet based on the distribution status of the target data in the middle platform and return it to the user;

[0044] Step S4: When the retrieval fails, the data storage change attribute information of the middle platform is predicted based on the uploaded data characteristics; based on the data storage change attribute information and the target data attribute information, data retrieval is performed again to return the retrieval results to the user end.

[0045] The beneficial effects of the above technical solution are as follows: the data index positioning and retrieval method of the middle platform adjusts the storage status of the uploaded data of the middle platform based on the uploaded data characteristics of all connected data source ends to the middle platform, and determines the retrieval priority based on the data inventory status of all storage intervals in the middle platform to realize orderly retrieval of the middle platform; based on the target data attribute information and retrieval priority that the user end needs to query, the query path is determined to mark the storage interval for data browsing, thereby improving the traceability of the data retrieval of the middle platform; when the target data is successfully retrieved, the retrieval result data packet is returned to the user end based on the distribution status of the target data in the middle platform to ensure comprehensive acquisition of the data in the middle platform; when the target data is not successfully retrieved, the data storage change attribute information of the middle platform is predicted, and data retrieval is performed again, which can avoid repeated and invalid retrieval of the middle platform, and perform orderly and accurate retrieval and search of the big data in the middle platform, thereby improving the data retrieval efficiency of the middle platform.

[0046] Preferably, in step S1, based on the communication log of the middle station, the characteristics of the uploaded data from all connected data source terminals to the middle station are determined, including:

[0047] Based on the gateway traffic change information within the network where the middle station is located, determine the active gateways within the network; monitor the active gateways and determine the identity information of all data sources accessing the active gateways;

[0048] Based on identity information, filter the data upload records from the data source to the middle platform from the communication log of the middle platform; based on the data upload records, determine the type and quantity of data uploaded by the data source to the middle platform, and use this as the feature of the uploaded data.

[0049] The beneficial effects of the above technical solution are as follows: the middle platform, as a big data storage platform, stores different types of data internally. To partition and store different types of data, each type of data is independently stored in one or more storage intervals within the middle platform. To ensure that the middle platform can receive different types of data in a timely and comprehensive manner, the middle platform also connects to different data sources via the network, allowing the data sources to upload their own data or data received from the outside to the middle platform. Each time a data source uploads data to the middle platform, the middle platform's communication log generates a corresponding data upload record, accurately and comprehensively recording the type and amount of data uploaded. During the actual data exchange process between the middle platform and the data source, the data source does not continuously upload data to the middle platform, that is, not all data sources update and add data within the middle platform. To accurately monitor the data upload status of the corresponding data source to the middle platform using minimal resources, the middle platform monitors the traffic changes of the gateways in the network where the middle platform is located (for example, the uplink traffic changes of each gateway in the network) to obtain the average traffic rate of each gateway uploading to the middle platform. If the average traffic rate exceeds a preset traffic threshold, the corresponding gateway is determined to be an active gateway. The identity information of all data sources accessed by active gateways is then used to identify and calibrate the corresponding data sources. Furthermore, based on the identity information of the data sources, the communication logs of the middle platform are used to filter the data types and volumes uploaded to the middle platform by the corresponding data sources. This provides a reliable basis for subsequent adjustments to the data storage status of the middle platform and for predicting future data upload trends of the corresponding data sources.

[0050] Preferably, in step S1, based on the characteristics of the uploaded data, adjusting the storage status of the uploaded data by the middle platform includes:

[0051] Obtain the data storage history of all storage intervals in the middle platform, analyze the data storage history, and obtain the data type and quantity of each storage interval in the middle platform;

[0052] Compare the data type and quantity uploaded by the data source to the middle platform with the stored data type and quantity of all storage intervals, and determine the storage interval that matches the uploaded data from the data source;

[0053] Based on the data upload rate of the data source, adjust the data storage transmission bandwidth allocated by the middle platform to the matching storage interval.

[0054] The beneficial effects of the above technical solution are as follows: the middle station includes multiple relatively independent storage intervals, each of which can store the same type of data or store data uploaded by the same data source. Whenever a storage interval stores data, the middle station's data storage history of the storage interval will be updated accordingly. By analyzing the data storage history, the stored data type and data volume of each storage interval in the middle station are obtained, thereby grasping the data storage status of each storage interval in real time. In order to avoid data storage confusion and crosstalk within the middle station, it is also necessary to adjust the storage status of the uploaded data in the middle station in a targeted manner according to the data uploaded by the data source. Specifically, the data type and quantity uploaded by the data source to the middle station are compared with the stored data type and quantity of each storage interval, and the degree of consistency between the data type uploaded by the data source to the middle station and the stored data type of each storage interval is determined. The storage interval whose consistency exceeds the preset consistency threshold is determined as the storage interval allowed for the data uploaded by the data source to the middle station; then, based on the remaining storage space of all stored storage intervals, the storage interval that matches the uploaded data from the data source is determined, and the storage interval with the largest remaining storage space is usually selected as the matching storage interval. In addition, based on the data upload rate from the data source to the middle platform, the size of the data storage transmission bandwidth allocated by the middle platform to the matching storage interval is adaptively adjusted to ensure that the data uploaded by the data source to the middle platform is stored in the matching storage interval in real time to avoid data storage omissions.

[0055] Preferably, in step S1, based on the data inventory status of all storage intervals in the middle platform, the search priority of all storage intervals is determined, including

[0056] Obtain the data storage capacity and data semantic information of each storage interval in the middle platform, and based on the data storage capacity and data semantic information, determine the content duplication characteristics of the data stored in each storage interval; wherein the content duplication characteristics refer to the proportion of data with the same or similar content in each storage interval;

[0057] Based on the content duplication feature, the search priority of all storage intervals is determined; wherein the search priority includes the search order of all storage intervals and the search time of each storage interval.

[0058] The beneficial effect of the above technical solution is: after the data source uploads data to the middle platform, the middle platform will directly store the received data in the storage interval, and will not filter and organize the data, resulting in duplicate data storage in the storage interval. When the proportion of duplicate data in the storage interval is high, the number of data in the storage interval that is actually meaningful for retrieval is correspondingly low. If the traversal search method is still used for the storage interval with a high proportion of duplicate data, it will not only increase the time consumption of data retrieval, but also cause a waste of retrieval computing resources. In order to ensure effective and accurate retrieval of all storage intervals within a limited time, the proportion of data with the same or similar content stored in each storage interval is determined based on the data storage capacity and data semantic information of all storage intervals in the middle platform. The greater the proportion of data with the same or similar content stored in the storage interval, the later its retrieval order in all storage intervals, and the shorter the retrieval time of the corresponding storage interval, so as to avoid wasting too much time on searching storage intervals with a large proportion of data with the same or similar content.

[0059] Preferably, in step S2, based on the query request of the user terminal, determining the target data attribute information that the user terminal needs to query includes:

[0060] Based on the identity information of the user terminal requesting access to the middle platform, determine the user terminal's data retrieval history on the middle platform; based on the data retrieval history, determine the user terminal's search occupancy time information on the middle platform, so as to judge whether the user terminal has the search permission;

[0061] When the user terminal has the search permission, the query request of the user terminal is parsed to obtain the target data index information that the user terminal needs to query, which is used as the target data attribute information; wherein the target data index information includes the keyword semantic information of the target data that the user terminal needs to query.

[0062] The beneficial effect of the above technical solution is: in order to avoid the user terminal frequently accessing and searching the middle platform and occupying the time of other user terminals accessing and searching the middle platform, based on the identity information of the user terminal requesting access to the middle platform, the user terminal's data search history of the middle platform is obtained, so as to determine the user terminal's historical search occupancy time information on the middle platform (that is, the occupied time interval during the user terminal's historical search process on the middle platform). If the time interval between the user terminal's most recent historical search time interval on the middle platform and the current time point is less than the preset time interval threshold, it is judged that the user terminal does not have the search permission; otherwise, it is judged that the user terminal has the search permission. When the user terminal has the search permission, the user terminal's query request is parsed to obtain the keyword semantic information of the target data that the user terminal needs to query, which is used as the basis for subsequent data screening and identification of the storage interval.

[0063] Preferably, in step S2, based on the target data attribute information and the search priority, a query path for the corresponding storage interval is determined, thereby marking the storage interval for data browsing, including:

[0064] Compare the target data index information with the storage directories of all storage intervals in the middle platform to screen out several target storage intervals; based on the retrieval priority, determine the retrieval order of all target storage intervals and the retrieval time for each target storage interval;

[0065] Based on the search order of all target storage intervals and the search time of each target storage interval, the query path for all target storage intervals is determined; based on the query path, the browsed data in the target storage interval is identified, so as to add an identification code to the data that matches the keyword semantic information.

[0066] The beneficial effects of the above technical solution are as follows: comparing the target data index information with the storage directories of all storage intervals in the middle platform, determining the semantic similarity between the target data index information and the storage directories of all storage intervals, and selecting the storage interval whose semantic similarity exceeds a preset similarity threshold as the target storage interval; then, combining the search priorities corresponding to all target storage intervals, determining the search order for all target storage intervals and the search duration for each target storage interval, thereby determining the query path for all target storage intervals (i.e., the query path for all target storage intervals in the time domain). Using this query path, the browsed data in the corresponding target storage interval is identified according to the corresponding time axis, thereby adding identification codes to the data that matches the keyword semantic information, and performing a preliminary screening of the data in the storage interval.

[0067] Preferably, in step S3, judging whether the target data to be queried by the user end is successfully retrieved based on the data browsing mark includes:

[0068] The creation time of the data with the identification code added in the storage interval is compared with the preset time range. If the creation time is within the preset time range, it is determined that the target data required by the user end is successfully retrieved; otherwise, it is determined that the target data required by the user end is not successfully retrieved.

[0069] The beneficial effect of the above technical solution is: comparing the creation time of the data with the identification code added in the storage interval with the preset time range to determine whether the target data that the user terminal needs to query is successfully retrieved, ensuring that the data matching the user terminal is accurately retrieved from the storage interval.

[0070] Preferably, in step S3, when the search is successful, a search result data packet is generated based on the distribution status of the target data in the middle platform and returned to the user end, including:

[0071] When the retrieval is successful, the data in the corresponding storage intervals is copied, extracted, integrated and compressed based on the distribution location and number of the target data in all storage intervals in the middle platform, and a retrieval result data packet is generated, and the retrieval result data packet is returned to the user end.

[0072] The beneficial effect of the above technical solution is: when the retrieval is successful, based on the distribution location and distribution quantity of the target data in all storage intervals in the middle platform, the corresponding storage intervals are copied, extracted, integrated and compressed to generate a retrieval result data packet to ensure that the retrieved data is returned to the user end in its entirety.

[0073] Preferably, in step S4, when the retrieval fails, the data storage change attribute information of the middle platform is predicted based on the uploaded data characteristics, including:

[0074] When the retrieval fails, the type and amount of data that the data source will upload to the middle platform in the future are predicted based on the characteristics of the uploaded data; based on the type and amount of the future uploaded data, the data storage change attribute information of all storage intervals in the middle platform is predicted; among them, the data storage change attribute information includes the increase in data volume of corresponding types of data in all storage intervals in the middle platform in the future time period.

[0075] The beneficial effects of the above technical solution are: when the retrieval fails, based on the uploaded data characteristics, the type and amount of data that the data source will upload to the middle platform in the future are predicted, so as to predict the increase in data volume of corresponding types of data in all storage intervals in the middle platform in the future time period, and define the scope for subsequent retrieval of the middle platform, avoiding repeated retrieval of data in the storage interval that has been retrieved last time.

[0076] Preferably, in step S4, based on the data storage change attribute information and the target data attribute information, data retrieval is performed again, thereby returning the retrieval result to the user end, including:

[0077] Comparing the increased data volume with a preset data volume threshold, if the increased data volume exceeds the preset data volume threshold, determining that data retrieval is allowed again; otherwise, determining that data retrieval is not allowed again;

[0078] When data retrieval is allowed again, the newly added data in all storage intervals in the middle platform will be retrieved based on the target data attribute information, and the retrieval results will be returned to the user end.

[0079] The beneficial effects of the above technical solution are: comparing the increased data volume with the preset data volume threshold. If the increased data volume exceeds the preset threshold, it indicates that the newly added data volume in the middle platform can ensure that the middle platform can fully utilize its own retrieval computing resources for effective data retrieval. At this time, it is determined that data retrieval is allowed again; when data retrieval is allowed again, based on the target data attribute information, the newly added data in all storage intervals in the middle platform is retrieved, so as to return the retrieval results to the user end, avoiding repeated and invalid searches on the middle platform, and performing orderly and accurate retrieval and search on the big data in the middle platform, thereby improving the data retrieval efficiency of the middle platform.

[0080] It can be seen from the contents of the above embodiments that the data index positioning and retrieval method of the middle platform is based on the uploaded data characteristics of all connected data source ends to the middle platform, adjusts the storage status of the uploaded data by the middle platform, and determines the retrieval priority based on the data inventory status of all storage intervals in the middle platform to achieve orderly retrieval of the middle platform; based on the target data attribute information and retrieval priority that the user end needs to query, the query path is determined to mark the storage interval for data browsing, thereby improving the traceability of the data retrieval of the middle platform; when the target data is successfully retrieved, the retrieval result data packet is returned to the user end based on the distribution status of the target data in the middle platform to ensure that the data in the middle platform is fully acquired; when the target data is not successfully retrieved, the data storage change attribute information of the middle platform is predicted, and data retrieval is performed again, which can avoid repeated and invalid retrieval of the middle platform, and perform orderly and accurate retrieval and search of the big data in the middle platform, thereby improving the data retrieval efficiency of the middle platform.

[0081] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. The middle platform data index positioning and retrieval method is characterized by: It includes the following steps: Step S1: Based on the communication log of the middle station, determine the characteristics of the uploaded data of all connected data source terminals to the middle station; based on the uploaded data characteristics, adjust the storage status of the uploaded data by the middle station; based on the data inventory status of all storage intervals in the middle station, determine the retrieval priority of all storage intervals; Step S2: determining target data attribute information that the user needs to query based on the query request of the user terminal; determining a query path for a corresponding storage interval based on the target data attribute information and the search priority, thereby marking the storage interval for data browsing; Step S3: judging whether the target data to be queried by the user terminal is successfully retrieved based on the data browsing mark; When the search is successful, a search result data packet is generated based on the distribution status of the target data in the middle platform and returned to the user terminal; Step S4: When the retrieval fails, the data storage change attribute information of the middle station is predicted based on the uploaded data characteristics; based on the data storage change attribute information and the target data attribute information, data retrieval is performed again to return the retrieval results to the user end.

2. The method for indexing, locating, and retrieving data from a central platform according to claim 1, wherein: In step S1, based on the communication log of the middle station, the characteristics of the uploaded data from all connected data source terminals to the middle station are determined, including: Based on the gateway traffic change information in the network where the middle station is located, determine the active gateway in the network; monitor the active gateway to determine the identity information of all data source terminals connected to the active gateway; Based on the identity information, the data upload records of the data source end to the middle station are filtered from the communication log of the middle station; based on the data upload records, the type and quantity of upload data from the data source end to the middle station are determined, which are used as the uploaded data features.

3. The method for indexing, locating, and retrieving data from a central platform as claimed in claim 2, wherein: In the step S1, based on the characteristics of the uploaded data, adjusting the storage status of the uploaded data by the middle platform includes: Obtaining data storage history records of all storage intervals in the middle platform, analyzing the data storage history records, and obtaining the type and quantity of data stored in each of the storage intervals in the middle platform; Compare the data type and quantity uploaded by the data source end to the middle platform with the data type and quantity stored in each of all storage intervals to determine the storage interval that matches the uploaded data of the data source end; Based on the data upload rate of the data source, the data storage transmission bandwidth allocated by the middle platform to the matching storage interval is adjusted.

4. The method for indexing, locating, and retrieving data from a central platform as claimed in claim 3, wherein: In step S1, based on the data storage status of all storage intervals in the middle platform, the search priority of all storage intervals is determined, including: Obtaining data storage capacity and data semantic information for each of the storage intervals in the middle platform, and determining content duplication characteristics of the data stored in each storage interval based on the data storage capacity and the data semantic information; wherein the content duplication characteristics refer to the proportion of data with identical or similar content in each storage interval; Based on the content repetition feature, a search priority for all storage intervals is determined; wherein the search priority includes a search order for all storage intervals and a search duration for each storage interval.

5. The method for indexing, locating, and retrieving data from a middle platform according to claim 1, wherein: In step S2, based on the query request of the user terminal, determining the target data attribute information that the user terminal needs to query includes: Based on the identity information of the user terminal requesting access to the middle station, determine the data search history of the user terminal on the middle station; based on the data search history, determine the search time information of the user terminal on the middle station, so as to judge whether the user terminal has the search permission; When the user terminal has the retrieval authority, the query request of the user terminal is parsed to obtain the target data index information that the user terminal needs to query, which is used as the target data attribute information; wherein, the target data index information includes the keyword semantic information of the target data that the user terminal needs to query.

6. The method for indexing, locating, and retrieving data from a middle platform according to claim 5, wherein: In step S2, based on the target data attribute information and the search priority, a query path for the corresponding storage interval is determined, thereby marking the storage interval for data browsing, including: Comparing the target data index information with the storage directories of all storage intervals in the middle platform to screen out several target storage intervals; determining the search order of all target storage intervals and the search time for each target storage interval based on the search priority; Based on the search order of all target storage intervals and the search time of each target storage interval, a query path for all target storage intervals is determined; based on the query path, the browsed data in the target storage interval is identified, so as to add an identification code to the data that matches the keyword semantic information.

7. The method for indexing, locating, and retrieving data from a central platform according to claim 1, wherein: In step S3, based on the data browsing mark, determining whether the target data to be queried by the user terminal is successfully retrieved includes: The creation time of the data with the identification code added in the storage interval is compared with the preset time range. If the creation time is within the preset time range, it is determined that the target data that the user terminal needs to query is successfully retrieved; otherwise, it is determined that the target data that the user terminal needs to query is not successfully retrieved.

8. The method for indexing, locating, and retrieving data from a middle platform according to claim 7, wherein: In step S3, when the search is successful, a search result data packet is generated based on the distribution status of the target data in the middle platform and returned to the user terminal, including: When the retrieval is successful, the data of the corresponding storage intervals is copied, extracted, integrated and compressed based on the distribution position and distribution quantity of the target data in all storage intervals in the middle platform, a retrieval result data packet is generated, and the retrieval result data packet is returned to the user end.

9. The method for indexing, locating, and retrieving data from a middle platform according to claim 1, wherein: In step S4, when the retrieval fails, the data storage change attribute information of the middle platform is predicted based on the uploaded data characteristics, including: When the retrieval fails, the type and amount of data that the data source will upload to the middle platform in the future are predicted based on the characteristics of the uploaded data; based on the type and amount of future uploaded data, the data storage change attribute information of all storage intervals in the middle platform is predicted; wherein, the data storage change attribute information includes the increased data volume of corresponding type data of all storage intervals in the middle platform in the future time period.

10. The method for indexing, locating, and retrieving data from a central platform according to claim 9, wherein: In step S4, based on the data storage change attribute information and the target data attribute information, data retrieval is performed again, thereby returning the retrieval result to the user terminal, including: Comparing the increased data volume with a preset data volume threshold, and if the increased data volume exceeds the preset data volume threshold, determining that data retrieval is allowed again; otherwise, determining that data retrieval is not allowed again; When data retrieval is allowed again, the newly added data in all storage intervals in the middle platform are retrieved based on the target data attribute information, and the retrieval results are returned to the user end.

Images