Multi-specialty real-time measurement data high-concurrency data sharing retrieval analysis method, system and equipment and storage medium

Abstract

The invention discloses a multi-specialty real-time measurement data high-concurrency data sharing retrieval analysis method, system and device and a storage medium, and the method comprises the following steps: obtaining and analyzing original retrieval requests, judging whether the original retrieval requests are the same or not, and if the original retrieval requests are the same, merging the retrieval requests; if not, taking the originally retrieved URL as shared retrieval information; original retrieval parameters are used as constraint conditions; accessing a shared retrieval cache according to the shared retrieval information to obtain a shared retrieval ID, querying a retrieval result cache according to the shared retrieval ID, if a result exists, directly obtaining a retrieval result, and if the result does not exist, obtaining a measurement data access address from the shared retrieval cache according to the shared retrieval ID; the shared retrieval obtains a shared retrieval result from the measurement database according to the measurement data access address, stores the shared retrieval result in a shared retrieval result cache, and notifies an original retrieval request; the original retrieval request obtains an original retrieval result from the shared retrieval result according to the constraint condition; according to the method, the calling frequency of the hotspot common service is reduced, and the overall external real-time data high-speed sharing capability of the common service is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent electricity technology, and in particular to a method, system, device and storage medium for sharing, retrieval and analysis of multi-professional real-time measurement data with high concurrency. Background Art

[0002] The new power system involves larger-scale data collection and control objects, which are gradually extended and sunk to the distribution side. Both the power supply side and the load side are highly random, which puts forward higher requirements for the real-time coordination of source, grid, load and storage and the digital level of the power grid. In the research related to the new power system and dual-carbon digital technology support, through the design of the framework of the new power system technology support system, an enterprise-level real-time measurement center is created to aggregate the measurement data of all links of the power grid online on demand to support high-speed sharing applications of various professions. At present, due to the large variety and volume of data aggregated by the enterprise real-time measurement center, it is often under tremendous retrieval pressure in the face of the data access needs of various business systems. The data access needs of various business systems have high concurrency, and most of the data needs are often concentrated on a small amount of hot data. Summary of the invention

[0003] Purpose of the invention: The purpose of the present invention is to provide a method, system, device and storage medium for sharing, retrieval and analysis of multi-professional real-time measurement data with high concurrency, so as to solve the problems existing in the background technology.

[0004] Technical solution: The method for sharing and retrieving multi-professional real-time measurement data with high concurrency includes the following steps: obtaining and analyzing the original search request to determine whether the original search requests are the same, and if so, merging the original search requests; if not, using the uniform resource locator URL of the original search as the shared search information; using the parameters of the original search as constraints; accessing the shared search cache according to the shared search information to obtain a shared search ID, querying the search result cache according to the shared search ID, and directly obtaining the search result if the result exists, and obtaining the measurement data access address from the shared search cache according to the shared search ID if the result does not exist; the shared search obtains the shared search result from the measurement database according to the measurement data access address, stores it in the shared search result cache, and notifies the original search request; the original search request obtains the original search result from the shared search result according to the constraints.

[0005] Furthermore, merging the original search requests specifically includes: merging the same search requests according to a certain time interval, setting a threshold according to a certain search frequency, and meeting the search requirements with high timeliness requirements.

[0006] Further, the original search request is merged, a shared search ID is generated by a hashing method, and the common service address is hashed based on the hash operator to obtain a hash value. The specific process is as follows: the original search ID, common service address, original search domain, original search constraint and other information are initialized into the instance model; the common service access address is used as the hash field string to calculate and generate the corresponding hash value; the hash value calculated in the previous step is used to query the corresponding information in the hash table. If the information is empty, a shared search identifier is generated according to the hash algorithm, and the shared search identifier of the search mapping relationship structure is filled in, and the structure is added to the position to end the merging process; otherwise, it is verified whether the common service access address of the structure in the position is consistent with that in the original query request. If it is consistent, the original search structure is added to the position, and the pointer in the existing structure in the position points to the newly added structure. Otherwise, it moves back one position in the hash table, and executes the hash algorithm cyclically to generate shared information until a hash value of an empty position is found.

[0007] Furthermore, upper and lower thresholds are set for the retrieval result cache according to a certain retrieval frequency. If the upper threshold is exceeded, the retrieval results are obtained from the measurement database through the measurement data access address and the retrieval result cache is updated; if it is lower than the lower threshold, the shared retrieval information is deleted from the shared retrieval cache and the shared retrieval result cache.

[0008] The multi-professional real-time measurement data high-concurrency data sharing and retrieval system described in the present invention comprises: Acquisition module: used to obtain the original search request and analyze it; Judgment module: used to judge whether the original search requests are the same. If they are the same, the original search requests are merged; if they are different, the URL of the original search is used as the shared search information; the parameters of the original search are used as constraints; according to the shared search information, the shared search cache is accessed to obtain the shared search ID; Shared module: used to query the search result cache according to the shared search ID. If the result exists, the search result is directly obtained. If the result does not exist, the measurement data access address is obtained from the shared search cache according to the shared search ID. Result module: used to notify the original search request to access the measurement database and obtain the search results based on the measurement data access address and the obtained constraints; used to update or delete the shared search-related cache and result cache based on the upper and lower thresholds set for the search frequency.

[0009] Furthermore, in the acquisition module, merging the original search requests specifically includes: merging the same search requests at a certain time interval.

[0010] Furthermore, in the acquisition module, the original search requests are merged, a shared search ID is generated using a hashing method, and a hash calculation is performed on the common service address based on a hash operator to obtain a hash value.

[0011] Furthermore, in the result module, upper and lower thresholds are set for the search result cache according to a certain search frequency. If the upper threshold is exceeded, the search results are obtained from the measurement database through the measurement data access address and the search result cache is updated; if it is lower than the lower threshold, the shared search information is deleted from the shared search cache and the shared search result cache.

[0012] An electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, any one of the methods, systems, devices, and storage media described in the present invention for sharing, retrieval, and analyzing multi-professional real-time measurement data with high concurrency is obtained.

[0013] A storage medium described in the present invention stores a computer program, and when the computer program is executed by a processor, it implements any one of the methods, systems, devices and storage media for sharing, retrieval and analysis of multi-professional real-time measurement data with high concurrency.

[0014] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: the present invention proposes merging original search requests, reduces the frequency of calling hot common services, improves the overall high-speed external real-time data sharing capabilities of common services, and provides strong support for high-frequency and high-concurrency data access needs of business applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flow chart of the method of the present invention; Figure 2 This is a schematic diagram of the original search merge request analysis of the present invention; Figure 3 It is a schematic diagram of the retrieval mapping relationship analysis based on the time window and the retrieval frequency threshold of the present invention; Figure 4 Schematic diagram of the screening and analysis of original search results for inventions. DETAILED DESCRIPTION

[0016] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0017] like Figure 1 As shown, the embodiments of the present invention provide basis.

[0018] Now, in conjunction with a specific case, a multi-professional real-time measurement data high-concurrency shared retrieval and analysis method of this embodiment includes the following steps: Step 1: Create a retrieval request mapping relationship Analyze the needs of the original search request, build a mapping relationship between the original search and the shared search request, and support the business process of subsequent search request sharing merging and shared result screening access.

[0019] The retrieval request mapping relationship is as follows: Map <String,List <retrievemaprelationstruct>>map_retrieveID_orginonalList. The key is the shared retrieval request ID, and the value is the queue of the original request information. The retrieveMapRelationStruct structure of the original request information is as follows: Struct retrieveMapRelationStruct { Int retrieveID; Int origionalID; String serviceURL; } In this structure, the retrieveID item represents the identifier of the shared retrieval request, which is represented by a 32-bit integer; the originalID item represents the original query request identifier, which is represented by a 32-bit integer; the serviceURL item represents the common service address called by the original retrieval request and the shared retrieval request; this structure can effectively record the mapping relationship between the original retrieval request and the shared retrieval request as well as the common service information of the query.

[0020] Step 2: Create search constraints For the original search requests of each business system, search constraints are constructed to support the subsequent acquisition of original search results by superimposing constraint analysis on the shared search results.

[0021] The retrieval constraint mapping relationship is as follows: Map<String,retrieveConstraintsStruct> map_orginonalID_constraintsInfo. The key is the original search request ID, and the value is the original request constraint information. The structure of the original request constraint information constraintsInfo is as follows: Struct retrieveConstraintsStruct { Int origionalID; String serviceURL; String fields; String filters; } In this structure, the originalID item represents the original query request identifier, which is represented by a 32-bit integer; the serviceURL item represents the common service address called by the original search request and the shared search request; the fields item represents the field list of the original search request, and multiple fields are separated by commas; the filters item represents the constraint list of the original search request, and multiple fields are separated by commas. This structure can effectively record the mapping relationship between the original search request and the constraint information of the original search request.

[0022] Step 3: Create a search result cache Build a search result cache mapping relationship to save the search results obtained by calling common services after sharing the search request, and support the screening of results by subsequent original search requests.

[0023] The retrieval result cache mapping structure is as follows: Map<String, retrieveResultStruct> map_retrieveID_retrieveResultStruct The key stores the shared search request identifier, and the value stores the search result cache information. The retrieveResultStruct structure of the search result cache information is as follows: Struct retrieveResultStruct { Int retrieveID; String serviceURL; long timestamp; / / The last query time long duration; / / Query the service duration long priorityLevel; / / priority List recordList; / / Call frequency, save the time of each service call } In this structure, the retrieveID item indicates the identifier of the shared retrieval request; the serviceURL item indicates the common service address called by the original retrieval request and the shared retrieval request; the timestamp item indicates the most recent call time of the common service; the duration item indicates the duration of the most recent call return of the common service; the priorityLevel item indicates the priority of the common service; the recordList item indicates the frequency of the call, saves the time of each service call, and deletes the call records that exceed the threshold. This structure is used to effectively record the structural information of the retrieval results.

[0024] The priority is determined by three factors: the last query time of the retrieval results, the hot spot frequency, and the query duration. longpriorityLevel=x(currentTimes-timestamp)+y* recordList.size()+z*duration; x, y, z are coefficients, currentTimes is the current time, timestamp is the last call time of the common service, recordList is the call time list, which saves the record of each call to the service and periodically deletes the call records that exceed the set threshold. The total number of records represents the call frequency.

[0025] Step 4: Original search request merge process Business applications continuously initiate call requests to common services based on demand. If all these query requests are merged together, the query request merging process will take too long, which cannot meet the business application's demand for data timeliness. In order to solve this problem, a time window and query number threshold are set in the high-concurrency data retrieval mechanism to merge query requests within the same time window or number threshold. This has two benefits: First, setting the time window and query number threshold avoids the time consumption of the merging task due to excessive merging time during the merging of retrieval requirements, thereby reducing the pressure of the merging task; second, the time window and query number threshold avoid the instability of the service caused by excessive memory overhead of the merging task due to too many merged retrieval requirements, thereby improving the stability of common services. The setting of the time window and query number threshold has a significant impact on the performance of the entire retrieval mechanism. If the parameter is set to a large value, more query request tasks will be merged, affecting the timeliness of the return. If the parameter is set to a small value, not enough query requests will be merged, and the performance improvement will not be obvious. The operation and maintenance personnel need to set the parameters according to the concurrency of the business application and the timeliness requirements of the data. Generally, smaller parameters are required for large concurrency and high timeliness requirements, while larger parameters are required for small concurrency and low timeliness requirements.

[0026] Within the same time window and time threshold, the common service continuously receives query requests from business applications for the service and stores the query requests in memory. When the request merging conditions are met, query requests for the same table are merged into shared query requests.

[0027] like Figure 2 As shown, the retrieval merge request mechanism merges access requests for the same common service to avoid the problem of hot data occupying too many request resources. When obtaining data, the retrieval request first reads the data into the cache by calling the common service, and then the original retrieval request obtains the required data from the cache according to their respective needs, reducing the number of times the common service reads from the database.

[0028] Step 5: Construct retrieval requirement mapping relationship When a new search request from a business application is received within the time window and the time period when the total number of searches is not reached, first determine whether it has been merged into the existing shared query request. To avoid the waste of resources and low time efficiency caused by reading all shared query requests during the matching and mapping process, the shared search ID is generated using a hashing method. The hash value is obtained by performing a hash calculation on the common service address based on the hash operator. Figure 3 As shown, the entire process of constructing the retrieval requirement mapping relationship is divided into the following steps: (a1) Construct the retrieveMapRelationStruct structure and initialize the originalID, serviceURL, fields, and filters into the structure; (a2) Using the common service access address (serviceURL) as the hash field string, calculate and generate the corresponding hash value; (a3) Use the hash value calculated in step (a2) to query the corresponding information in the hash table. If the information is empty, execute step (a4), otherwise execute step (a5) (a4) Generate a shared retrieval identifier based on the hash value calculated in step (a2), fill in the shared retrieval identifier (retrieveID) of the retrieval mapping relationship (retrieveMapRelationStruct) structure, add the structure to the position, and execute step (a8) (a5) Verify whether the common service access address of the structure at the location is consistent with that in the original query request. If it is consistent, execute step (a6); otherwise, execute step (a7); (a6) Add the original search structure to the location and point the pointer in the existing structure at the location to the newly added structure. Execute step (a8).

[0029] (a7) Move back one position in the hash table and execute step (a3) (a8) End the merging process.

[0030] Step 6: Original search results screening process The shared search request obtains the search results by calling the common service, and each original search request obtains the data it needs according to the screening conditions, wherein each original search request can read the shared search results in parallel.

[0031] In order to ensure that each original query request can obtain the required business data, in the data return process, the server-side serial and client-side parallel technical routes are adopted to implement the data interaction process of retrieving data from the common service and obtaining the required data from the search results in parallel for the original request. Among them, the server calls the common service to obtain data, and the client filters the search results and returns the original query request. The access service of the server obtains data through the access address of the common service and sends a signal to the client that the shared search has ended; the client's filtering service contains the filtering constraints of the original interpolation request, and is responsible for filtering and filtering the search results according to the search constraints contained in the parameters, obtaining query results that meet the search constraints, and returning them to the original query service.

[0032] String getRetrieveResult(String serviceURL) { Get result from serviceURL; Return signal; } String getOriginResult(String originParameter) { Get result by originParameter; Return result; } like Figure 4 As shown, the specific process of screening the original search results is as follows: (s1) Share the search ID to check whether the record exists in the search result cache. If it does, execute step (3); if not, execute step (2); (s2) The shared search request obtains the shared search results by calling the access address of the common service. On the one hand, the search results are stored in the Map using the shared search ID as the key.<String,retrieveResultStruct> In the map_retrieveID_retrieveResultStruct structure cache, on the one hand, a consumption signal is sent to each original retrieval request; (s3) Each original search request listens to the consumption signal sent by the shared search result, obtains its own constraint conditions according to the search constraint condition mapping relationship, and obtains the search result corresponding to retrieveResultStruct from the search result cache according to the shared search ID; (s4) Each original search request obtains the required data by adding constraints to the search results and analyzing them.

[0033] The specific process of updating shared search results is as follows: (b1) polling the shared search result cache at a certain period, and comprehensively analyzing the priority according to the query time, hot frequency, query duration, etc. If the priority is greater than the time threshold, executing step (2); otherwise executing step (4); (b2) Determine whether the query time of the shared search results exceeds the data timeliness requirement. If so, execute step (b3); if not, terminate the process; (b3) By calling the common service address in the shared search information, the latest result is updated in the shared search cache, the results are updated, and the process ends; (b4) The information is deleted from the shared search results and the process ends.

[0034] The multi-professional real-time measurement data high-concurrency data sharing and retrieval system described in the present invention comprises: Acquisition module: used for acquiring and analyzing the original search request; wherein, merging the original search request specifically includes: merging the same search request at a certain time interval. Judgment module: used to determine whether the original search requests are the same. If they are the same, the search requests are merged; if they are different, the URL of the original search is used as the shared search information; the parameters of the original search are used as constraints; according to the shared search information, the shared search cache is accessed to obtain the shared search ID; the search requests are merged, and the shared search ID is generated using the hash hashing method. The common service address is hashed based on the hash operator to obtain the hash value.

[0035] Shared module: used to query the search result cache according to the shared search ID. If the result exists, the search result is directly obtained. If the result does not exist, the measurement data access address is obtained from the shared search cache according to the shared search ID. Result module: used to notify the original search request to access the measurement database and obtain the search results according to the measurement data access address and the obtained constraints; used to update or delete the shared search-related cache and result cache according to the upper and lower thresholds set for the search frequency. The upper and lower thresholds are set for the search result cache according to a certain search frequency. If the upper threshold is exceeded, the search results are obtained from the measurement database through the measurement data access address and the search result cache is updated; if it is below the lower threshold, the shared search information is deleted from the shared search cache and the shared search result cache.

[0036] An embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, any one of the methods, systems, devices, and storage media described in the invention can be used to share, retrieve, and analyze multi-professional real-time measurement data with high concurrency.

[0037] An embodiment of the present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements any one of the methods, systems, devices and storage media for sharing, retrieval and analysis of multi-professional real-time measurement data with high concurrency.< / retrievemaprelationstruct>

Claims

1. A multi-professional real-time measurement data high-concurrency data sharing and retrieval method, characterized in that: The following steps are involved: Obtain the original search request and analyze it to determine whether the original search requests are the same. If they are the same, merge the search requests; if they are different, use the uniform resource locator URL of the original search as the shared search information; use the parameters of the original search as constraints; access the shared search cache according to the shared search information to obtain the shared search ID, query the search result cache according to the shared search ID, and directly obtain the search result if the result exists; if the result does not exist, obtain the measurement data access address from the shared search cache according to the shared search ID; obtain the shared search result from the measurement database according to the measurement data access address, store it in the shared search result cache, and notify the original search request; the original search request obtains the original search result from the shared search result according to the constraints.

2. According to claim 1, a multi-professional real-time measurement data high-concurrency data sharing and retrieval method is characterized in that: The merging of original search requests specifically includes: merging the same search requests according to a certain time interval, setting a threshold according to a certain search frequency, and meeting the search requirements with high timeliness requirements.

3. According to claim 2, a multi-professional real-time measurement data high-concurrency data sharing and retrieval method is characterized in that: The search requests are merged, a shared search ID is generated using a hashing method, and the common service address is hashed based on a hash operator to obtain a hash value.

4. According to claim 1, a multi-professional real-time measurement data high-concurrency data sharing and retrieval method is characterized in that: The search result cache is set with upper and lower thresholds according to a certain search frequency. If the upper threshold is exceeded, the search result is obtained from the measurement database through the measurement data access address, and the search result cache is updated; If it is lower than the lower threshold, the shared search information is deleted from the shared search cache and the shared search result cache.

5. A multi-professional real-time measurement data high-concurrency data sharing and retrieval system, characterized in that: include: Acquisition module: used to obtain the original search request and analyze it; Judgment module: used to judge whether the original search requests are the same. If they are the same, the original search requests are merged; if they are different, the original search uniform resource locator URL is used as the shared search information; the parameters of the original search are used as constraints; according to the shared search information, the shared search cache is accessed to obtain the shared search ID; Shared module: used to query the search result cache according to the shared search ID. If the result exists, the search result is directly obtained. If the result does not exist, the measurement data access address is obtained from the shared search cache according to the shared search ID. Result module: used to notify the original search request to access the measurement database and obtain the search results according to the measurement data access address and the obtained constraints; Used to update or delete shared search-related caches and result caches based on the upper and lower thresholds set for search frequency.

6. A multi-professional real-time measurement data high-concurrency data sharing and retrieval system according to claim 5, characterized in that: In the acquisition module, merging the original search requests specifically includes: merging the same search requests according to a certain time interval.

7. The multi-professional real-time measurement data high-concurrency data sharing and retrieval system according to claim 5 is characterized in that: In the acquisition module, the original search requests are merged, a shared search ID is generated using a hashing method, and a hash calculation is performed on the common service address based on a hash operator to obtain a hash value.

8. The multi-professional real-time measurement data high-concurrency data sharing and retrieval system according to claim 5 is characterized in that: In the result module, the upper and lower thresholds are set for the search result cache according to a certain search frequency. If the upper threshold is exceeded, the search result is obtained from the measurement database through the measurement data access address, and the search result cache is updated; If it is lower than the lower threshold, the shared search information is deleted from the shared search cache and the shared search result cache.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into a processor, a high-concurrency data sharing and retrieval method for multi-professional real-time measurement data according to any one of claims 1 to 4 is implemented.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a high-concurrency data sharing and retrieval method for multi-professional real-time measurement data according to any one of claims 1 to 4 is implemented.

Images