Method for solving fast query of big data storage in intelligent video retrieval
Through the hierarchical query cache and calculation methods, the query speed of the intelligent video retrieval system is optimized, the problem of slow big data storage query is solved, the response time of 1s to 5s is achieved, and the system efficiency is improved through cache and asynchronous processing.
Patent Information
- Application Number
- CN202411012574.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-12-06
AI Technical Summary
When the existing intelligent video retrieval system processes large batches of video data, the query speed is slow and cannot meet the user experience requirements. Especially when processing AI detection data of more than 20W per day, it takes more than 1 minute to query normally.
Using hierarchical query cache and calculation methods, the first level queries data concurrently from Elasticsearch, the second level caches the query results to Redis, the third level caches it into server memory, and performs data deduplication and aggregation within the time range. The image center point coordinates are used to perform area area interchange and compare IoU calculations, optimize query performance, and improve response speed through performance tuning and asynchronous processing.
Reducing the query speed from the original 1 minute or more to 1s to 5s, the fast query of big data storage is realized, memory overflow is prevented, and system efficiency is improved through asynchronous processing and data cleaning mechanisms.
Smart Images

Figure CN119089002A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of intelligent video retrieval, and in particular to a method for solving fast query of big data storage in intelligent video retrieval. Background Art
[0002] At present, our intelligent video retrieval system uses Elasticsearch 7.x to manage the AI detection data of all camera frames. On average, each camera generates more than 200,000 AI detection scene data per day, such as human intrusion and human attribute detection data. We obtain the video detection data of a camera on a daily basis or detect the list of events that occur in a certain area on a daily basis. Among them, these 200,000+ AI detection data need to be quickly queried and processed into point information on the 24-hour time axis according to the time period. If we query Elasticsearch normally and calculate the full amount, it will take more than 1 minute to complete the processing, which is unacceptable in terms of user experience.
[0003] However, most of the existing technologies are aimed at optimizing retrieval capabilities, but fail to achieve optimization in terms of fast query when intelligent video retrieval of large amounts of data is performed. Summary of the invention
[0004] The purpose of the present invention is to solve the shortcomings of the above-mentioned prior art and provide a method for solving the problem of fast query of big data storage in intelligent video retrieval, comprising the following steps:
[0005] S1: Set up hierarchical query cache and calculation. The first-level query uses paging to concurrently query data from Elasticsearch.
[0006] S2: Cache the query data result as a second-level query in Redis according to a set time range, and then cache it in the server memory as a third-level query. When querying, the query data is preferentially obtained from the third level;
[0007] S3: After obtaining the query data, the data returned to the Web is deduplicated according to several concurrent calculations at a time and then aggregated to form a time axis data format with the start time of the detection event as a point;
[0008] S4: Optimize the calculation of detecting events occurring in a certain area of the camera screen according to the time range set when caching in Redis, and calculate the intersection over union (IoU) of the detection area according to the coordinates of the center point of the image (x, y, w, h).
[0009] Furthermore, when performing concurrent queries in step S1, performance tuning and setting of the data volume and concurrent number of word queries are performed, specifically including:
[0010] S11: Test the amount of data for each query when the response speed is optimal based on the current device environment and Elastic service environment;
[0011] S12: According to the current device environment and the Elastic service environment, the concurrency number that satisfies the shortest query time and does not cause pressure on the Elastic service is tested, and the concurrency number of the query is set accordingly.
[0012] Furthermore, in step S2, during query, the server memory is preferentially detected and data is obtained therefrom. If the detected data does not exist, the Redis is detected and obtained therefrom. If the data still does not exist, the Elasticsearch is queried.
[0013] Preferably, in step S2, a scheduled task is set to add the data of the previous hour to the Redis and the server memory on an hourly basis, so in step S3, the data from one hour ago is always processed asynchronously, and the data of the latest hour is calculated in real time and then aggregated.
[0014] Furthermore, in step S3, after the data is aggregated, the order of the data will change, and the data will be sorted according to time using a sorting function.
[0015] Furthermore, before performing step S4, an upper threshold is set according to device performance, and then hotspot data is set, and data of frequently used cameras is cached according to different time strategies. Once the set time strategy is exceeded, data is cleared.
[0016] Preferably, step S4 specifically includes:
[0017] The calculation optimization is to detect the occurrence of events in a certain area of the camera screen according to the time range set when caching in Redis, and the image center point coordinates (x, y, w, h) are used to perform data filtering by the intersection of the detection area and the IoU calculation method. The calculated x, y, w, h parameter range is used as a condition to filter the x, y, w, h of all data in the cache, and then the filtered result is returned.
[0018] Furthermore, the Elasticsearch query volume defaults to 10,000. If it exceeds 10,000, the value of index.max_result_window is additionally set according to the amount of data queried.
[0019] Preferably, the optimization is performed by using both Redis cache and server memory cache. The cached kv data is not a single parameter. The conversion between json string and data object requires serialization and deserialization. It is also time-consuming to retrieve and deserialize structured data from redis, so the third-level query is required.
[0020] Compared with the prior art, the present invention has the following beneficial effects:
[0021] (1) The present invention reduces the call interface response speed from more than 1 minute to 1s to 5s by setting up hierarchical query cache and calculation and performing performance testing during query.
[0022] (2) The present invention divides the query data by time, concurrently calculates more than 200,000 detection data generated in one day, concurrently calculates by hour, removes duplicates, and then aggregates them to form a time axis data format with the start time of the detection event as the point;
[0023] (3) The present invention sets a third-level query, a data cached in the server memory to prevent memory overflow and clean up the mechanism, and the rules for cleaning and caching data are related to the camera online and offline, and the data heat ranking (access frequency);
[0024] (4) The present invention uses a hierarchical query cache and an asynchronous execution mechanism for calculation, and always asynchronously processes data from one hour ago, and the data from the latest hour is calculated in real time and then aggregated;
[0025] (5) The present invention optimizes the calculation of events occurring in a certain area of the camera screen on a daily basis, and calculates the intersection over union (IoU) ratio of the detection area according to the coordinates (x, y, w, h) of the center point of the image. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:
[0027] Figure 1 A flow chart of a method for solving fast query of big data storage in intelligent video retrieval according to the present invention;
[0028] Figure 2 A schematic diagram of a method for quickly querying big data storage in intelligent video retrieval according to the present invention;
[0029] Figure 3 A schematic diagram of calculating the intersection over union (IoU) ratio of a detection area according to the present invention;
[0030] Figure 4 This is a display diagram of the response of a fast query calling interface for big data storage in intelligent video retrieval according to the present invention. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0032] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.
[0033] The specific implementation of the present invention is described below with reference to the accompanying drawings and embodiments.
[0034] Example 1
[0035] like Figure 1 As shown, a method for solving fast query of big data storage in intelligent video retrieval provided by this embodiment includes the following steps:
[0036] S1: Set up hierarchical query cache and calculation. The first-level query uses paging to concurrently query data from Elasticsearch.
[0037] S2: Cache the query data result as a second-level query in Redis according to a set time range, and then cache it in the server memory as a third-level query. When querying, the query data is preferentially obtained from the third level;
[0038] S3: After obtaining the query data, the data returned to the Web is deduplicated according to several concurrent calculations at a time and then aggregated to form a time axis data format with the start time of the detection event as a point;
[0039] S4: Optimize the calculation of detecting events occurring in a certain area of the camera screen according to the time range set when caching in Redis, and calculate the intersection over union (IoU) of the detection area according to the coordinates of the center point of the image (x, y, w, h).
[0040] When performing concurrent query in step S1, the performance test and the setting of the data volume and the number of concurrent queries for word queries are performed, which specifically includes:
[0041] S11: Test the amount of data for each query when the response speed is optimal based on the current device environment and Elastic service environment;
[0042] S12: According to the current device environment and the Elastic service environment, the concurrency number that satisfies the shortest query time and does not cause pressure on the Elastic service is tested, and the concurrency number of the query is set accordingly.
[0043] Furthermore, in step S2, during query, the server memory is preferentially detected and data is obtained therefrom. If the detected data does not exist, the Redis is detected and obtained therefrom. If the data still does not exist, the Elasticsearch is queried.
[0044] In addition, in step S2, a scheduled task is set to add the data of the previous hour to the Redis and the server memory on an hourly basis. Therefore, in step S3, the data from one hour ago is always processed asynchronously, and the data of the latest hour is calculated in real time and then aggregated.
[0045] Furthermore, in step S3, after the data is aggregated, the order of the data will change, and the data will be sorted according to time using a sorting function.
[0046] Before performing step S4, an upper threshold is set according to the device performance, and then hotspot data is set. Frequently used cameras are cached according to different time strategies. Once the set time strategy is exceeded, data is cleared.
[0047] In this embodiment, step S4 is as follows: Figure 3 As shown, specifically including:
[0048] The calculation optimization is to detect the occurrence of events in a certain area of the camera screen according to the time range set when caching in Redis, and the image center point coordinates (x, y, w, h) are used to perform data filtering by the intersection of the detection area and the IoU calculation method. The calculated x, y, w, h parameter range is used as a condition to filter the x, y, w, h of all data in the cache, and then the filtered result is returned.
[0049] The default value of the Elasticsearch query volume is 10,000. If it exceeds 10,000, the value of index.max_result_window is additionally set according to the amount of data queried.
[0050] Preferably, the Redis cache and server memory cache are used together for optimization. The cached kv data is not a single parameter. The conversion between the json string and the data object requires serialization and deserialization. It is also time-consuming to retrieve and deserialize structured data from redis, so the third-level query is required.
[0051] After the above optimization, the response speed of calling the interface is reduced from more than 1 minute to 1s to 5s. Figure 4 shown.
[0052] The specific steps of this embodiment are as follows: Figure 2 shown.
[0053] If you want to query more than 10,000 data in Elasticsearch, you need to set the "index.max_result_window": parameter on the created index. If you want to query 200,000 data at a time, use HTTP PUT to request index name / __settings and set the request parameter to {"index.max_result_window":200000}.
[0054] Then, query cache and calculation are performed in stages. The first stage is to concurrently query data from Elasticsearch in a paging-like manner. After query conditions are spliced, one piece of data that meets the conditions is first queried (response time is a few milliseconds), and the total number of data that meets the conditions is obtained; then a paging-like query is performed to improve the query speed.
[0055] Next, we make performance adjustments. Normally, it takes 1.5s to 2s to query 10,000 pieces of data, and less than 1 second to query 5,000 pieces of data. After testing, we found that the best response speed is to query 4,000 pieces of data each time, with an average of 200ms. Elastic query supports querying data in the from and size modes, and then designs concurrent threads to make concurrent requests to solve the problem of slow queries. The number of concurrent requests is obtained by dividing the total data to be queried by the number of queries each time. The last query contains less than 4,000 pieces of data.
[0056] When performing concurrent queries, you need to set the number of concurrent requests reasonably. For example, to query 203,000 data, normally 200,000 / 4,000+3,000 / 4,000~=51 concurrent requests are needed to complete the query in the shortest time. However, Elastic's concurrent query obviously cannot meet 51 simultaneous requests, and it will put a lot of pressure on the service, which may cause the Elastic service to crash. After a lot of tests, the maximum concurrent request is 16, which is the best. That is, to query 200,000 data, all the data can be found in 1.5s to 5s according to the above method.
[0057] The core design function codes of the performance adjustment part are as follows:
[0058]
[0059]
[0060]
[0061]
[0062] Since the order of data will change after concurrent queries and reaggregation, we can use the sorting function to sort in descending or ascending order according to time. Since the scenario of the present invention is to render data from 0:00 to 24:00 a day, all data are sorted in ascending order according to time.
[0063] The second-level query is cached in Redis. The above operations still cannot respond to the Web in a short time, so the query results are cached in Redis on a daily basis.
[0064] The data returned to the web does not need to be all the data, but only the intermittent time period data detected by AI such as the presence of people in the video. Therefore, the present invention processes all the data concurrently on an hourly basis, and deduplicates the data and the data segment points that need to be rendered (with a start time) at intervals of 20 seconds by default. After the above steps, the average response time of calling the interface of this function is between 1.5 seconds and 5 seconds.
[0065] If the request interface response time continues to be compressed to the second level, we use the redis cache + memory cache method for optimization. If only memory is used, the cached data will be lost once the service crashes or restarts. If only redis is used, the time to read 20w data for deserialization will not reach the second level, because the cached kv data is not a single parameter, and serialization and deserialization are required to convert between json strings and data objects.
[0066] Processing cache data strategy, the scheduled task adds the data of the previous hour to the redis cache and memory every hour. When querying, it first detects whether there is data in the memory, and then detects and queries the data in redis and elastic.
[0067] The cache data cleaning mechanism will set an upper threshold based on the device memory, set hotspot data, and cache data for frequently used cameras according to different time strategies. If a period of time exceeds the set time strategy, the data will be cleaned up.
[0068] like Figure 3 As shown, this embodiment optimizes the calculation of events occurring in a certain area of the camera screen on a daily basis, uses the image center point coordinates (x, y, w, h) to perform data filtering by using the intersection over union (IoU) calculation method for the detection area area, and uses the calculated x, y, w, h parameter range as a condition to filter the x, y, w, h of all data in the cache, and then returns the filtered result.
[0069] Finally, it should be noted that the above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.
[0070] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
Claims
1. A method for solving fast query of big data storage in intelligent video retrieval, characterized in that: The steps include: S1: Set up hierarchical query cache and calculation. The first-level query uses paging to concurrently query data from Elasticsearch. S2: Cache the query data result as a second-level query in Redis according to a set time range, and then cache it in the server memory as a third-level query. When querying, the query data is preferentially obtained from the third level; S3: After obtaining the query data, the data returned to the Web is deduplicated according to several concurrent calculations at a time and then aggregated to form a time axis data format with the start time of the detection event as a point; S4: Optimize the calculation of detecting events occurring in a certain area of the camera screen according to the time range set when caching in Redis, and calculate the intersection over union (IoU) of the detection area according to the coordinates (x, y, w, h) of the center point of the image.
2. The method for solving fast query of big data storage in intelligent video retrieval according to claim 1 is characterized in that: When performing concurrent queries in step S1, performance tuning and setting of the data volume and concurrent number of word queries are performed, specifically including: S11: Test the amount of data for each query when the response speed is optimal based on the current device environment and Elastic service environment; S12: According to the current device environment and the Elastic service environment, the concurrency number that satisfies the shortest query time and does not cause pressure on the Elastic service is tested, and the concurrency number of the query is set accordingly.
3. The method for solving fast query of big data storage in intelligent video retrieval according to claim 1 is characterized in that: In step S2, during query, the server memory is first detected and data is obtained therefrom. If the detected data does not exist, the Redis is detected and obtained therefrom. If the data still does not exist, the Elasticsearch is queried.
4. The method for solving fast query of big data storage in intelligent video retrieval according to claim 3 is characterized in that: In step S2, a scheduled task is set to add the data of the previous hour to the Redis and the server memory on an hourly basis. Therefore, in step S3, the data from one hour ago is always processed asynchronously, and the data of the latest hour is calculated in real time and then aggregated.
5. The method for solving fast query of big data storage in intelligent video retrieval according to claim 1 is characterized in that: In step S3, after the data is aggregated, the order of the data will change, and the data will be sorted by time using a sorting function.
6. The method for solving fast query of big data storage in intelligent video retrieval according to claim 1, characterized in that: Before performing step S4, an upper threshold is set according to the device performance, and then hotspot data is set. Frequently used cameras are cached according to different time strategies. Once the set time strategy is exceeded, data is cleared.
7. The method for solving fast query of big data storage in intelligent video retrieval according to claim 1 is characterized in that: Step S4 specifically includes: The calculation optimization is to detect the occurrence of events in a certain area of the camera screen according to the time range set when caching in Redis, and the image center point coordinates (x, y, w, h) are used to perform data filtering by the intersection of the detection area and the IoU calculation method. The calculated x, y, w, h parameter range is used as a condition to filter the x, y, w, h of all data in the cache, and then the filtered result is returned.
8. The method for solving fast query of big data storage in intelligent video retrieval according to claim 2 is characterized in that: The default value of the Elasticsearch query volume is 10,000. If it exceeds 10,000, the value of index.max_result_window is additionally set according to the amount of data queried.
9. The method for solving fast query of big data storage in intelligent video retrieval according to claim 3 is characterized in that: The Redis cache and the server memory cache are used together for optimization. The cached kv data is not a single parameter. The conversion between the json string and the data object requires serialization and deserialization. It is also time-consuming to retrieve and deserialize the structured data from the Redis, so the third-level query is required.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for solving fast query of big data storage in intelligent video retrieval as described in any one of claims 1-9 is implemented.
11. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the method for solving fast query of big data storage in intelligent video retrieval as described in any one of claims 1-9.
