Lightweight distributed retrieval method and system based on dubbo

Abstract

This invention discloses a lightweight distributed retrieval method and system based on Dubbo. The method includes: extracting unstructured vector features from image data captured and parsed by a device or video stream; storing the unstructured vector features in a feature writing node, storing them in a file with device ID and date as a partition; setting up a retrieval module including multiple retrieval nodes to provide unstructured vector feature retrieval capabilities; setting up a routing module; and setting up a distributed retrieval management module, which, when making a retrieval request, extends Dubbo's consistent hash routing mechanism, requests the retrieval scope according to device and time partitions, and sends the unit composed of device ID and day in the request to the retrieval node calculated by the corresponding route for multi-threaded requests, with each request retrieving the top N most similar items.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, specifically relating to a lightweight distributed retrieval method and system based on Dubbo. Background Technology

[0002] With the continuous development of the internet and the ongoing advancement of information technology construction in various regions, related equipment and system platforms generate massive amounts of facial, human, and vehicle feature data. The integration and analysis of this data is of paramount value to enterprises and government agencies. However, in a single-node scenario, the speed at which such massive amounts of data can be analyzed to provide accurate information retrieval services is often unsatisfactory. Problems such as long retrieval times, low hit rates, and low efficiency exist, thus creating a demand for technological improvements.

[0003] Existing technologies generally employ two approaches: vertical scaling and horizontal scaling. Vertical scaling involves upgrading server hardware resources, but it has performance limitations, and the higher the performance of the machine, the higher the price. Due to cost and resource utilization considerations, vertical scaling is limited in application. Horizontal scaling uses multiple inexpensive machines to provide services; the processing efficiency of multiple machines can be multiplied, and it can be rapidly scaled as business volume increases. However, horizontal scaling also places high demands on data management and server management, requiring the simultaneous maintenance of multiple servers and effective data management. Existing distributed systems are often overly complex and redundant, requiring significant technical learning costs and development and maintenance capabilities. Summary of the Invention

[0004] In view of the above-mentioned problems, this invention provides a lightweight distributed retrieval method and system based on Dubbo.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] One aspect of this invention provides a lightweight distributed retrieval method based on Dubbo, comprising:

[0007] Unstructured vector feature extraction is performed on the image data captured and parsed by the device or video stream;

[0008] The unstructured vector features are stored in the feature writing node, and stored as a file with the device ID and date as a segment;

[0009] The retrieval module includes multiple retrieval nodes, providing unstructured vector feature retrieval capabilities and a Dubbo-based remote program call service interface for client requests and invocation. After startup, each retrieval node registers with the registry center, which monitors and reports its status in real time. Automatic service registration and discovery are introduced, eliminating the need to hardcode retrieval node addresses. The registry center queries the IP address of the retrieval node based on the interface name and can smoothly add or delete retrieval nodes.

[0010] The routing module is configured to route requests based on the device list and time range requested, using device ID and day as a unit. Consistent hashing is then performed on each unit, ensuring that requests for the same device are routed to the same search node. Subsequent search requests within the same unit are also routed to the same search node. For the first search or when the memory feature cache expires, the search node needs to load the features into memory immediately. If the valid memory features exist within the specified time range, there is no need to reload the feature file; the results can be retrieved quickly directly from memory.

[0011] A distributed retrieval management module is set up to extend Dubbo's consistent hashing routing mechanism when making retrieval requests. The retrieval scope is divided into device and time segments. The unit composed of device ID and day is sent to the retrieval node calculated by the corresponding route for multi-threaded requests. Each request retrieves the top N similar items.

[0012] Preferably, to improve retrieval efficiency, when there are multiple retrieval nodes, a retrieval thread pool is pre-created; when making a request, each retrieval node is responsible for one retrieval request according to the routing results; the request thread waits for the return of all node request results, re-aggregates and sorts all retrieval node results, and then extracts the top N results with the highest similarity.

[0013] Preferably, the Dubbo thread sets a timeout mechanism. The rule of the timeout mechanism is that if the service provider does not return within a certain period of time, the call is considered to have failed.

[0014] Preferably, the Dubbo thread is configured with a retry mechanism. When a call fails, the retry mechanism calls the call again. If the call fails again within the configured number of calls, the request is considered abnormal and an exception is thrown.

[0015] Preferably, the unstructured feature vector includes at least one of face, body, and vehicle information.

[0016] Another aspect of this invention provides a lightweight distributed retrieval system based on Dubbo, including a device acquisition module, a feature file storage module, a retrieval module, a routing module, and a distributed retrieval management module, wherein...

[0017] The device acquisition module is used to extract unstructured vector features from the image data acquired and parsed by the device or video stream;

[0018] The feature file storage module includes multiple feature writing nodes, which are used to store the unstructured vector features and store them as a file in a way that the device ID and date are used as a segment;

[0019] The retrieval module includes multiple retrieval nodes, which provide unstructured vector feature retrieval capabilities and a Dubbo-based remote program call service interface for client requests and invocation. After the retrieval node is started, it registers with the registry center, which monitors and reports the status of the retrieval node in real time. Automatic service registration and discovery are introduced, eliminating the need to hardcode the retrieval node address. The registry center queries the IP address of the retrieval node based on the interface name and can smoothly add or delete retrieval nodes.

[0020] When making a retrieval request, the routing module uses the device ID and time range as a unit, and performs consistent hash routing based on the device ID and day. Retrieval requests for the same device are routed to the same retrieval node. Subsequent retrieval requests within the same unit are routed to the same retrieval node. When a retrieval is performed for the first time or when the memory feature cache expires, the retrieval node needs to immediately load the features into memory. If valid memory features exist within the specified time range, there is no need to reload the feature file; the results can be quickly retrieved directly from memory.

[0021] When making a retrieval request, the distributed retrieval management module extends Dubbo's consistent hashing routing mechanism. The retrieval scope is divided into device and time segments. The request sends a unit consisting of the device ID and the day to the retrieval node calculated by the corresponding route for multi-threaded requests. Each request retrieves the top N items with the highest similarity.

[0022] Preferably, to improve retrieval efficiency, when there are multiple retrieval nodes, a retrieval thread pool is pre-created; when making a request, each retrieval node is responsible for one retrieval request according to the routing results; the request thread waits for the return of all node request results, re-aggregates and sorts all retrieval node results, and then extracts the top N results with the highest similarity.

[0023] Preferably, the Dubbo thread sets a timeout mechanism. The rule of the timeout mechanism is that if the service provider does not return within a certain period of time, the call is considered to have failed.

[0024] Preferably, the Dubbo thread is configured with a retry mechanism. When a call fails, the retry mechanism calls the call again. If the call fails again within the configured number of calls, the request is considered abnormal and an exception is thrown.

[0025] Preferably, the unstructured feature vector includes at least one of face, body, and vehicle information.

[0026] The present invention has the following beneficial effects: Through the above settings, a lightweight distributed retrieval based on a distributed solution registry center and Dubbo is realized. It can solve the management problem when the retrieval server nodes are horizontally expanded, improve retrieval efficiency, reduce the learning and maintenance costs of technical personnel, and has the characteristics of convenient horizontal expansion. It can be widely used in government, rail transit and other industry scenarios. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating the steps of a lightweight distributed retrieval method based on Dubbo, according to an embodiment of the present invention.

[0028] Figure 2 This is a schematic diagram of the structure of a lightweight distributed retrieval system based on Dubbo, according to an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Reference Figure 1 The diagram shows a flowchart of the lightweight distributed retrieval method based on Dubbo according to an embodiment of the present invention, including:

[0031] S10, extract unstructured vector features from the image data captured and parsed by the device or video stream;

[0032] S20, the unstructured vector features are stored in the feature writing node, and stored as a file with device ID and date as each shard; the storage supports common NFS protocols and object storage. To improve efficiency, updates are overwritten, deletions are only marked for deletion, and file reorganization is performed during idle periods. This allows for targeted file loading during retrieval. The stored vector feature file provides loading and calling capabilities for each retrieval node, achieving decoupling between computation and storage.

[0033] S30 sets up a retrieval module including multiple retrieval nodes to provide unstructured vector feature retrieval capabilities and a Dubbo-based remote program call service interface for client requests and invocation. After the retrieval nodes are started, they are registered with the registry center, which monitors and reports the status of the retrieval nodes in real time. Automatic service registration and discovery are introduced, eliminating the need to hardcode the retrieval node addresses. The registry center queries the IP address of the retrieval node based on the interface name and can smoothly add or delete retrieval nodes.

[0034] S40 sets up a routing module. When making a retrieval request, it uses the device ID and time range as a unit to perform consistent hash routing based on the requested device list and time range. Requests for the same device are routed to the same retrieval node. Subsequent retrieval requests in the same unit are routed to the same retrieval node. When it is the first retrieval or the memory feature cache expires, the retrieval node needs to load the features into memory immediately. If the valid memory features exist within the time range, there is no need to load the feature file again, and the results can be retrieved quickly directly in memory.

[0035] The S50 module features a distributed retrieval management module. When making a retrieval request, it extends Dubbo's consistent hashing routing mechanism, sharding the retrieval scope by device and time. The request sends a unit consisting of the device ID and the day to the corresponding retrieval node calculated by the route, performing a multi-threaded request. Each request retrieves the top N most similar results. The first retrieval requires reading the feature file from disk into memory; this process can be pre-loaded when the server is idle, especially when memory space is sufficient.

[0036] In one embodiment of the present invention, in S10, the unstructured feature vector includes at least one of face, body, and vehicle information. Common devices are intelligent parsing boxes or snapshot cameras. The video stream is one of GA / T 1400, GB / T28181, or RTSP. By interfacing with devices or protocols that have parsing capabilities, the parsed images of faces, bodies, vehicles, etc., are brought in.

[0037] In one embodiment of the present invention, in S40, the routing interface extension based on Dubbo routing strategy extension calls the request to the corresponding main retrieval node through the unit calculated by consistent hashing in the request parameters. Consistent hashing is designed to handle distributed caching problems; it not only performs a hashing function but also minimizes data migration when a server crashes. Therefore, it is widely used in the routing function of state services. It ensures that requests with the same parameters are always sent to the same provider. When a provider goes offline, requests originally destined for that provider are distributed to other providers based on virtual nodes, without causing drastic changes. The routing unit consists of device ID + day: for example, if the device ID is [23, 54], that is, one device ID is 23 and another device ID is 54, and the start and end times are 2022-07-01 17:12:28-20220703 09:12:18, then the unit is [23-220701, 23-220702, 23-220703, 54-220701, 54-220702, 54-220703]. This allows for on-demand loading.

[0038] In one embodiment of the present invention, consistent hashing routing sends requests with the same parameters to the same retrieval node, with the first parameter being the default. Here, by exposing the toKey method to a custom business implementation, the business can decide whether to send requests to the same node based on specific interface or parameter values.

[0039] In one embodiment of the present invention, in order to improve retrieval efficiency, when there are multiple retrieval nodes, a retrieval thread pool is pre-created; when a request is made, each retrieval node is responsible for a retrieval request according to the routing result; the request thread waits for the request results of all nodes to be returned, and then re-aggregates and sorts all the retrieval node results before extracting the top N results with the highest similarity.

[0040] In one embodiment of the present invention, the Dubbo thread sets a timeout mechanism. The rule of the timeout mechanism is that if the service provider does not return within a certain period of time, the call is considered to have failed.

[0041] In one embodiment of the present invention, the Dubbo thread is configured with a retry mechanism. When a call fails, the retry mechanism calls the call again. If the call fails again within the configured number of calls, the request is considered abnormal and an exception is thrown.

[0042] Corresponding to the method embodiment of the present invention, this embodiment provides a lightweight distributed retrieval system based on Dubbo, including a device acquisition module 10, a feature file storage module 20, a retrieval module 30, a routing module 40, and a distributed retrieval management module 50. The device acquisition module 10 is used to extract unstructured vector features from image data acquired and parsed from devices or video streams. The feature file storage module 20 includes multiple feature writing nodes 201, used to store the unstructured vector features, storing them as a file with device ID and date as a segment. The retrieval module 30 includes multiple retrieval nodes 301, used to provide unstructured vector feature retrieval capabilities and provide a Dubbo-based remote program call service interface for client requests and invocation. After starting, the retrieval nodes 301 register with the registration center 60, which monitors and reports the status of the retrieval nodes in real time. Automatic service registration and discovery are introduced, eliminating the need to hardcode the retrieval node addresses; the registration center 60 queries the IP address of the retrieval node based on the interface name. Furthermore, it can smoothly add or delete retrieval nodes; when the routing module 40 makes a retrieval request, it uses the device ID and day as a unit based on the list of devices to be retrieved and the time range, and performs consistent hash routing based on the unit. The same device retrieval request is routed to the same retrieval node; subsequent retrieval requests in the same unit are routed to the same retrieval node. When the first retrieval or the memory feature cache expires, the retrieval node needs to load the features into memory immediately; if the valid memory features exist within the time range, there is no need to load the feature file again, and the results can be retrieved quickly directly in memory; when the distributed retrieval management module 50 makes a retrieval request, it extends Dubbo's consistent hash routing mechanism, and the retrieval range is sharded by device and time. The unit composed of device ID and day in the request is sent to the retrieval node calculated by the corresponding route for multi-threaded requests. Each request retrieves the top N similarity results.

[0043] In one embodiment of the present invention, the unstructured feature vector in the device acquisition module 10 includes at least one of face, body, and vehicle information. Common devices are intelligent parsing boxes or snapshot cameras. The video stream conforms to one of GA / T 1400, GB / T28181, or RTSP. By interfacing with devices or protocols capable of parsing, the parsed images of faces, bodies, vehicles, etc., are input.

[0044] In one embodiment of the present invention, in order to improve retrieval efficiency, when there are multiple retrieval nodes, a retrieval thread pool is pre-created; when a request is made, each retrieval node is responsible for a retrieval request according to the routing result; the request thread waits for the request results of all nodes to be returned, and then re-aggregates and sorts all the retrieval node results before extracting the top N results with the highest similarity.

[0045] In one embodiment of this invention, the routing module, based on the Dubbo routing strategy extension, implements a routing interface extension that uses the unit calculated from the consistent hash in the request parameters to call the corresponding main retrieval node. Consistent hashing is designed to handle distributed caching issues; it not only performs a hashing function but also minimizes data migration when a server crashes. Therefore, it is widely used in the routing function of state services. It ensures that requests with the same parameters are always sent to the same provider. When a provider goes offline, requests originally destined for that provider are distributed to other providers based on virtual nodes, preventing drastic changes. The routing unit consists of device ID + day: for example, if the device ID is [23, 54] and the start and end times are 2022-07-01 17:12:28-20220703 09:12:18, then the unit is [23-220701, 23-220702, 23-220703, 54-220701, 54-220702, 54-220703], which can achieve on-demand loading.

[0046] In one embodiment of the present invention, consistent hashing routing sends requests with the same parameters to the same retrieval node, with the first parameter being the default. Here, by exposing the toKey method to a custom business implementation, the business can decide whether to send requests to the same node based on specific interface or parameter values.

[0047] In one embodiment of the present invention, the Dubbo thread sets a timeout mechanism. The rule of the timeout mechanism is that if the service provider does not return within a certain period of time, the call is considered to have failed.

[0048] In one embodiment of the present invention, the Dubbo thread is configured with a retry mechanism. When a call fails, the retry mechanism calls the call again. If the call fails again within the configured number of calls, the request is considered abnormal and an exception is thrown.

[0049] The lightweight distributed retrieval method and system based on Dubbo, implemented above, realizes a lightweight distributed retrieval based on a distributed solution registry center and Dubbo. It can solve the management problem when the retrieval server nodes are horizontally expanded, improve retrieval efficiency, reduce the learning and maintenance costs of technical personnel, and has the characteristics of convenient horizontal expansion. It can be widely used in government, rail transit and other industry scenarios.

[0050] It should be understood that the exemplary embodiments described herein are illustrative and not restrictive. Although one or more embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A lightweight distributed retrieval method based on Dubbo, characterized in that, include: Unstructured vector feature extraction is performed on the image data captured and parsed by the device or video stream; The unstructured vector features are stored in the feature writing node, and stored as a file with the device ID and date as a segment; The retrieval module includes multiple retrieval nodes, providing unstructured vector feature retrieval capabilities and a Dubbo-based remote program call service interface for client requests and invocation. After starting, the retrieval nodes register with the registry center, which monitors and reports the status of the retrieval nodes in real time. Automatic service registration and discovery are introduced, eliminating the need to hardcode the retrieval node addresses. The registry center queries the IP address of the retrieval node based on the interface name and can smoothly add or delete retrieval nodes. The routing module is configured to, when making a retrieval request, route requests based on the list of devices and time range requested, using device ID and day as a unit, and perform consistent hashing based on the unit. retrieval requests for the same device are routed to the same retrieval node; subsequent retrieval requests within the same unit are routed to the same retrieval node. The retrieval node needs to load features into memory immediately upon the first retrieval or when the memory feature cache expires. If the valid memory features exist within the specified time frame, the results can be quickly retrieved directly from memory without reloading the feature file. A distributed retrieval management module is set up to extend Dubbo's consistent hashing routing mechanism when making retrieval requests. The retrieval scope is divided into device and time segments. The unit composed of device ID and day is sent to the retrieval node calculated by the corresponding route for multi-threaded requests. Each request retrieves the top N similar items. When there are multiple search nodes, a search thread pool is created in advance; when a request is made, each search node is responsible for one search request based on the routing results; the request thread waits for the request results from all nodes to be returned, and then re-aggregates and sorts all the search node results before extracting the top N results with the highest similarity.

2. The lightweight distributed retrieval method based on Dubbo as described in claim 1, characterized in that, Dubbo threads have a timeout mechanism. The rule of the timeout mechanism is that if the service provider does not return within a certain time, the call is considered to have failed.

3. The lightweight distributed retrieval method based on Dubbo as described in claim 1, characterized in that, Dubbo threads are configured with a retry mechanism. If a call fails, it will be called again. If the call fails again within the configured number of times, the request is considered abnormal and an exception is thrown.

4. The lightweight distributed retrieval method based on Dubbo as described in claim 1, characterized in that, The unstructured feature vector includes at least one of face, body, and vehicle information.

5. A lightweight distributed retrieval system based on Dubbo, characterized in that, It includes a device acquisition module, a feature file storage module, a retrieval module, a routing module, and a distributed retrieval management module, among which... The device acquisition module is used to extract unstructured vector features from the image data acquired and parsed by the device or video stream; The feature file storage module includes multiple feature writing nodes, which are used to store the unstructured vector features and store them as a file in a way that the device ID and date are used as a segment; The retrieval module includes multiple retrieval nodes, which provide unstructured vector feature retrieval capabilities and a Dubbo-based remote program call service interface for client requests and invocation. After the retrieval node is started, it registers with the registry center, which monitors and reports the status of the retrieval node in real time. Automatic service registration and discovery are introduced, eliminating the need to hardcode the retrieval node address. The registry center queries the IP address of the retrieval node based on the interface name and can smoothly add or delete retrieval nodes. When making a retrieval request, the routing module uses the device ID and time range as a unit, and performs consistent hash routing based on the unit. retrieval requests for the same device are routed to the same retrieval node. Subsequent retrieval requests within the same unit are routed to the same retrieval node. When a retrieval is performed for the first time or when the memory feature cache expires, the retrieval node needs to load the features into memory immediately. If valid memory features exist within the specified time range, there is no need to reload the feature file; the results can be retrieved quickly directly from memory. When making a retrieval request, the distributed retrieval management module extends Dubbo's consistent hashing routing mechanism. The retrieval scope is divided into device and time segments. The unit consisting of device ID and day is sent to the retrieval node calculated by the corresponding route for multi-threaded requests. Each request retrieves the top N items with the highest similarity. When there are multiple search nodes, a search thread pool is created in advance; when a request is made, each search node is responsible for one search request based on the routing results; the request thread waits for the request results from all nodes to be returned, and then re-aggregates and sorts all the search node results before extracting the top N results with the highest similarity.

6. The lightweight distributed retrieval system based on Dubbo as described in claim 5, characterized in that, Dubbo threads have a timeout mechanism. The rule of the timeout mechanism is that if the service provider does not return within a certain time, the call is considered to have failed.

7. The lightweight distributed retrieval system based on Dubbo as described in claim 5, characterized in that, Dubbo threads are configured with a retry mechanism. If a call fails, it will be called again. If the call fails again within the configured number of times, the request is considered abnormal and an exception is thrown.

8. The lightweight distributed retrieval system based on Dubbo as described in claim 5, characterized in that, The unstructured feature vector includes at least one of face, body, and vehicle information.

Images