Multi-spectral small target reasoning acceleration method and system based on segmentation calculation

Abstract

The present invention discloses a multi-spectral small target reasoning acceleration method based on segmentation calculation, comprising the following steps: constructing a multi-spectral small target recognition model, and performing segmentation calculation on the calculation tasks in the multi-spectral small target recognition model with a cutting point as the boundary; loading configuration to obtain the structure of the multi-spectral small target recognition model; collecting client information and network environment to evaluate the computing power of the client and the computing power of the server; querying and caching approximate segmentation points according to the model structure and network conditions, and recording the approximate segmentation points as cutting points; calculating the optimal segmentation points according to the model structure and network conditions and recording them as cutting points; according to the cutting points, the client and the server perform calculation reasoning respectively; the server aggregates the calculation reasoning results, and performs feature fusion and target recognition; the target recognition result is compared with the previously saved result, and the cutting point is updated according to the similarity between the two; the beneficial effects of the present invention are: accelerating model reasoning.

Description

Technical Field

[0001] The present invention relates to the field of network model acceleration and optimization, and specifically to a multi-spectral small target reasoning acceleration method and system based on segmentation calculation. Background Art

[0002] Multispectral small target recognition is achieved by combining visible light (RGB) images and infrared (IR) images, fusing the features of the two modalities to effectively identify small targets. The multispectral small target recognition model combines RGB and IR modalities, so it can maintain high accuracy in a variety of complex environments and weather conditions.

[0003] However, as the accuracy of multispectral small target recognition increases, the model parameters and the amount of computation required by the model are also increasing. The inference computing power of mobile devices is often limited, which leads to large inference delays in most multispectral small target recognition models, seriously affecting the user experience. Some studies focus on model distillation and lightweighting, but this often requires more training costs, requires training multiple models, and even requires model search to adapt the hardware to obtain the best performance. At present, most research on multispectral small target recognition is still centered on the use of high-power GPU graphics cards for calculations, and fewer methods can use low-cost acceleration of these existing multispectral small target recognition models. Another solution is to calculate on the cloud server, but as the model input is getting larger and larger, especially images and videos, this approach consumes a lot of bandwidth and energy and causes delays. In addition, although mobile devices have limited computing power, they still have unused computing power when the entire computing task is offloaded (allocated), and using these capabilities can reduce the load on the server.

[0004] With more and more research on end-cloud collaboration, the method of split computing has emerged in recent years. Split computing will cut the original model according to the calculation graph, running part of it on the client and the rest on the server. In the past, most models were serially calculated and split by layer alone, which was conducive to model splitting. However, as the model becomes more and more complex, the calculation graph of the model becomes more and more complex. It has become more and more difficult to split by layer alone, and the optimization results are getting worse and worse.

[0005] Therefore, the present invention optimizes the reasoning of the above-mentioned multi-spectral small target recognition on resource-constrained devices, and proposes a multi-spectral small target reasoning acceleration method based on segmentation calculation on resource-constrained devices; this method utilizes the characteristics of the multi-spectral small target recognition model and segmentation calculation to jointly accelerate model reasoning. This method can dynamically find the optimal parallel acceleration reasoning node according to the performance of the client and the server, so that the multi-spectral small target recognition model can be reasoned quickly. Summary of the invention

[0006] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a multi-spectral small target reasoning acceleration method and system based on segmentation calculation, which is used to solve the problem that the prior art still uses the model segmentation according to the layer alone, and cannot achieve good optimization effect in the current complex model.

[0007] To achieve the above object, the present invention provides a multi-spectral small target reasoning acceleration method based on segmentation calculation, comprising the following steps:

[0008] Step 1: Construct a multispectral small target recognition model based on the cutting points To perform cutting calculations for the computational tasks in the multi-spectral small target recognition model for the purpose of boundary;

[0009] The cutting point Obtained by the following steps 2 to 7;

[0010] Step 2: Load the configuration and obtain the structure of the multispectral small target recognition model; collect client information and network environment to evaluate the computing power of the client and the server, in order to find the optimal segmentation point for step 4;

[0011] Step 3: Query and cache approximate segmentation points based on the model structure and network conditions. The approximate segmentation points are recorded as cutting points. If no result is found, proceed to step 4;

[0012] Step 4: Calculate the optimal segmentation point based on the model structure and network conditions and record it as the cutting point ;

[0013] Step 5: According to the cutting point ,The client and server perform calculation and reasoning respectively;

[0014] Step 6: The server aggregates the calculation and reasoning results of step 5, and performs feature fusion and target recognition;

[0015] Step 7: The server compares the target recognition result of step 6 with the target recognition result of the cut point saved in the cache query function M, and updates the cut point according to the similarity between the two. .

[0016] In step 1, the multi-spectral small target recognition model formula is as follows:

[0017] ;

[0018] in, is the input, Represents the total number of layers in the model, Used to indicate the The operations performed by the layer;

[0019] No. The intermediate representation of the layer, i.e. The output of the layer is recorded as ,Will Recursive formulation, the formula is as follows:

[0020] ;

[0021] Among them, when When it is equal to 0, , That is the input;

[0022] The multi-spectral small target recognition model also includes a feature fusion model; the feature fusion model is divided into three stages: feature extraction stage, feature fusion stage and feature recognition stage;

[0023] In the feature extraction stage, the feature fusion model is cut using the characteristics of multispectral recognition. The feature fusion model is divided into the front and back parts for the boundary;

[0024] The aforementioned front-end computing tasks are computed on the client side;

[0025] The latter computing tasks are calculated on the server side;

[0026] Since the feature extraction of RGB mode and IR mode in the multi-spectral small target recognition model is separate, some tasks in the feature extraction can be performed on the server side, which can speed up the inference speed of the multi-spectral small target recognition model.

[0027] Further: the specific method of evaluating the computing power of the client and the computing power of the server in step 2 is as follows:

[0028] The computing power of the client and the server is evaluated by the floating point operations per second (FLOPS). The calculation formula for FLOPS is as follows:

[0029] ;

[0030] ;

[0031] in, Indicates the number of cores, Indicates the main frequency, Indicates the SIMD vector width of each core, Indicates the computing power of the client. Indicates the computing power of the server;

[0032] The present invention mainly uses FLOPS (floating point operations per second). When calculating FLOPS, some performance parameters must be known first, including the number of cores, the main frequency, and the SIMD vector width of each core; then, according to these parameters, corresponding formulas are used for calculation; generally speaking, the higher the FLOPS parameter, the faster the system operation speed and the stronger the processing capability. The computing capability of the client and the computing capability of the server inverted by the FLOPS parameter provide information for the analysis model structure in step 4.

[0033] Further: Step 3 obtains the cutting point The specific method is as follows:

[0034] Added the cutting cache query function M, which uses the hash algorithm to find nodes.

[0035] Calculate network bandwidth several times , and introduce hash tables to multiple network bandwidth and The corresponding cutting point information is stored for subsequent reasoning queries;

[0036] Network bandwidth Divided into N levels, using approximate storage method, only save the approximate value of network bandwidth at each level And its corresponding cutting point information;

[0037] Cutting Point Query by the following formula:

[0038] ;

[0039] In order to reduce the time of finding the split point, the present invention adds a cutting cache query function M, which uses a hash algorithm to find the split point internally; a hash table is used to store the previously calculated cut points, and the corresponding cut points are quickly found through the hash algorithm; the core idea of ​​the hash algorithm is to map the input (network status and cut point information) to a unique hash value, and then quickly find it in the hash table; the network bandwidth calculated each time The corresponding cut point information will be stored in the cache. When the system needs to infer again, it only needs to query the network bandwidth in the cache. and cutting points to quickly obtain results;

[0040] In order to speed up the query and avoid directly storing the complete network status every time, the present invention adopts an approximate storage method to save only the network bandwidth. Since directly saving the complete network status information will increase the query time, we choose to use the network bandwidth It is divided into several levels and saves the optimal cutting point at each level; specifically, we only save the network bandwidth in the cache Divided into several levels, the approximate value of network bandwidth at each level And its corresponding cutting point information.

[0041] Further: Step 4 obtains the cutting point The specific method is deduced by the following steps:

[0042] S4-1: Establish an algorithm for calculating the main time consumption, which is the time consumed on the client side, the server side, and the time required for the client data to be transmitted to the server side, that is, ; Indicates the client calculation time,

[0043] Indicates the server calculation time, Indicates data transmission time;

[0044] S4-2: According to the main time consumption algorithm, all feature extraction layers are traversed, the cut point in each feature extraction layer is calculated, and the minimum value in the calculation result is selected, which is the shortest inference time. The cut point of the RGB mode at this time is , the cut point of the IR mode is , at this time As the optimal cutting point;

[0045] Said , and They are obtained by the following formulas:

[0046] Client calculation time :

[0047] ;

[0048] in, The client side calculation amount of cutting the RGB mode at position n is shown. represents the client-side computational effort of cutting the IR modality at position m, Indicates the computing power of the client;

[0049] Server calculation time :

[0050] ;

[0051] in, Represents the total computational effort of the model, Indicates the computing power of the server;

[0052] Data transfer time :

[0053] ;

[0054] in, , They represent the data that needs to be transmitted to the server in RGB mode and IR mode after cutting, respectively, and B represents the network bandwidth;

[0055] If no approximate segmentation point can be found in step 3, the segmentation point with the shortest inference time is found through the above algorithm.

[0056] Further: the specific method of step 5 is:

[0057] According to the cutting point ,The feature fusion stage in the multi-spectral small target recognition model is divided into the front and back parts. The front part is directly calculated by the client, and the back part is calculated on the server;

[0058] when and If any of the items is 0, data transmission is performed directly;

[0059] or When it is equal to 0, it is an extreme case, which means either the network bandwidth is very high or the computing power of the device is weak. Since data transmission consumes almost no computing power, it can be regarded as the client and server inferring the model in parallel, realizing parallel reasoning optimization of the model.

[0060] Further: the specific method of step 7 is:

[0061] The server returns the target identification result to the client and updates the cache query function M. The client compares it with the processing result of the cutting node in the saved cutting point information in the hash table;

[0062] If the same node information does not exist, the data is directly inserted;

[0063] If similar nodes already exist, calculate the relative error RE between the two. The RE calculation formula is as follows:

[0064] ;

[0065] in, Indicates the target recognition result returned by the server; Indicates the processing result of the saved cutting node, that is, the target recognition result of the cutting point saved in the cache query function M; is a small constant used to avoid division by 0;

[0066] If the error Greater than the set error threshold , the latest processing result is used to replace the original cutting point.

[0067] The present invention also provides a multi-spectral small target reasoning acceleration system based on segmentation calculation, comprising: a computer readable medium;

[0068] The computer-readable medium stores the steps of the multi-spectral small target reasoning acceleration method based on segmentation calculation as described above.

[0069] The beneficial effects of the present invention are:

[0070] 1. Utilize the characteristics of the multispectral model to dynamically segment and parallelize the RGB data and IR data to accelerate model reasoning.

[0071] 2. Dynamically select and split computing nodes based on the computing power of the device, the computing power of the server, and the network conditions of the edge environment; and use the local cache approximate solution method to obtain node information, so that the network status information query process remains stable and ensures that the cache can bring about an overall performance improvement.

[0072] 3. Improve the inference speed of resource-constrained devices, improve user experience, and help users save deployment costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the embodiments of the present invention and the technical solutions in the prior art, the drawings required for use in the embodiments and the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some implementation methods recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0074] Figure 1 is a flow chart of the method of the present invention;

[0075] Figure 2 It is a schematic diagram of the cutting algorithm of the feature fusion stage of the present invention;

[0076] Figure 3 It is a schematic diagram of the specific implementation process of the present invention. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments provided by the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0078] Obviously, the drawings described below are only some examples or embodiments of the present invention. For ordinary technicians in this field, the present invention can be applied to other similar scenarios based on these drawings without any creative work.

[0079] In addition, it can be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present invention, some design, manufacturing or production changes based on the technical contents disclosed in the present invention are just conventional technical means and should not be understood as the contents disclosed in the present invention are insufficient.

[0080] If not otherwise specified, all embodiments and optional embodiments of the present invention can be combined with each other to form a new technical solution.

[0081] See also Figure 1 and Figure 3 , respectively, are a flow chart of the method of the present invention and a schematic diagram of a specific implementation process;

[0082] The steps include:

[0083] Step 1: Construct a multispectral small target recognition model based on the cutting points To perform cutting calculations for the computational tasks in the multi-spectral small target recognition model for the purpose of boundary;

[0084] Step 2: Load the configuration and obtain the structure of the multispectral small target recognition model; collect client information and network environment to evaluate the computing power of the client and the server, in order to find the optimal segmentation point for step 4;

[0085] Step 3: Query and cache approximate segmentation points based on the model structure and network conditions. The approximate segmentation points are recorded as cutting points. If no result is found, proceed to step 4;

[0086] Step 4: Calculate the optimal segmentation point based on the model structure and network conditions and record it as the cutting point ;

[0087] Step 5: According to the cutting point , the client and the server perform calculation and reasoning respectively;

[0088] Step 6: The server aggregates the calculation and reasoning results of step 5, and performs feature fusion and target recognition;

[0089] Step 7: The server compares the target recognition result of step 6 with the target recognition result of the cut point saved in the cache query function M, and updates the cut point according to the similarity between the two. ;

[0090] Specifically:

[0091] like Figure 2 As shown, the purpose of the present invention is to cut the algorithm in the feature extraction stage of the feature fusion model in the multi-spectral small target recognition model. Since the feature extraction of RGB mode and IR mode is separate in the feature extraction stage, some tasks in the feature extraction can be performed on the server side, which can speed up the reasoning speed of the multi-spectral small target recognition model. The formula of the multi-spectral small target recognition model is as follows:

[0092] ;

[0093] in, is the input, Represents the total number of layers in the model, Used to indicate the The operations performed by the layer;

[0094] No. The intermediate representation of the layer, i.e. The output of the layer is recorded as ,Will Recursive formulation, the formula is as follows:

[0095] ;

[0096] Among them, when When it is equal to 0, , That is the input;

[0097] In the feature fusion model of the multispectral small target recognition model, the feature extraction stage uses the characteristics of multispectral recognition to cut the feature fusion model, specifically by the cutting point The feature fusion model is divided into the front part and the back part as the boundary; the calculation tasks of the front part are calculated on the client side; and the calculation tasks of the back part are calculated on the server side.

[0098] Determine the cutting point , is the cutting point of RGB mode, is the cutting point of the IR mode, which is obtained by the following operations:

[0099] To prepare for the analysis of the model structure, we first evaluate the computing power of the client and the server, which is determined by the floating point operations per second FLOPS value. The calculation formula for FLOPS is as follows:

[0100] ;

[0101] ;

[0102] in, Indicates the number of cores, Indicates the main frequency, Indicates the SIMD vector width of each core, Indicates the computing power of the client. Indicates the computing power of the server.

[0103] Then add the cutting cache query function M, which uses the hash algorithm to find nodes.

[0104] Calculate network bandwidth several times , and introduce hash tables to multiple network bandwidth and The corresponding cutting point information is stored for subsequent reasoning queries;

[0105] Network bandwidth Divided into N levels, using approximate storage method, only save the approximate value of network bandwidth at each level And its corresponding cutting point information;

[0106] Cutting Point Query by the following formula:

[0107] .

[0108] If no approximate split point is found in the hash table, the algorithm will be used to find the split point with the shortest inference time. It is known that the main time consumed in the entire inference process is the client calculation time, the server calculation time, and the time required for the client data to be transmitted to the server. Therefore, the inference time calculation formula is expressed as follows:

[0109] ;

[0110] Indicates the client calculation time, Indicates the server calculation time, Indicates data transmission time;

[0111] , and They are obtained by the following formulas:

[0112] Client calculation time :

[0113] ;

[0114] in, represents the client-side computational effort of cutting the RGB modality at position n. represents the client-side computational effort of cutting the IR modality at position m, Indicates the computing power of the client;

[0115] Server calculation time :

[0116] ;

[0117] in, Represents the total computational effort of the model, Indicates the computing power of the server;

[0118] Data transfer time :

[0119] ;

[0120] in, , They represent the data that needs to be transmitted to the server in RGB mode and IR mode after cutting, respectively, and B represents the network bandwidth;

[0121] The inference time calculation formula is used to traverse all feature extraction layers, calculate the cut point in each feature extraction layer, and select the minimum value in the calculation result, which is the shortest inference time. The cut point of the RGB mode at this time is , the cut point of the IR mode is , at this time as the optimal cutting point.

[0122] According to the obtained optimal cutting point, the front part is directly calculated on the local client, and the remaining part is calculated on the server. or When it is equal to 0, it is an extreme case, which means that the network bandwidth is very high or the computing power of the device is weak. In this case, data transmission is performed directly, and data transmission consumes almost no computing power. Therefore, it can be regarded as the client and server inferring the model in parallel, realizing parallel reasoning optimization of the model.

[0123] Since the subsequent model fusion stage involves the comprehensive processing of multiple features and is not easy to perform parallel acceleration, the subsequent feature fusion and feature recognition are implemented on the server in most cases. Therefore, the calculation and reasoning results of step 5 are aggregated for the server, and feature fusion and target recognition are performed. In addition, considering that only the second half is offloaded to the server for processing instead of completely offloading the processing, there are also advantages: it can enhance user privacy, reduce the risk of transmitting sensitive data on the network, and ensure that user data is safer because only necessary information will be processed on the server.

[0124] Finally, the server compares the target recognition result of step 6 with the calculation and reasoning result of step 5, and compares the processing results of the cutting nodes in the cache query function M and the saved cutting point information in the hash table according to the similarity between the reasoning result and the target recognition result;

[0125] If the same node information does not exist, the data is directly inserted;

[0126] If similar nodes already exist, calculate the relative error RE between the two. The RE calculation formula is as follows:

[0127] ;

[0128] in, Indicates the target recognition result returned by the server; Indicates the processing results of the saved cutting nodes; is a small constant used to avoid division by 0;

[0129] If the error Greater than the set error threshold , the latest processing result is used to replace the original cutting point.

Claims

1. A multi-spectral small target reasoning acceleration method based on segmentation calculation, characterized in that: The steps include: Step 1: Construct a multispectral small target recognition model based on the cutting points To perform cutting calculations for the computational tasks in the multi-spectral small target recognition model for the purpose of boundary; The cutting point Obtained by the following steps 2 to 7; Step 2: Load the configuration and obtain the structure of the multispectral small target recognition model; collect client information and network environment to evaluate the computing power of the client and the server, in order to find the optimal segmentation point for step 4; Step 3: Query and cache approximate segmentation points based on the model structure and network conditions. The approximate segmentation points are recorded as cutting points. If no result is found, proceed to step 4; Step 4: Calculate the optimal segmentation point based on the model structure and network conditions and record it as the cutting point ; Step 5: According to the cutting point ,The client and server perform calculation and reasoning respectively; Step 6: The server aggregates the calculation and reasoning results of step 5, and performs feature fusion and target recognition; Step 7: The server compares the target recognition result of step 6 with the target recognition result of the cut point saved in the cache query function M, and updates the cut point according to the similarity between the two. ; In step 1, the multispectral small target recognition model formula is as follows: ； in, is the input, Represents the total number of layers in the model, Used to indicate the The operations performed by the layer; No. The intermediate representation of the layer, i.e. The output of the layer is recorded as ,Will Recursive formulation, the formula is as follows: ； Among them, when When it is equal to 0, , That is the input; The multi-spectral small target recognition model also includes a feature fusion model; the feature fusion model is divided into three stages: feature extraction stage, feature fusion stage and feature recognition stage; In the feature extraction stage, the feature fusion model is cut using the characteristics of multispectral recognition. The feature fusion model is divided into the front and back parts for the boundary; The aforementioned front-end computing tasks are computed on the client side; The latter computing tasks are calculated on the server side.

2. The multi-spectral small target reasoning acceleration method based on segmentation calculation according to claim 1 is characterized in that: The specific method of evaluating the computing power of the client and the computing power of the server in step 2 is as follows: The computing power of the client and the server is evaluated by the floating point operations per second (FLOPS). The calculation formula for FLOPS is as follows: ； ； in, Indicates the number of cores, Indicates the main frequency, Indicates the SIMD vector width of each core, Indicates the computing power of the client. Indicates the computing power of the server.

3. The multi-spectral small target reasoning acceleration method based on segmentation calculation according to claim 1 is characterized in that: Step 3 obtains the cutting point The specific method is as follows: Added the cutting cache query function M, which uses the hash algorithm to find nodes. Calculate network bandwidth several times , and introduce hash tables to multiple network bandwidth and The corresponding cutting point information is stored for subsequent reasoning queries; Network bandwidth Divided into N levels, using approximate storage method, only save the approximate value of network bandwidth at each level And its corresponding cutting point information; Cutting Point Query by the following formula: 。 4. The multi-spectral small target reasoning acceleration method based on segmentation calculation according to claim 1 is characterized in that: Step 4 obtains the cutting point The specific method is deduced by the following steps: S4-1: Establish an algorithm for calculating the main time consumption, which is the time consumed on the client side, the server side, and the time required for the client data to be transmitted to the server side, that is, ; Indicates the client calculation time, Indicates the server calculation time, Indicates data transmission time; S4-2: According to the main time consumption algorithm, all feature extraction layers are traversed, the cut point in each feature extraction layer is calculated, and the minimum value in the calculation result is selected, which is the shortest inference time. The cut point of the RGB mode at this time is , the cut point of the IR mode is , at this time As the optimal cutting point; Said , and They are obtained by the following formulas: Client calculation time : ； in, represents the client-side computational effort of cutting the RGB modality at position n. represents the client-side computational effort of cutting the IR modality at position m, Indicates the computing power of the client; Server calculation time : ； in, Represents the total computational effort of the model, Indicates the computing power of the server; Data transfer time : ； in, , They represent the data that need to be transmitted to the server in RGB mode and IR mode after cutting, and B represents the network bandwidth.

5. The multi-spectral small target reasoning acceleration method based on segmentation calculation according to claim 1 is characterized in that: The specific method of step 5 is: According to the cutting point ,The feature fusion stage in the multi-spectral small target recognition model is divided into the front and back parts. The front part is directly calculated by the client, and the back part is calculated on the server; when and If any of the items is 0, data transmission is performed directly.

6. The multi-spectral small target reasoning acceleration method based on segmentation calculation according to claim 1 is characterized in that: The specific method of step 7 is: The server returns the target identification result to the client and updates the cache query function , the client compares it with the processing result of the cutting node in the saved cutting point information in the hash table; If the same node information does not exist, the data is directly inserted; If similar nodes already exist, calculate the relative error RE between the two. The RE calculation formula is as follows: ； in, Indicates the target recognition result returned by the server; Indicates the processing result of the saved cut node, that is, it has been saved to the cache query function Target recognition results of the cutting points in ; is a small constant used to avoid division by 0; If the error Greater than the set error threshold , the latest processing result is used to replace the original cutting point.

7. Multi-spectral small target reasoning acceleration system based on segmentation calculation, characterized by: include: Computer readable medium, several clients and several servers; The client and the server both communicate with a computer-readable medium; The computer-readable medium stores the steps of the multi-spectral small target reasoning acceleration method based on segmentation calculation according to any one of claims 1 to 6.

Images