Data processing methods and related equipment

Abstract

This application provides a data processing method and related equipment. The method includes: when a data processing request is received, caching M pieces of data to be processed corresponding to the data processing request in a batch data cache management warehouse, where M is a positive integer; retrieving batch data from the batch data cache management warehouse in P batches, and performing batch processing on each batch of data retrieved in the P batches to obtain a processing result, where P is a positive integer, and the batch data includes N pieces of data to be processed, where N is a positive integer; if M≤N, then P=1, and let N=M; if M>N, then the batch data retrieved in the first P-1 batches in the P batches includes N pieces of data to be processed, and the batch data retrieved in the last batch in the P batches includes M-(P-1)×N pieces of data to be processed. Using this application embodiment, both small-scale and large-scale data processing needs can be met.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method and related equipment. Background Technology

[0002] The core technology of the deployment engine aims to solve the problem of real-time monitoring of specific targets in video image application scenarios. The core computing technology used for real-time monitoring of specific targets is feature value similarity analysis. For feature value similarity analysis with a scale of millions or even tens of millions, an accelerated computing platform is required, such as a graphics processing unit (GPU) or other neural network processors. For feature value similarity analysis with a smaller scale, a central processing unit (CPU) can be used for calculation.

[0003] However, such solutions can only meet the needs of independent computation and analysis on a single computing platform, and cannot reconcile the contradictions between the demands of feature value analysis (both small and large scales) and the costs of hardware and software maintenance, as well as high computational performance. In other words, existing deployment engines cannot simultaneously support feature value similarity analysis using CPUs and feature value similarity analysis using GPUs. Summary of the Invention

[0004] This application discloses a data processing method and related equipment that can meet both small-scale and large-scale data processing needs.

[0005] The first aspect of this application discloses a data processing method applied to an electronic device. The method includes: when a data processing request is received, caching M pieces of data to be processed corresponding to the data processing request in a batch data cache management warehouse, wherein M is a positive integer; retrieving batch data from the batch data cache management warehouse in P transactions, and performing batch processing on each batch of data retrieved in the P transactions to obtain a processing result, wherein P is a positive integer, the batch data includes N pieces of the data to be processed, and N is a positive integer; if M≤N, then P=1, and let N=M; if M>N, then The batch data acquired in the first P-1 of the P times includes N data items to be processed, and the batch data acquired in the last of the P times includes M-(P-1)×N data items to be processed.

[0006] In one possible implementation, the step of outputting the batch data from the batch data cache management warehouse through the batch data output management module includes: determining whether there is cached data in the batch data cache management warehouse through the batch data output management module; if there is cached data in the batch data cache management warehouse, then outputting the batch data from the batch data cache management warehouse through the batch data output management module.

[0007] A second aspect of this application discloses a data processing apparatus applied to an electronic device. The data processing apparatus includes: a caching unit, configured to cache M pieces of data to be processed corresponding to the data processing request in a batch data cache management repository when a data processing request is received, wherein M is a positive integer; and a processing unit, configured to retrieve batch data from the batch data cache management repository in P steps, and perform batch processing on each of the batch data retrieved in the P steps to obtain a processing result, wherein P is a positive integer, the batch data includes N pieces of the data to be processed, and N is a positive integer; if M ≤ N, then P = 1, and let N = M; if M > N, then... The batch data acquired in the first P-1 of the P times includes N data items to be processed, and the batch data acquired in the last of the P times includes M-(P-1)×N data items to be processed.

[0008] In one possible implementation, in the aspect of outputting the batch data from the batch data cache management warehouse through the batch data output management module, the processing unit is specifically configured to: determine whether there is cached data in the batch data cache management warehouse through the batch data output management module; if there is cached data in the batch data cache management warehouse, then output the batch data from the batch data cache management warehouse through the batch data output management module.

[0009] A third aspect of this application discloses an electronic device including a processor, a memory, a communication interface, and one or more programs stored in the memory and configured to be executed by the processor. The programs include instructions for performing steps in the method as described in any one of the first aspects of this application.

[0010] A fourth aspect of this application discloses a chip, characterized in that it includes: a processor for calling and running a computer program from a memory, causing a device on which the chip is installed to perform the method as described in any one of the first aspects of this application.

[0011] The fifth aspect of this application discloses a computer-readable storage medium, characterized in that it stores a computer program for electronic data interchange, wherein the computer program causes a computer to perform the method as described in any one of the first aspects of this application.

[0012] A sixth aspect of this application discloses a computer program product that causes a computer to perform the method as described in any one of the first aspects of this application.

[0013] As can be seen, in this embodiment, when the electronic device receives a data processing request, it first caches the M pending data items corresponding to the data processing request in the batch data cache management warehouse, where M is a positive integer; then, it retrieves batch data from the batch data cache management warehouse in P batches, and each batch of retrieved batch data includes N pending data items, where P is a positive integer and N is a positive integer; since if M≤N, then P=1, let N=M; if M>N, then The first P-1 batches of data acquired in P batches consist of N data points to be processed, and the last batch acquired in P batches consists of M-(P-1)×N data points to be processed. After P batches of data acquisition, all M data points to be processed cached in the batch data cache management repository have been retrieved. Each time batch data is retrieved from the batch data cache management repository, it is processed in batches to obtain the processing results. For different levels of data processing needs, simply changing the number of data points to be processed in each batch, i.e., changing the value of N, can achieve different levels of batch processing requirements, thus accommodating both small and large-scale data processing needs. When this batch processing is feature value similarity analysis in real-time monitoring, the data to be processed are feature values. Changing the number of feature values ​​acquired each time can achieve feature value similarity analysis of different levels, thus accommodating both small and large-scale feature value similarity analysis. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application.

[0016] Figure 2 This is a schematic diagram of the internal operating logic of a batch data cache management warehouse provided in an embodiment of this application.

[0017] Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application.

[0018] Figure 4 This is a flowchart illustrating another data processing method provided in the embodiments of this application.

[0019] Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application.

[0020] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The embodiments of this application are described below with reference to the accompanying drawings.

[0022] Please see Figure 1 , Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application. The data processing method can be applied to electronic devices and includes, but is not limited to, the following steps.

[0023] Step 101: When a data processing request is received, cache M pieces of data to be processed corresponding to the data processing request in the batch data cache management repository, where M is a positive integer.

[0024] The data processing request can be a feature value similarity analysis request, and the data to be processed can be M feature values ​​whose similarity needs to be analyzed.

[0025] It should be understood that the batch data cache management repository can be empty before caching the M pending data items corresponding to the data processing request in the batch data cache management repository, meaning that the batch data cache management repository may not store any data. Alternatively, the batch data cache management repository can also be non-empty before caching the M pending data items corresponding to the data processing request in the batch data cache management repository, but its remaining cache space must not be less than the storage space required to store the M pending data items.

[0026] Step 102: Retrieve batch data from the batch data cache management repository in P batches, and perform batch processing on each batch of data retrieved in the P batches to obtain the processing result, wherein P is a positive integer, the batch data includes N data items to be processed, and N is a positive integer; if M≤N, then P=1, and let N=M; if M>N, then The batch data acquired in the first P-1 of the P times includes N data items to be processed, and the batch data acquired in the last of the P times includes M-(P-1)×N data items to be processed.

[0027] in, This indicates rounding up the ratio of M to N. For example, when M = 99 and N = 10,

[0028] This batch processing includes batch computation, such as batch computation of feature value similarity. Specifically, when it is necessary to calculate the similarity of M feature values, the M feature values ​​are first cached in a batch data cache management repository; then, batches of feature values ​​are retrieved from the batch data cache management repository in P steps, and batch computation of feature value similarity is performed on each batch of feature values ​​retrieved in the P steps, that is, the similarity of each of the N feature values ​​retrieved each time is calculated with the other N-1 feature values, thereby realizing the batch computation of similarity of N feature values; at this time, the processing result is also the similarity analysis result of the N feature values ​​retrieved in each of the P steps.

[0029] Before caching the M pending data items corresponding to the data processing request in the batch data cache management repository, the method further includes: initializing the batch data cache management repository to determine the number of batch data items output by the batch data cache management repository each time. Specifically, for batch processing of different magnitudes, the number of batch data items output by the batch data cache management repository each time should be set first. For example, if the magnitude of the batch processing is A, that is, the maximum processing volume of a batch processing is A pending data items, then the maximum data volume output by the batch data cache management repository each time needs to be initialized and set to A.

[0030] In this embodiment, when the electronic device receives a data processing request, it first caches the M pending data items corresponding to the data processing request in a batch data cache management warehouse, where M is a positive integer; then, it retrieves batch data from the batch data cache management warehouse in P batches, and each batch of retrieved batch data includes N pending data items, where P is a positive integer and N is a positive integer; since if M≤N, then P=1, let N=M; if M>N, then The first P-1 batches of data acquired in P batches consist of N data points to be processed, and the last batch acquired in P batches consists of M-(P-1)×N data points to be processed. After P batches of data acquisition, all M data points to be processed cached in the batch data cache management repository have been retrieved. Each time batch data is retrieved from the batch data cache management repository, it is processed in batches to obtain the processing results. For different levels of data processing needs, simply changing the number of data points to be processed in each batch, i.e., changing the value of N, can achieve different levels of batch processing requirements, thus accommodating both small and large-scale data processing needs. When this batch processing is feature value similarity analysis in real-time monitoring, the data to be processed are feature values. Changing the number of feature values ​​acquired each time can achieve feature value similarity analysis of different levels, thus accommodating both small and large-scale feature value similarity analysis.

[0031] In one possible implementation, the electronic device includes a central processing unit (CPU) and a graphics processing unit (GPU); if the CPU is used to process the batch data acquired in each of the P times, then N = A, where A is the maximum amount of data that the CPU can process in a batch, and A is a positive integer; if the GPU is used to process the batch data acquired in each of the P times, then N = B, where B is the maximum amount of data that the GPU can process in a batch, and B is a positive integer.

[0032] For example, if the data processing request is a feature value similarity analysis request, and the data to be processed consists of M feature values, and the CPU is used to perform batch calculation of feature value similarity on N feature values ​​obtained in each of P operations, then N = A, where A is the maximum amount of feature values ​​that the CPU can calculate at one time when performing batch calculation of feature value similarity. If the GPU is used to perform batch calculation of feature value similarity on N feature values ​​obtained in each of P operations, then N = B, where B is the maximum amount of feature values ​​that the GPU can calculate at one time when performing batch calculation of feature value similarity.

[0033] In this embodiment, if a central processing unit (CPU) is used to process batch data, the amount of data to be processed contained in each batch of data retrieved from the batch data cache management repository is the maximum amount of data that the CPU can process in a batch. If a graphics processing unit (GPU) is used to process batch data, the amount of data to be processed contained in each batch of data retrieved from the batch data cache management repository is the maximum amount of data that the GPU can process in a batch. Since the CPU has a smaller processing capacity and the GPU has a larger processing capacity, it is possible to meet the batch data processing needs of both small and large amounts.

[0034] In one possible implementation, the batch data cache management repository includes a data cache management module; caching the M pending data corresponding to the data processing request in the batch data cache management repository includes: determining whether the batch data cache management repository is performing batch data output through the data cache management module; if the batch data cache management repository is not performing batch data output, then caching the M pending data corresponding to the data processing request in the batch data cache management repository through the data cache management module.

[0035] Specifically, in the batch data cache management repository provided in this application embodiment, data caching and data output are mutually exclusive; that is, caching data into the batch data cache management repository and reading data from the batch data cache management repository cannot be performed simultaneously. Therefore, before caching data into the batch data cache management repository, it is necessary to determine whether data caching permissions are available, that is, whether batch data output is in progress. Data caching can only be performed if data caching permissions are available.

[0036] In this embodiment, when performing data caching, the data caching management module determines whether the batch data caching management warehouse is performing batch data output. If the batch data caching management warehouse is not performing batch data output, then the M pending data corresponding to the data processing request are cached in the batch data caching management warehouse. This ensures that the pending data cached in the batch data caching management warehouse and the pending data retrieved from the batch data caching management warehouse are data corresponding to the same data processing request. In other words, the pending data cached in the batch data caching management warehouse and the pending data retrieved from the batch data caching management warehouse are data corresponding to the same scale, which helps to ensure that batch processing at each scale can be successfully performed.

[0037] In one possible implementation, caching the M pending data items corresponding to the data processing request in the batch data cache management repository through the data cache management module includes: determining whether the batch data cache management repository has remaining cache space through the data cache management module; if the batch data cache management repository has remaining cache space, then caching the M pending data items corresponding to the data processing request in the remaining cache space through the data cache management module.

[0038] In this embodiment of the application, before data caching, it is first determined whether there is remaining cache space in the batch data cache management warehouse. If there is remaining cache space in the batch data cache management warehouse, the data to be processed is cached in the remaining cache space, which is conducive to successful data caching.

[0039] In one possible implementation, the batch data cache management repository further includes a batch data output management module; if the batch data cache management repository has no remaining cache space, the method further includes: pausing the caching of M pending data corresponding to the data processing request in the batch data cache management repository; sending a batch data output instruction to the batch data output management module through the data cache management module, wherein the batch data output instruction is used to trigger the batch data output management module to perform batch data output; if the data cache management module receives a data caching instruction from the batch data output management module, then caching the M pending data corresponding to the data processing request in the batch data cache management repository through the data cache management module.

[0040] The reason why the batch data cache management warehouse has no remaining cache space may be because the amount of M data to be processed is too large. That is, the current remaining cache space of the batch data cache management warehouse cannot fully cache some or all of these M data to be processed. Data needs to be output first so that the batch data cache management warehouse has enough remaining cache space to cache some or all of these M data to be processed.

[0041] It should be understood that after receiving a batch data output instruction, the batch data output management module performs batch data output, thereby freeing up cache space. In this way, the M pending data corresponding to the data processing request can continue to be cached in the batch data cache management repository.

[0042] In this embodiment of the application, when there is no remaining cache space in the batch data cache management warehouse, the caching of data to be processed in the batch data cache management warehouse is paused first, and data output is triggered to output some or all of the data in the batch data cache management warehouse. After freeing up cache space, the M data to be processed corresponding to the data processing request are cached in the batch data cache management warehouse, which is conducive to successful data caching.

[0043] In one possible implementation, the batch data cache management repository includes a batch data output management module; obtaining batch data from the batch data cache management repository includes: determining whether the batch data cache management repository is caching data through the batch data output management module; if the batch data cache management repository is not caching data, then outputting the batch data from the batch data cache management repository through the batch data output management module.

[0044] It should be understood that, since the batch data cache management repository provided in this application embodiment has mutually exclusive data caching and data output, that is, caching data into the batch data cache management repository and reading data from the batch data cache management repository cannot be performed simultaneously. Therefore, when outputting batch data from the batch data cache management repository, it is necessary to determine whether there is data output permission, that is, to determine whether data caching is in progress. Data output is only performed if data output permission is granted.

[0045] It should be noted that batch data is output from the batch data cache management repository for batch processing; and once the batch data is output from the batch data cache management repository, the batch data will no longer exist in the batch data cache management repository. That is, when the batch data is output from the batch data cache management repository, the batch data cache management repository will delete the batch data stored therein.

[0046] In this embodiment, when outputting data, it is determined whether the batch data cache management warehouse is caching data. If the batch data cache management warehouse is not caching data, data is read from the batch data cache management warehouse. This ensures that the data to be processed cached in the batch data cache management warehouse and the data to be processed retrieved from the batch data cache management warehouse correspond to the same data processing request. In other words, the data to be processed cached in the batch data cache management warehouse and the data to be processed retrieved from the batch data cache management warehouse are data of the same magnitude, which helps to ensure that batch processing at each magnitude can be successfully performed.

[0047] In one possible implementation, the step of outputting the batch data from the batch data cache management warehouse through the batch data output management module includes: determining whether there is cached data in the batch data cache management warehouse through the batch data output management module; if there is cached data in the batch data cache management warehouse, then outputting the batch data from the batch data cache management warehouse through the batch data output management module.

[0048] It should be understood that determining whether there is cached data in the batch data cache management repository means determining whether there is part or all of the M pending data corresponding to the data processing request in the batch data cache management repository.

[0049] In this embodiment of the application, since the data to be processed can only be read from the batch data cache management warehouse when there is cached data, it is first determined whether there is cached data in the batch data cache management warehouse. Only when there is cached data in the batch data cache management warehouse will the batch data be output from it, which is conducive to successfully obtaining the batch data.

[0050] In one possible implementation, the batch data cache management repository further includes a data cache management module; if there is no cached data in the batch data cache management repository, the step of outputting the batch data from the batch data cache management repository through the data batch output management module includes: pausing the output of the batch data from the batch data cache management repository; if the data batch output management module receives a data output instruction from the data cache management module within a preset time period, then the batch data is output from the batch data cache management repository through the data batch output management module.

[0051] Specifically, for the first P retrievals of batch data from the batch data cache management repository, if there is no cached data in the repository, it means that M pending data items have not yet been cached. For subsequent P retrievals of batch data from the batch data cache management repository, if there is no cached data, it means that some of the M pending data items have not yet been cached. After the data cache management module caches data in the batch data cache management repository, it sends a data output command to the batch data output management module. Upon receiving the data output command, the batch data output management module recognizes that there is cached data in the batch data cache management repository and can continue with data output.

[0052] It should be understood that if the data batch output management module has not received a data output instruction from the data cache management module after the preset time, it may be due to an execution error in this data processing request. For example, if caching fails when the M pending data items corresponding to this data processing request are cached into the batch data cache management repository, subsequent batch processing cannot continue; or, if there is no pending data to be cached into the batch data cache management repository for this data processing request, then the batch processing corresponding to this data processing request does not need to be executed; in this case, reading the pending data corresponding to this data processing request from the batch data cache management repository will be terminated, and the execution of this data processing request will be terminated.

[0053] In this embodiment, if the batch data cache management warehouse has no cached data, then the output of batch data from it is paused; if the batch data output management module receives a data output instruction from the data cache management module within a preset time period, then the batch data cache management warehouse already has cached data, and the batch data is output from it through the batch data output management module, which is conducive to successfully obtaining batch data.

[0054] Please see Figure 2 , Figure 2This is a schematic diagram illustrating the internal operating logic of a batch data cache management repository provided in an embodiment of this application. The batch data cache management repository includes a data cache management module and a data batch output management module. The operating logic of the data cache management module, that is, caching data in the batch data cache management repository, includes, but is not limited to, the following steps:

[0055] Step 201: Determine if you have data caching permissions.

[0056] Specifically, when the data cache management module caches data in the batch data cache management repository, it first determines whether the batch data cache management repository is currently outputting data. If the batch data cache management repository is currently outputting data, it does not have data caching permissions and returns to the data caching start node; otherwise, it has data caching permissions and executes step 202.

[0057] Step 202: Determine if there is any remaining cache space.

[0058] Specifically, determine if there is remaining cache space in the batch data cache management repository; if there is remaining cache space, proceed to step 203; otherwise, proceed to steps 204 and 205. Figure 2 The solid black rectangles in the diagram represent branching nodes.

[0059] Step 203: Cache data.

[0060] Specifically, the process involves caching the data to be processed into the remaining cache space of the batch data cache management repository. After caching the data, step 207 is executed.

[0061] Step 204: Send the batch data output command.

[0062] Specifically, the data cache management module sends a data batch output instruction to the data batch output management module, triggering the data batch output instruction to execute batch data output in order to free up cache space.

[0063] Step 205: Wait for the data buffering instruction to be received.

[0064] Specifically, since the data cache management module sends a data batch output instruction to the data batch output management module, the data cache management module needs to wait for the data batch output management module to output batch data in order to free up cache space.

[0065] Step 206: Receive data caching instructions.

[0066] Specifically, the data caching instruction comes from the data batch output management module. After the data batch output management module executes batch data output and frees up cache space, it sends the data caching instruction to the data caching management module. After receiving the data caching instruction, the data caching management module knows that there is still cache space available, so it executes step 203 to cache the data.

[0067] Step 207: Determine if there is any remaining cache space.

[0068] Specifically, to ensure that the next data caching can proceed smoothly, after the current data caching is completed, the data caching management module continues to determine whether there is any remaining cache space in the batch data caching management repository; if there is no remaining cache space, it continues to execute step 204 and sends a batch data output instruction to the batch data output management module; if there is remaining cache space, it returns to the data caching start node.

[0069] The operational logic of the batch data output management module, which involves retrieving batch data from the batch data cache management repository, includes, but is not limited to, the following steps:

[0070] Step 208: Determine if there is data output permission.

[0071] Specifically, when outputting data from the batch data cache management warehouse through the batch data output management module, it is first determined whether the batch data cache management warehouse is currently caching data; if the batch data cache management warehouse is currently caching data, then there is no data output permission, and the data output start node is returned; otherwise, there is data output permission, and step 209 is executed.

[0072] Step 209: Determine if there is cached data.

[0073] Specifically, the batch data output management module determines whether there is cached data in the batch data cache management warehouse; if there is cached data, it executes step 213; if there is no cached data, it executes step 210.

[0074] Step 210: Wait for the batch output instruction to receive data.

[0075] Specifically, the batch data output instruction comes from the data cache management module. When the data cache management module determines that there is no remaining cache space in the batch data cache management repository, it will send the batch data output instruction to the batch data output management module. When the batch data output management module receives the batch data output instruction, it means that there is already cached data in the batch data cache management repository.

[0076] Step 211: Wait for timeout.

[0077] Specifically, if the data cache management module times out while waiting to receive the batch data output instruction from the data cache management module, it means that there is always remaining cache space in the batch data cache management warehouse. It may be that the data caching failed and the data to be processed corresponding to this data processing request was not cached in the batch data cache management warehouse. In this case, the data output will stop and the data output start node will be returned.

[0078] Step 212: Receive batch data output instructions.

[0079] Specifically, the data batch output management module receives data batch output instructions from the data cache management module.

[0080] Step 213, Data Output.

[0081] Specifically, if the data batch output management module receives the data batch output instruction without timeout, then the data is output.

[0082] Step 214: Send data buffering command.

[0083] Specifically, after the data batch output management module finishes outputting the data corresponding to this data output, it sends a data caching instruction to the data cache management module, thereby notifying the data cache management module that the batch data cache management warehouse has added more remaining cache space and can continue to cache data.

[0084] Step 215: Data caching instruction sent.

[0085] Specifically, after the data caching instruction is sent, the data output start node is returned.

[0086] It should be noted that, Figure 2 The description of the illustrated embodiments can also be referred to accordingly. Figure 1 The corresponding description of the method embodiments shown.

[0087] Please see Figure 3 , Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application. This data processing method can be applied to an electronic device, which includes a control engine. The data processing method includes, but is not limited to, the following steps.

[0088] Step 301: Determine the computing platform type.

[0089] Specifically, after the deployment engine starts, it first determines the type of platform currently used for computation. The purpose of determining the computation platform type is to determine the maximum number of batch computations in the batch data cache management warehouse based on the determined computation platform type.

[0090] Step 302: If the computing platform is a CPU, the maximum data volume for batch data push to the batch data cache warehouse is set to A, where A is the maximum data volume that the CPU can process in batches.

[0091] Step 303: If the computing platform is a GPU, the maximum amount of data pushed to the batch data cache warehouse is set to B, where B is the maximum amount of data that the GPU can process in batches.

[0092] It should be understood that the high performance and real-time capability of batch feature value similarity calculation on CPU and GPU computing platforms lies in batch computation. CPU or GPU computing platforms can perform 1:1 similarity calculations or X:Y similarity calculations; X:Y similarity calculation is batch computation, where X is the number of features to be calculated in the batch, and Y equals X-1. If the deployment engine receives X feature values ​​instantaneously, it can directly perform X:Y calculations, with the computation time matching that of 1:Y calculations. This achieves accelerated computation, eliminating the need to perform 1:Y calculations on each of the received X feature values ​​individually. However, the number of features to be calculated in the batch varies across different computing platforms, which is related to the characteristics of the hardware platform. To accommodate both CPU and GPU computing platforms for batch computation, the batch computation size for each platform is set during engine startup (i.e., initialization) by allocating queues. Specifically, the batch size for the CPU platform is denoted as A, with a minimum value of 20; the batch size for the GPU platform is denoted as B, where B can take any value.

[0093] It should be noted that, Figure 3 The description of the illustrated embodiments can also be referred to accordingly. Figure 1 or Figure 2 The corresponding description of the method embodiments shown.

[0094] Please see Figure 4 , Figure 4 This is a flowchart illustrating another data processing method provided in this application embodiment. The data processing method can be applied to an electronic device, which includes a control engine. The data processing method includes, but is not limited to, the following steps.

[0095] Step 401: Received feature value similarity analysis request.

[0096] Step 402: Cache M feature values ​​to be calculated to the batch data cache management repository.

[0097] Specifically, when the deployment engine receives a feature value similarity analysis request, it caches the feature values ​​to be analyzed in the batch data cache management repository.

[0098] Step 403: Determine the computing platform type.

[0099] Specifically, it determines whether the feature value similarity analysis request is executed via a CPU computing platform or a GPU computing platform.

[0100] Step 404: If the computing platform is a CPU, then push a maximum of A feature values ​​through the batch data cache management warehouse. When M is less than A, push M feature values; otherwise, push A feature values. Here, A is the maximum amount of data that the CPU can process in batches.

[0101] Step 405: The CPU computing platform completes batch calculations.

[0102] Specifically, the CPU computing platform completes batch similarity calculation of A feature values ​​to obtain A×(A-1) feature value similarities. Each of these A feature values ​​corresponds to A-1 similarities, and the A-1 similarities for each feature value are the similarity between that feature value and other A-1 feature values.

[0103] Step 406: If the computing platform is a GPU, then push up to B feature values ​​through the batch data cache management warehouse. When M is less than B, push M feature values; otherwise, push B feature values. Here, B is the maximum amount of data that the GPU can process in batches.

[0104] Step 407: The GPU computing platform completes batch calculations.

[0105] Specifically, the GPU computing platform completes batch similarity calculation of B feature values, resulting in B×(B-1) feature value similarities. Each of these B feature values ​​corresponds to B-1 similarities, and the B-1 similarities for each feature value represent the similarity between that feature value and the other B-1 feature values.

[0106] Step 408: Sort and output the calculation results.

[0107] Specifically, the similarity calculation results can be sorted and output as follows: for any feature value, output the multiple similarities corresponding to this feature value in descending order of similarity.

[0108] It should be noted that, Figure 4 The description of the illustrated embodiments can also be referred to accordingly. Figures 1 to 3 The corresponding description of the method embodiments shown.

[0109] The key to the technical solution provided in this application lies in the batch data cache management warehouse. Batch computation is achieved through this warehouse, which caches feature values ​​for calculating the similarity of feature values ​​to be computed. Then, a certain number of feature values ​​are pushed to the computing platform according to the set batch quantity. By adapting to the varying data quantities required for batch computation on different computing platforms, the batch data cache management warehouse can accommodate the different data quantity requirements of each platform. It can not only handle the computation and analysis of millions or tens of millions of feature values ​​on GPU computing platforms, but also complete the computation and analysis of smaller feature values ​​on CPU computing platforms. It is compatible with both computing platforms and can also meet high-performance computing analysis needs. This provides strong flexibility and significantly reduces maintenance and solution implementation costs. Without requiring software system replacement, it can flexibly guide basic hardware configuration according to scenario requirements and upgrade hardware configuration based on incremental scenario needs.

[0110] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0111] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a data processing device 500 provided in an embodiment of this application. The data processing device 500 is applied to an electronic device and may include a cache unit 501 and a processing unit 502. The detailed description of each unit is as follows:

[0112] The caching unit 501 is used to cache M pieces of data to be processed corresponding to the data processing request in the batch data cache management warehouse when a data processing request is received, wherein M is a positive integer;

[0113] Processing unit 502 is configured to retrieve batch data from the batch data cache management repository in P batches, and perform batch processing on each batch of data retrieved in the P batches to obtain a processing result, wherein P is a positive integer, the batch data includes N pieces of data to be processed, and N is a positive integer; if M≤N, then P=1, and let N=M; if M>N, then The batch data acquired in the first P-1 of the P times includes N data items to be processed, and the batch data acquired in the last of the P times includes M-(P-1)×N data items to be processed.

[0114] In one possible implementation, the electronic device includes a central processing unit (CPU) and a graphics processing unit (GPU); if the CPU is used to process the batch data acquired in each of the P times, then N = A, where A is the maximum amount of data that the CPU can process in a batch, and A is a positive integer; if the GPU is used to process the batch data acquired in each of the P times, then N = B, where B is the maximum amount of data that the GPU can process in a batch, and B is a positive integer.

[0115] In one possible implementation, the batch data cache management repository includes a data cache management module; regarding caching the M pending data corresponding to the data processing request in the batch data cache management repository, the cache unit 501 is specifically used to: determine whether the batch data cache management repository is performing batch data output through the data cache management module; if the batch data cache management repository is not performing batch data output, then cache the M pending data corresponding to the data processing request in the batch data cache management repository through the data cache management module.

[0116] In one possible implementation, in the step of caching the M pending data corresponding to the data processing request in the batch data cache management warehouse through the data cache management module, the caching unit 501 is specifically used to: determine whether the batch data cache management warehouse has remaining cache space through the data cache management module; if the batch data cache management warehouse has remaining cache space, then cache the M pending data corresponding to the data processing request in the remaining cache space through the data cache management module.

[0117] In one possible implementation, the batch data cache management warehouse further includes a batch data output management module; if the batch data cache management warehouse has no remaining cache space, the cache unit 501 is further configured to: pause caching the M pending data corresponding to the data processing request in the batch data cache management warehouse; send a batch data output instruction to the batch data output management module through the data cache management module, wherein the batch data output instruction is used to trigger the batch data output management module to perform batch data output; if the data cache management module receives a data cache instruction from the batch data output management module, then cache the M pending data corresponding to the data processing request in the batch data cache management warehouse through the data cache management module.

[0118] In one possible implementation, the batch data cache management warehouse includes a batch data output management module; regarding the acquisition of batch data from the batch data cache management warehouse, the processing unit 502 is specifically configured to: determine whether the batch data cache management warehouse is performing data caching through the batch data output management module; if the batch data cache management warehouse is not performing data caching, then output the batch data from the batch data cache management warehouse through the batch data output management module.

[0119] In one possible implementation, in terms of outputting the batch data from the batch data cache management warehouse through the batch data output management module, the processing unit 502 is specifically configured to: determine whether there is cached data in the batch data cache management warehouse through the batch data output management module; if there is cached data in the batch data cache management warehouse, then output the batch data from the batch data cache management warehouse through the batch data output management module.

[0120] In one possible implementation, the batch data cache management repository further includes a data cache management module; if there is no cached data in the batch data cache management repository, in terms of outputting the batch data from the batch data cache management repository through the data batch output management module, the processing unit 502 is specifically used to: pause the output of the batch data from the batch data cache management repository; if the data batch output management module receives a data output instruction from the data cache management module within a preset time period, then the batch data is output from the batch data cache management repository through the data batch output management module.

[0121] It should be noted that the implementation of each unit can also be referenced accordingly. Figures 1 to 4 The corresponding description of the method embodiments shown is provided below. Of course, the data processing apparatus 500 provided in this application embodiment includes, but is not limited to, the above-described unit modules. For example, the data processing apparatus 500 may also include a storage unit 503, which can be used to store the program code and data of the data processing apparatus 500.

[0122] exist Figure 5 In the described data processing apparatus 500, when a data processing request is received, the M pending data items corresponding to the data processing request are first cached in a batch data cache management repository, where M is a positive integer; then, batch data is retrieved from the batch data cache management repository in P installments, and each batch of retrieved batch data includes N pending data items, where P and N are positive integers; since if M≤N, then P=1, let N=M; if M>N, then... The first P-1 batches of data acquired in P batches consist of N data points to be processed, and the last batch acquired in P batches consists of M-(P-1)×N data points to be processed. After P batches of data acquisition, all M data points to be processed cached in the batch data cache management repository have been retrieved. Each time batch data is retrieved from the batch data cache management repository, it is processed in batches to obtain the processing results. For different levels of data processing needs, simply changing the number of data points to be processed in each batch, i.e., changing the value of N, can achieve different levels of batch processing requirements, thus accommodating both small and large-scale data processing needs. When this batch processing is feature value similarity analysis in real-time monitoring, the data to be processed are feature values. Changing the number of feature values ​​acquired each time can achieve feature value similarity analysis of different levels, thus accommodating both small and large-scale feature value similarity analysis.

[0123] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device 610 provided in an embodiment of this application. The electronic device 610 includes a processor 611, a memory 612 and a communication interface 613. The processor 611, the memory 612 and the communication interface 613 are interconnected through a bus 614.

[0124] The memory 612 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), and is used for related computer programs and data. The communication interface 613 is used for receiving and sending data.

[0125] The processor 611 can be one or more central processing units (CPUs). If the processor 611 is a CPU, the CPU can be a single-core CPU or a multi-core CPU.

[0126] The processor 611 in the electronic device 610 is used to read the computer program code stored in the memory 612 and execute it. Figures 1 to 4 Any of the methods shown in the table.

[0127] It should be noted that the implementation of each operation can also be referenced accordingly. Figures 1 to 4The corresponding description of the method embodiments shown.

[0128] exist Figure 6 In the described electronic device 610, when a data processing request is received, the M pending data items corresponding to the data processing request are first cached in a batch data cache management warehouse, where M is a positive integer; then, batch data is retrieved from the batch data cache management warehouse in P installments, and each batch of retrieved batch data includes N pending data items, where P and N are positive integers; since if M≤N, then P=1, let N=M; if M>N, then... The first P-1 batches of data acquired in P batches consist of N data points to be processed, and the last batch acquired in P batches consists of M-(P-1)×N data points to be processed. After P batches of data acquisition, all M data points to be processed cached in the batch data cache management repository have been retrieved. Each time batch data is retrieved from the batch data cache management repository, it is processed in batches to obtain the processing results. For different levels of data processing needs, simply changing the number of data points to be processed in each batch, i.e., changing the value of N, can achieve different levels of batch processing requirements, thus accommodating both small and large-scale data processing needs. When this batch processing is feature value similarity analysis in real-time monitoring, the data to be processed are feature values. Changing the number of feature values ​​acquired each time can achieve feature value similarity analysis of different levels, thus accommodating both small and large-scale feature value similarity analysis.

[0129] This application also provides a chip, which includes at least one processor, a memory, and an interface circuit. The memory, the transceiver, and the at least one processor are interconnected via circuits. The at least one memory stores a computer program. When the computer program is executed by the processor... Figures 1 to 4 The method flow shown in either of the above can be implemented.

[0130] This application also provides a computer-readable storage medium storing a computer program, which, when run on a computer... Figures 1 to 4 The method flow shown in either of the above can be implemented.

[0131] This application also provides a computer program product, which, when run on a computer, provides a more convenient and efficient way to run such a program. Figures 1 to 4 The method flow shown in either of the above can be implemented.

[0132] It should be understood that the processor mentioned in the embodiments of this application can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0133] It should also be understood that the memory mentioned in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0134] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, the memory (storage module) is integrated into the processor.

[0135] It should be noted that the memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.

[0136] It should also be understood that the first, second, third, fourth and various numerical designations used herein are merely for descriptive convenience and are not intended to limit the scope of this application.

[0137] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0138] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0139] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0140] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0141] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0142] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0143] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0144] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods shown in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0145] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0146] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0147] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, Applied to an electronic device, the electronic device including a central processing unit and a graphics processing unit, the method includes: When a data processing request is received, M pieces of data to be processed corresponding to the data processing request are cached in the batch data cache management repository, where M is a positive integer; Determine the computing platform type to determine the maximum data volume N of the batch data push to the batch data cache management warehouse; if the computing platform type is the central processing unit, then N=A, where A is the maximum data volume that the central processing unit can process in batches, and A is a positive integer; if the computing platform type is the graphics processing unit, then N=B, where B is the maximum data volume that the graphics processing unit can process in batches, and B is a positive integer. Batch data is retrieved from the batch data cache management repository in P batches, and batch processing is performed on each batch of data retrieved in the P batches to obtain the processing result, wherein P is a positive integer, the batch data includes N data to be processed, and N is a positive integer; If M≤N, then P=1, and let N=M; if M>N, then P=⌈M / N⌉, the batch data obtained in the first P-1 times of the P times includes N data to be processed, and the batch data obtained in the last time of the P times includes M-(P-1)×N data to be processed.

2. The method according to claim 1, characterized in that, The batch data cache management repository includes a data cache management module; caching the M pending data items corresponding to the data processing request in the batch data cache management repository includes: The data cache management module determines whether the batch data cache management repository is performing batch data output. If the batch data cache management repository is not performing batch data output, the data cache management module caches the M pending data corresponding to the data processing request in the batch data cache management repository.

3. The method according to claim 2, characterized in that, The step of caching the M pending data items corresponding to the data processing request in the batch data cache management repository through the data cache management module includes: The data cache management module determines whether there is remaining cache space in the batch data cache management repository. If the batch data cache management repository has remaining cache space, the data cache management module will cache the M pending data corresponding to the data processing request in the remaining cache space.

4. The method according to claim 3, characterized in that, The batch data cache management repository also includes a batch data output management module; if the batch data cache management repository has no remaining cache space, the method further includes: Pause caching of the M pending data items corresponding to the data processing request in the batch data cache management repository; The data cache management module sends a data batch output instruction to the data batch output management module, wherein the data batch output instruction is used to trigger the data batch output management module to perform data batch output; If the data cache management module receives a data cache instruction from the data batch output management module, it caches the M pending data corresponding to the data processing request in the batch data cache management repository through the data cache management module.

5. The method according to claim 1, characterized in that, The batch data cache management repository includes a batch data output management module; obtaining batch data from the batch data cache management repository includes: The data batch output management module determines whether the batch data cache management repository is caching data. If the batch data cache management repository is not caching data, the batch data is output from the batch data cache management repository through the batch data output management module.

6. The method according to claim 5, characterized in that, The batch data cache management repository further includes a data cache management module; if there is no cached data in the batch data cache management repository, the step of outputting the batch data from the batch data cache management repository through the data batch output management module includes: Pause the output of the batch data from the batch data cache management repository; If the data batch output management module receives a data output instruction from the data cache management module within a preset time period, the batch data will be output from the batch data cache management warehouse through the data batch output management module.

7. A data processing apparatus, characterized in that, Applied to an electronic device, the electronic device including a central processing unit and a graphics processing unit, the data processing device includes: A caching unit is used to cache M pieces of data to be processed corresponding to the data processing request in the batch data cache management repository when a data processing request is received, wherein M is a positive integer; The processing unit is used to determine the computing platform type in order to determine the maximum data volume N of the batch data pushed to the batch data cache management warehouse; if the computing platform type is the central processing unit, then N=A, where A is the maximum data volume that the central processing unit can process in batches, and A is a positive integer; if the computing platform type is the graphics processing unit, then N=B, where B is the maximum data volume that the graphics processing unit can process in batches, and B is a positive integer. Batch data is retrieved from the batch data cache management repository in P batches, and batch processing is performed on each batch of data retrieved in the P batches to obtain the processing result, wherein P is a positive integer, the batch data includes N data to be processed, and N is a positive integer; If M≤N, then P=1, and let N=M; if M>N, then P=⌈M / N⌉, the batch data obtained in the first P-1 times of the P times includes N data to be processed, and the batch data obtained in the last time of the P times includes M-(P-1)×N data to be processed.

8. An electronic device, characterized in that, It includes a processor, a memory, a communication interface, and one or more programs, said one or more programs being stored in the memory and configured to be executed by the processor, said programs including instructions for performing the steps of the method as claimed in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, It stores a computer program for electronic data interchange, wherein the computer program causes the computer to perform the method as described in any one of claims 1-6.

Images