Image processing method and device, equipment and storage medium

Abstract

The invention discloses an image processing method and device, equipment and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: analyzing an image processing request when an image processing control terminal receives the image processing request, so as to obtain original image data, determining an image processing mode related to the image processing request and related image processing parameters; sending the image processing mode and related image processing parameters to an image processing acceleration card for storage through an image processing control terminal, and sending an image processing start instruction containing original image data to a target GPU chip in the image processing acceleration card; and when the target GPU chip receives an image processing start instruction, performing image processing operation on the original image data in a pipeline form through the target GPU chip and other GPU chips in sequence based on the image processing mode and the corresponding image processing parameters. The image processing efficiency can be improved, and the delay can be reduced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, equipment and storage medium. Background Art

[0002] GPU (Graphics Processing Unit) is a processor specially designed for processing graphics and images. It is originally designed to accelerate graphics rendering to improve the performance and quality of computer graphics. The working principle of GPU is based on its parallel processing ability, processing multiple data elements at the same time to accelerate tasks such as matrix operations, image processing, machine learning and deep learning. The architecture of GPU usually includes graphics processing core, texture unit, rasterization unit and memory controller, etc. Each part has its own specific functions, and they work together to achieve efficient graphics rendering and computing tasks.

[0003] However, with the rapid development of artificial intelligence (AI) technology, the number of image applications involved by users is increasing day by day, including complex pattern recognition, AI mapping, convolutional neural network applications, etc., while traditional GPU image processing accelerator cards are only equipped with a single GPU chip. Due to the constraints of on-chip resources, GPU chips cannot be configured with a fairly large-scale graphics processing core. Therefore, when faced with real-time processing tasks of massive images, traditional GPU image processing accelerator cards will inevitably face task redundancy problems, resulting in ultra-high latency, and therefore cannot meet user needs.

[0004] In order to solve the above problems, it is proposed to use multiple GPU image processing accelerator cards to process massive images in real-time in parallel. However, multiple GPU image processing accelerator cards need to be coordinated and scheduled by the CPU (central processing unit) of the image processing server / PC (Personal Computer). It is difficult to implement synchronous processing and it will also cause the loss of CPU computing power resources, thus affecting the execution of other important applications. In addition, the GPU image processing accelerator card needs to be connected to the server / PC terminal through the PCIe (peripheral component interconnectexpress, a high-speed serial computer expansion bus standard) interface. However, most of the current servers / PC terminals cannot provide a sufficient number of PCIe card slots (usually only 1-2 card slots), so ideal multi-card collaboration is difficult to achieve. Summary of the invention

[0005] In view of this, the purpose of this application is to provide an image processing method, device, equipment and storage medium, which can withstand the concurrent processing of massive image tasks in a single period, realize the collaborative image processing of multiple GPU image processing accelerator cards, improve image processing efficiency and reduce latency. The specific scheme is as follows:

[0006] In a first aspect, the present application discloses an image processing method, which is applied to an image processing control terminal connected to an image processing accelerator card equipped with multiple GPU chips via a PCIe bus, comprising:

[0007] When the image processing control terminal receives an image processing request, the image processing request is parsed to obtain the original image data to be processed, and the image processing mode involved in the image processing request and image processing parameters related to the image processing mode are determined;

[0008] Sending the image processing mode and the related image processing parameters to the image processing acceleration card for storage through the image processing control terminal, and sending an image processing start instruction containing the original image data to the target GPU chip in the image processing acceleration card;

[0009] When the target GPU chip receives the image processing start instruction, the target GPU chip and other GPU chips are sequentially used to perform image processing operations on the original image data in a pipeline form based on the image processing mode and the corresponding image processing parameters to obtain an image processing result.

[0010] Optionally, the image processing control terminal includes a CPU and a main memory; the image processing acceleration card also includes a memory, and each GPU chip located in the image processing acceleration card is deployed with a preset number of image processing cores of a single category, and different GPU chips are connected via GPIO.

[0011] Optionally, when the image processing control terminal receives an image processing request, the image processing request is parsed to obtain original image data to be processed, and an image processing mode involved in the image processing request and image processing parameters related to the image processing mode are determined, including:

[0012] When the image processing control terminal receives an image processing request sent by any user terminal among the distributed user terminals, the image processing request is parsed by the CPU in the image processing control terminal to obtain the original image data to be processed, and the original image data is saved in the local main memory. The image processing mode involved in the current image processing request and the image processing parameters related to the image processing mode are then determined by the compiler.

[0013] Optionally, sending the image processing mode and the related image processing parameters to an image processing acceleration card for storage through the image processing control terminal includes:

[0014] The image processing mode and the related image processing parameters are obtained from the local main memory by the CPU in the image processing control terminal, and the image processing mode and the corresponding image processing parameters are sent to the corresponding storage partition in the memory of the image processing acceleration card through the PCIe bus for storage.

[0015] Optionally, the performing image processing operations on the original image data in a pipeline form by sequentially passing through the target GPU chip and other GPU chips and based on the image processing mode and the corresponding image processing parameters to obtain an image processing result includes:

[0016] Determine the target storage partitions corresponding to each GPU chip respectively, read the image processing mode and the corresponding image processing parameters from the target storage partitions, and then save the read image processing mode and the corresponding image processing parameters into the data cache of the corresponding GPU chip;

[0017] The target GPU chip and other GPU chips are sequentially used to perform corresponding image processing operations on the original image data in a pipeline form based on the image processing modes and the corresponding image processing parameters in the respective data cache cores to obtain image processing results.

[0018] Optionally, after performing the image processing operation on the original image data in a pipeline form to obtain the image processing result, the method further includes:

[0019] Sorting the plurality of image processing results in a preset order to obtain a sorted processing result;

[0020] Determining whether the data format of the sorted processing result is a preset data format;

[0021] If the data format of the sorted processing result is not the preset data format, the data format of the sorted processing result is converted to obtain a converted processing result, and the converted processing result is sent to the main memory of the image processing control terminal through the PCIe bus for storage.

[0022] Optionally, when the image processing control terminal receives an image processing request, the image processing request is parsed to obtain original image data to be processed, and an image processing mode involved in the image processing request and image processing parameters related to the image processing mode are determined, including:

[0023] When the image processing control terminal receives an image processing request sent by any user terminal among the distributed user terminals, the identity of the any user terminal is verified;

[0024] If the identity authentication of any user terminal is passed, obtaining the authority of any user terminal and obtaining the user authority;

[0025] If the user authority indicates that any of the user terminals has image processing authority, the image processing request is parsed to obtain the original image data to be processed, and an image processing mode involved in the image processing request and image processing parameters related to the image processing mode are determined;

[0026] Correspondingly, after performing the image processing operation on the original image data in a pipeline form to obtain the image processing result, the method further includes:

[0027] The image processing result is sent to the image processing control terminal, and the image processing result is forwarded to any one of the user terminals through the image processing control terminal.

[0028] In a second aspect, the present application discloses an image processing device, which is applied to an image processing control terminal connected to an image processing acceleration card equipped with multiple GPU chips via a PCIe bus, comprising:

[0029] a request parsing module, configured to parse an image processing request when the image processing control terminal receives the image processing request, so as to obtain the original image data to be processed, and to determine the image processing mode involved in the image processing request and the image processing parameters related to the image processing mode;

[0030] A storage and sending module, used to send the image processing mode and the related image processing parameters to the image processing acceleration card for storage through the image processing control terminal, and send an image processing start instruction containing the original image data to the target GPU chip in the image processing acceleration card;

[0031] The image processing module is used to perform image processing operations on the original image data in a pipeline form through the target GPU chip and other GPU chips in sequence and based on the image processing mode and the corresponding image processing parameters when the target GPU chip receives the image processing start instruction to obtain an image processing result.

[0032] In a third aspect, the present application discloses an electronic device, comprising a processor and a memory; wherein the processor implements the aforementioned image processing method when executing a computer program stored in the memory.

[0033] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned image processing method is implemented.

[0034] It can be seen that the present application is applied to an image processing control terminal connected to an image processing acceleration card deployed with multiple GPU chips through a PCIe bus. When the image processing control terminal receives an image processing request, the image processing request is first parsed to obtain the original image data to be processed, and the image processing mode involved in the image processing request and the image processing parameters related to the image processing mode are determined. Then, the image processing mode and the related image processing parameters are sent to the image processing acceleration card for storage through the image processing control terminal, and an image processing start instruction containing the original image data is sent to the target GPU chip in the image processing acceleration card. When the target GPU chip receives the image processing start instruction, the image processing operation is performed on the original image data in a pipeline form through the target GPU chip and other GPU chips in sequence and based on the image processing mode and the corresponding image processing parameters, to obtain an image processing result. When processing large-scale image processing tasks, the present application is specifically carried out through an image processing control terminal connected to an image processing accelerator card deployed with multiple GPU chips. The received image processing request is first parsed to obtain the original image data, and then the image processing mode and related image processing parameters involved in the image processing request are determined and sent to the image processing accelerator card for storage. Then, an image processing start instruction containing the original image data is sent to the target GPU chip in the image processing accelerator card, so that the original image data is processed accordingly in a pipeline form through the target GPU chip and other GPU chips in sequence and based on the image processing mode and the corresponding image processing parameters. It can be seen that the present application is When processing large-scale image processing tasks, the multiple stages involved in image processing are decomposed into multiple GPU chips, and image processing is performed collaboratively by multiple GPU chips. The GPU chips adopt a pipeline design, so they can withstand the concurrent processing of massive image tasks in a single period of time, and realize collaborative image processing by multiple GPU image processing accelerator cards, thereby improving image processing efficiency and reducing latency. In addition, image processing accelerator cards based on multiple GPU chips are flexible and convenient to use. They only require a single PCIe interface to connect to the image processing control terminal without adding additional peripherals. Therefore, the user experience is greatly improved. In addition, through the unified scheduling control of the image processing control terminal, the application performance of large-scale image processing can be significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0036] Figure 1 A flow chart of an image processing method disclosed in this application;

[0037] Figure 2 This is a specific application architecture diagram of a multi-GPU-based image processing acceleration card disclosed in this application;

[0038] Figure 3 A hardware structure diagram of a specific multi-GPU-based image processing acceleration card disclosed in this application;

[0039] Figure 4 A GPU core architecture diagram of a specific image processing accelerator card disclosed in this application;

[0040] Figure 5 A flowchart of a specific image processing method disclosed in this application;

[0041] Figure 6 A schematic diagram of the structure of an image processing device disclosed in this application;

[0042] Figure 7 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0044] The present application embodiment discloses an image processing method, which is applied to an image processing control terminal connected to an image processing accelerator card with multiple GPU chips deployed via a PCIe bus. Figure 1 As shown, the method includes:

[0045] Step S11: When the image processing control terminal receives an image processing request, it parses the image processing request to obtain original image data to be processed, and determines the image processing mode involved in the image processing request and image processing parameters related to the image processing mode.

[0046] It should be pointed out that the image processing solution proposed in the present application is specifically applied to an image processing control terminal, and the image processing control terminal is connected to an image processing acceleration card deployed with multiple GPU chips through a PCIe bus; wherein, the image processing acceleration card is collaboratively completed by multiple GPU chips to complete image processing tasks, and a pipeline design is adopted between different GPU chips, which can withstand the needs of concurrent processing of massive images in a single period of time, with high processing efficiency and low latency. When performing image processing, the multiple stages involved in image processing can be decomposed into multiple GPU chips for processing, thereby meeting the needs of large-scale image real-time processing in the context of the big data era, and solving the problems of high latency and low efficiency in large-scale image processing; the image processing control terminal is used to complete task scheduling to an image processing acceleration card based on multiple GPU chips, send image processing requests containing original image data, and promptly receive image processing results, and then feed back the results to the user terminal.

[0047] In this embodiment, when the image processing control terminal receives an image processing request, it first parses the image processing request to obtain the original image data to be processed, and then determines the image processing mode involved in the image processing request and the image processing parameters related to the image processing mode. Figure 2 As shown, in order to meet the remote image processing needs of multiple users, the image processing control terminal can specifically be an image processing control server or a PC terminal, and is connected to a distributed user terminal including multiple user terminals through a network (such as a wired network or a wireless network), and is also connected to an image processing accelerator card deployed with multiple GPU chips through a PCIe bus.

[0048] It should be noted that the image processing request may be sent by any user terminal in the distributed user terminals, or may be triggered by a user (such as an administrator) who logs into an image processing control terminal (such as an image processing control server or a PC terminal).

[0049] The image processing control terminal includes a CPU and a main memory; the image processing acceleration card also includes a memory, and each GPU chip located in the image processing acceleration card is deployed with a preset number of image processing cores of a single category, and different GPU chips are connected via GPIO. For details, see Figure 3As shown, the image processing control server or PC terminal is centered on the CPU and main memory, and realizes communication and information exchange with the image processing acceleration card containing 4 GPU chips of the same model (GPU1, GPU2, GPU3, GPU4) through the PCIe interface. In addition, each GPU chip of the image processing acceleration card is deployed with a preset number of single-category image processing cores to cope with large-scale concurrent image processing tasks in a single period of time. The image processing acceleration card also includes a DDR4 (Double Data Rate4) memory, as well as peripheral electronic devices such as reset switch, board power supply, clock crystal oscillator, etc., and can also include PCB (Printed Circuit Board) board and necessary resistors, capacitors and other devices. It should be pointed out that different GPU chips exchange transmission signals through GPIO. This solution can centrally coordinate large-scale image processing tasks and significantly improve the application performance of large-scale image processing through the unified scheduling control of the CPU in the image processing control server / PC terminal.

[0050] In a specific embodiment, see Figure 4As shown, the image processing accelerator card contains four GPU chips, namely GPU1, GPU2, GPU3, and GPU4, and a DDR4 memory. It should be pointed out that each GPU chip is deployed with multiple single-category image processing cores, and the cores between different chips all adopt a pipeline design. The image processing cores deployed in different GPU chips are all different, and each core can complete an image processing task only by working in coordination. Among them, GPU1 is deployed with convolution processing data cache, arbitration, and transmission kernels, convolution processing kernels 1 to convolution processing kernels n (the number of kernels n is associated with the on-chip resources of GPU1), convolution processing result cache 1 to convolution processing result cache n (the number of caches n needs to be consistent with the convolution processing kernel), and convolution processing mode and related parameter cache kernels; GPU2 is deployed with SDP single-point data processing kernels 1 to SDP single-point data processing kernels n (the number of kernels n is associated with the on-chip resources of GPU2), SDP single-point data processing result cache 1 to SDP single-point data processing result cache n (the number of caches n needs to be consistent with the SDP single-point data processing kernel), and SDP single-point processing mode and related parameter cache kernels; GPU3 is deployed with PDP plane data processing kernels 1 to PDP Plane data processing kernel n (the number of kernels n is associated with the on-chip resources of GPU3), PDP plane data processing result cache 1~PDP plane data processing result cache n (the cache number n needs to be consistent with the PDP plane data processing kernel), and PDP plane processing mode and related parameter cache kernel; GPU4 is deployed with CDP cross-channel data processing kernel 1~CDP cross-channel data processing kernel n (the number of kernels n is associated with the on-chip resources of GPU4), data Reshape engine kernel 1~data Reshape engine kernel n (the number of kernels n is consistent with the number of CDP cross-channel data processing kernels), CDP cross-channel data processing, data Reshape engine working mode and related parameter cache kernel, as well as image processing result merging, caching, arbitration, and sending kernels.

[0051] Specifically, located Figure 4 The convolution processing data cache, arbitration, and emission kernels in GPU1 are used to cache the original image data and image data processing methods, and arbitrate the allocation method according to the image scale and target convolution processing mode, and send the original image data in groups to the target convolution processing kernel; the convolution processing kernel is used to process two groups of data: one group is the "weights" of offline training (which remain unchanged between each inference), and the other group is the input "feature" data (which changes with the input of the network), and performs relevant convolution calculations on the received image data according to the target convolution processing mode and related image processing parameters required by the user (such as direct convolution mode, image input convolution mode, Winograd convolution mode, or batch convolution mode and other target convolutions).

[0052] Specifically, located Figure 4 The SDP single-point data processing kernel in GPU2 is used to apply linear and nonlinear functions to single data points, and perform linear / nonlinear calculation operations (such as ReLU, PReLU, precision scaling, batch normalization, bias addition, etc.) on the received data according to the SDP single-point processing mode and related image processing parameters required by the user.

[0053] Specifically, located Figure 4 The PDP plane data processing kernel in GPU3 is used to support specific spatial operations commonly used in CNN (convolutional neural network) applications, and performs plane data calculation operations (such as maximum pooling, minimum pooling, average pooling, etc.) on the received data according to the PDP plane processing mode and related image processing parameters required by the user.

[0054] Specifically, located Figure 4 The CDP cross-channel data processing kernel in GPU4 is used to apply the Local Response Normalization (LRN) function, and perform specific normalization function calculations on the received data according to the CDP cross-channel processing mode and related image processing parameters required by the user; the data Reshape engine kernel is used to perform specific format conversion processing (such as splitting or slicing, merging, shrinking, reshaping and transposing, etc.) on the data according to the working mode of data format conversion required by the user and the related parameters of its processing method.

[0055] in addition, Figure 4 The convolution processing result cache, SDP single-point data processing result cache, and PDP plane data processing result cache are used to cache the corresponding data processing results and send them to the lower-level kernels in sequence; the convolution processing mode and related parameter cache kernel, the SDP single-point processing mode and related parameter cache kernel, the PDP plane processing mode and related parameter cache kernel, the CDP cross-channel data processing, the data Reshape engine working mode and related parameter cache kernel are used to read and cache the image mode required by the user and the related image processing parameters corresponding to the image processing mode from the specific storage partition in the DDR4 memory; the image processing result merging, caching, arbitration, and sending kernel are used to merge and integrate the data processing results output by each data Reshape engine kernel, cache them in sequence, and output the current image processing results in sequence through the PCIe interface after format identification and arbitration of a specific mode.

[0056] Specifically, when the image processing control terminal receives an image processing request, it parses the image processing request to obtain the original image data to be processed, and determines the image processing mode involved in the image processing request and the image processing parameters related to the image processing mode. This may include: when the image processing control terminal receives an image processing request sent by any user terminal in the distributed user terminals, the image processing request is parsed by the CPU in the image processing control terminal to obtain the original image data to be processed, and the original image data is saved in the local main memory, and then the image processing mode involved in the current image processing request and the image processing parameters related to the image processing mode are determined by the compiler. For details, see Figure 3 As shown, when the image processing control server / PC terminal receives an image processing request sent by any user terminal in the distributed user terminals, the image processing request can be first parsed by the local CPU to obtain the original image data to be processed, and then the original image data is saved in the local main memory, and then the image processing mode involved in the current image processing request and the image processing parameters related to each image processing mode are determined by the compiler; wherein the image processing mode includes but is not limited to the convolution processing mode, the SDP (Single Data Point Processor) single point processing mode, the PDP (Planar Data Processor) planar processing mode, the CDP (Continuous Data Protection) cross-channel data processing mode, the data Reshape engine (i.e., data reconstruction engine, Data reshape engine) working mode, etc.

[0057] Step S12: sending the image processing mode and the related image processing parameters to the image processing acceleration card for storage through the image processing control terminal, and sending an image processing start instruction containing the original image data to the target GPU chip in the image processing acceleration card.

[0058] In this embodiment, after determining the image processing mode and related image processing parameters involved in the image processing request, further, the image processing mode and related image processing parameters are sent to the image processing acceleration card through the image processing control terminal for storage, and then an image processing start instruction containing the original image data is sent to the target GPU chip in the image processing acceleration card; wherein the target GPU chip defaults to the first acceleration card in the image processing acceleration card, such as Figure 3The GPU1 in the image processing accelerator card can also be specified as a target GPU chip from multiple GPU chips according to actual application requirements. In addition, when obtaining the image processing mode and related image processing parameters, the data scale can be obtained and saved in the image processing accelerator card.

[0059] In a specific implementation, the image processing mode and the related image processing parameters are sent to the image processing acceleration card for storage through the image processing control terminal, which may specifically include: obtaining the image processing mode and the related image processing parameters from the local main memory through the CPU in the image processing control terminal, and sending the image processing mode and the corresponding image processing parameters to the corresponding storage partition in the memory of the image processing acceleration card through the PCIe bus for storage. That is, the image processing mode and the related image processing parameters are obtained from the main memory of the image processing control terminal through the CPU, and then the image processing mode and the corresponding image processing parameters are sent to the corresponding storage partition of the memory (such as DDR4 memory) of the image processing acceleration card through the PCIe bus for storage.

[0060] Step S13: When the target GPU chip receives the image processing start instruction, the target GPU chip and other GPU chips are sequentially used to perform image processing operations on the original image data in a pipeline form based on the image processing mode and the corresponding image processing parameters to obtain an image processing result.

[0061] In this embodiment, see Figure 3 As shown, when the target GPU chip receives the image processing start instruction sent by the above-mentioned image processing control terminal, it can pass through the above-mentioned target GPU chip and other GPU chips in sequence (such as passing through GPU1 to GPU4 in sequence), and based on the above-mentioned image processing mode and corresponding image processing parameters, perform image processing operations on the original image data in a pipeline form to obtain corresponding image processing results.

[0062] Specifically, the performing image processing operations on the original image data in a pipeline form through the target GPU chip and other GPU chips in sequence and based on the image processing mode and the corresponding image processing parameters to obtain the image processing result may include: respectively determining the target storage partitions corresponding to each GPU chip, and reading the image processing mode and the corresponding image processing parameters from the target storage partitions, and then saving the read image processing mode and the corresponding image processing parameters to the data cache of the corresponding GPU chip; performing corresponding image processing operations on the original image data in a pipeline form through the target GPU chip and other GPU chips in sequence and based on the image processing mode and the corresponding image processing parameters in their respective data cache cores to obtain the image processing result. That is, first determine the target storage partition corresponding to each GPU chip, then read the stored image processing mode and the corresponding image processing parameters from the target storage partition, then save the read image processing mode and the corresponding image processing parameters into the data cache of the corresponding GPU chip, and finally perform corresponding image processing operations on the original image data in a pipeline form through the target GPU chip and other GPU chips (such as GPU1 to GPU4) in sequence and based on the image processing mode and the corresponding image processing parameters in their respective data cache cores.

[0063] For example, see Figure 4 As shown in FIG. 1 , the image processing process performed by each image processing core in a pipeline manner is specifically as follows:

[0064] 1. When GPU1 receives the image processing start signal, the convolution data cache, arbitration, and transmission core reads the image scale, convolution processing mode, and related image processing parameters involved in the current image convolution processing from the storage partition corresponding to the DDR4 memory, and then caches the original data of all input images, and arbitrates according to the image scale and target convolution processing mode, and sends the image data groups to several convolution processing cores of GPU1;

[0065] 2. Several convolution processing kernels of GPU1 read the target convolution processing mode and convolution processing related image processing parameters from the convolution processing mode and related parameter cache kernel (the cache data comes from the storage partition corresponding to the convolution processing of the DDR4 memory), and then perform the required direct convolution mode, image input convolution mode, Winograd convolution mode or batch convolution mode on the original data. The processing results are sent to the corresponding convolution processing result cache, and then sent to GPU2 in sequence through the GPIO between GPU chips;

[0066] 3. After GPU2 receives the convolution processing result, several SDP single-point data processing cores read the SDP single-point processing mode and related parameters from the SDP single-point processing mode and related parameter cache core (the cache data comes from the SDP processing corresponding storage partition of the DDR4 memory), and perform the required nonlinear calculation operations (such as ReLU (linear rectification function), PReLU (Parametric Rectified Linear Unit, an activation function), precision scaling, batch normalization, deviation addition, etc.) and related linear calculation operations (such as bias calculation, scaling calculation, etc.) on the convolution processing result, and send the processing results to the corresponding SDP single-point data processing result cache, and then send them to GPU3 in sequence through the GPIO between GPUs;

[0067] 4. After GPU3 receives the SDP single-point data processing result, several PDP plane data processing cores read the PDP plane processing mode and related image processing parameters from the PDP plane processing mode and related parameter cache core (the cache data comes from the PDP processing corresponding storage partition of the DDR4 memory), perform the required plane data calculation operations (such as maximum pooling, minimum pooling, average pooling, etc.) on the SDP processing result, and send the processing results to the corresponding PDP plane data processing result cache, and then send them to GPU4 in sequence through the GPIO between GPUs;

[0068] 5. After GPU4 receives the PDP plane data processing results, several CDP cross-channel data processing cores read the CDP cross-channel processing mode and related image processing parameters from the CDP cross-channel data processing, data Reshape engine working mode and related parameter cache core (the cache data comes from the CDP cross-channel processing and Reshape engine corresponding storage partition of the DDR4 memory), and perform the required specific normalization function calculation on the PDP processing results, and then the CDP core sends the calculation results to the corresponding Reshape engine core;

[0069] 6. After the Reshape engine core receives the CDP cross-channel data processing results, several Reshape engine cores read the working mode (contract mode, split mode, merge mode) of the Reshape engine for performing data format conversion and the related image processing parameters of its processing method from the CDP cross-channel data processing, data Reshape engine working mode and related parameter cache cores, and perform the required specific data format conversion processing (such as splitting or slicing, merging, shrinking, shaping and transposing, etc.) on the CDP processing results, and then send the processing results to the image processing result merging, caching, arbitration and sending cores through the data Reshape engine core;

[0070] 7. The image processing result merging, caching, arbitration and sending kernel merges and integrates the data processing results output by each data Reshape engine kernel, caches them in sequence, and sends the current image processing results in sequence through the PCIe interface after format identification and arbitration in a specific mode.

[0071] Furthermore, after performing image processing operations on the original image data in a pipeline form and obtaining image processing results, the method may also include: sorting multiple image processing results in a preset order to obtain sorted processing results; determining whether the data format of the sorted processing results is a preset data format; if the data format of the sorted processing results is not the preset data format, performing data format conversion on the sorted processing results to obtain converted processing results, and sending the converted processing results to the main memory of the image processing control terminal via the PCIe bus for storage. In this embodiment, after the original image data is processed, if multiple image processing results are output after passing through the last GPU chip, in order to ensure that the final output result is the result the user wants, the multiple image processing results can be sorted in a preset order to obtain a sorted processing result, and then it is determined whether the data format of the sorted processing result is a preset data format. If not, the data format of the sorted processing result can be converted to obtain a converted processing result, and then the converted processing result can be sent to the main memory of the image processing control terminal through the PCIe bus for storage, so that the CPU can read it and forward it to the user terminal that initiated the image processing request, such as sending it to Figure 2 The target terminal in the distributed user terminals in the network completes this large-scale image processing task.

[0072] It can be seen that the embodiment of the present application is applied to an image processing control terminal that is connected to an image processing acceleration card deployed with multiple GPU chips through a PCIe bus. When the image processing control terminal receives an image processing request, the image processing request is first parsed to obtain the original image data to be processed, and the image processing mode involved in the image processing request and the image processing parameters related to the image processing mode are determined. Then, the image processing mode and the related image processing parameters are sent to the image processing acceleration card for storage through the image processing control terminal, and an image processing start instruction containing the original image data is sent to the target GPU chip in the image processing acceleration card. When the target GPU chip receives the image processing start instruction, the image processing operation is performed on the original image data in a pipeline form through the target GPU chip and other GPU chips in sequence and based on the image processing mode and the corresponding image processing parameters, to obtain an image processing result. When processing large-scale image processing tasks, the present application specifically performs the processing through an image processing control terminal connected to an image processing accelerator card that is deployed with multiple GPU chips. The received image processing request is first parsed to obtain the original image data, and then the image processing mode and related image processing parameters involved in the image processing request are determined and sent to the image processing accelerator card for storage. Then, an image processing start instruction containing the original image data is sent to the target GPU chip in the image processing accelerator card, so that the original image data is processed accordingly in a pipeline form through the target GPU chip and other GPU chips in turn and based on the image processing mode and the corresponding image processing parameters. It can be seen that the embodiments of the present application When processing large-scale image processing tasks, the multiple stages involved in image processing are decomposed onto multiple GPU chips, and image processing is performed collaboratively by multiple GPU chips. A pipeline design is adopted between GPU chips, so it can withstand the concurrent processing of massive image tasks in a single period of time, and realize collaborative image processing by multiple GPU image processing accelerator cards, thereby improving image processing efficiency and reducing latency. In addition, image processing accelerator cards based on multiple GPU chips are flexible and convenient to use. Only a single PCIe interface is required to connect to the image processing control terminal without adding additional peripherals, thereby greatly improving the user experience. In addition, through the unified scheduling control of the image processing control terminal, the application performance of large-scale image processing can be significantly improved.

[0073] The present application embodiment discloses a specific image processing method, which is applied to an image processing control terminal connected to an image processing accelerator card with multiple GPU chips deployed via a PCIe bus. Figure 5 As shown, the method includes:

[0074] Step S21: When the image processing control terminal receives an image processing request sent by any user terminal among the distributed user terminals, the identity of the any user terminal is verified.

[0075] In this embodiment, in order to improve data security, during the image processing process, the identity and authority of the object initiating the image processing request can be verified. Specifically, when the image processing control terminal (such as Figure 2 When the image processing control server / PC terminal in the distributed user terminal receives an image processing request sent by any user terminal, the identity of the user terminal is first verified, such as through the login user name and password verification.

[0076] Step S22: If the identity authentication of any user terminal is passed, the authority of any user terminal is obtained to obtain the user authority.

[0077] In this embodiment, if the identity authentication of any user terminal is passed, the data processing authority of any user terminal can be further acquired to obtain the user authority.

[0078] Step S23: If the user authority indicates that any user terminal has image processing authority, the image processing request is parsed to obtain the original image data to be processed, and the image processing mode involved in the image processing request and the image processing parameters related to the image processing mode are determined.

[0079] In this embodiment, if the above-mentioned user permissions indicate that any user terminal has image processing permissions, the received image processing request is parsed to obtain the original image data to be processed, and then the image processing mode involved in the image processing request and the image processing parameters related to the image processing mode are determined.

[0080] Step S24: sending the image processing mode and the related image processing parameters to the image processing acceleration card for storage through the image processing control terminal, and sending an image processing start instruction containing the original image data to the target GPU chip in the image processing acceleration card.

[0081] Step S25: When the target GPU chip receives the image processing start instruction, the target GPU chip and other GPU chips are sequentially used to perform image processing operations on the original image data in a pipeline form based on the image processing mode and the corresponding image processing parameters to obtain an image processing result.

[0082] Step S26: sending the image processing result to the image processing control terminal, and forwarding the image processing result to any one of the user terminals through the image processing control terminal.

[0083] In this embodiment, after obtaining the image processing result, the image processing result can be sent to the above-mentioned image processing control terminal, and then the above-mentioned image processing result is forwarded to any user terminal that initiates the image processing request through the image processing control terminal, thereby completing the response to the image processing request.

[0084] For more specific processing procedures of the above steps S24 and S25, reference may be made to the corresponding contents disclosed in the above embodiments, which will not be described in detail here.

[0085] It can be seen that in the embodiment of the present application, when the image processing control terminal receives an image processing request sent by any user terminal in the distributed user terminals, the identity of the any user terminal is verified. If the identity authentication of the any user terminal is passed, the authority of the any user terminal is obtained to obtain the user authority. If the user authority indicates that the any user terminal has the image processing authority, the image processing request is parsed to obtain the original image data to be processed, and the image processing mode involved in the image processing request and the image processing parameters related to the image processing mode are determined. The image processing mode and the related image processing parameters are then sent to the image processing acceleration card through the image processing control terminal for storage, and an image processing start instruction containing the original image data is sent to the target GPU chip in the image processing acceleration card; when the target GPU chip receives the image processing start instruction, the image processing operation is performed on the original image data in a pipeline form through the target GPU chip and other GPU chips in sequence and based on the image processing mode and the corresponding image processing parameters. The embodiment of the present application can ensure the security of data in the image processing process by verifying the identity and authority of the user terminal that initiates the image processing request, thereby preventing the data from being stolen by illegal users. In addition, through an image processing acceleration card based on multiple GPU chips, only a single PCIe interface is required to connect to the image processing control terminal, without adding additional peripherals, thereby improving the user experience. In addition, a pipeline design is adopted between multiple GPU chips, which can withstand the concurrent processing of massive images in a single period of time, has high processing efficiency and low latency, and supports multi-user use, so it has a wide range of applications.

[0086] Accordingly, the embodiment of the present application also discloses an image processing device, which is applied to an image processing control terminal connected to an image processing acceleration card with multiple GPU chips deployed via a PCIe bus, see Figure 6 As shown, the device comprises:

[0087] The request parsing module 11 is used for parsing the image processing request when the image processing control terminal receives the image processing request, so as to obtain the original image data to be processed, and determine the image processing mode involved in the image processing request and the image processing parameters related to the image processing mode;

[0088] A storage and sending module 12 is used to send the image processing mode and the related image processing parameters to the image processing acceleration card for storage through the image processing control terminal, and send an image processing start instruction containing the original image data to the target GPU chip in the image processing acceleration card;

[0089] The image processing module 13 is used to perform image processing operations on the original image data in a pipeline form through the target GPU chip and other GPU chips in sequence and based on the image processing mode and the corresponding image processing parameters when the target GPU chip receives the image processing start instruction to obtain an image processing result.

[0090] Among them, the specific working processes of the above-mentioned modules can refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0091] It can be seen that the embodiment of the present application is applied to an image processing control terminal that is connected to an image processing acceleration card deployed with multiple GPU chips through a PCIe bus. When the image processing control terminal receives an image processing request, the image processing request is first parsed to obtain the original image data to be processed, and the image processing mode involved in the image processing request and the image processing parameters related to the image processing mode are determined. Then, the image processing mode and the related image processing parameters are sent to the image processing acceleration card for storage through the image processing control terminal, and an image processing start instruction containing the original image data is sent to the target GPU chip in the image processing acceleration card. When the target GPU chip receives the image processing start instruction, the image processing operation is performed on the original image data in a pipeline form through the target GPU chip and other GPU chips in sequence and based on the image processing mode and the corresponding image processing parameters, to obtain an image processing result. When processing large-scale image processing tasks, the present application specifically performs the processing through an image processing control terminal connected to an image processing accelerator card that is deployed with multiple GPU chips. The received image processing request is first parsed to obtain the original image data, and then the image processing mode and related image processing parameters involved in the image processing request are determined and sent to the image processing accelerator card for storage. Then, an image processing start instruction containing the original image data is sent to the target GPU chip in the image processing accelerator card, so that the original image data is processed accordingly in a pipeline form through the target GPU chip and other GPU chips in turn and based on the image processing mode and the corresponding image processing parameters. It can be seen that the embodiments of the present application When processing large-scale image processing tasks, the multiple stages involved in image processing are decomposed onto multiple GPU chips, and image processing is performed collaboratively by multiple GPU chips. A pipeline design is adopted between GPU chips, so it can withstand the concurrent processing of massive image tasks in a single period of time, and realize collaborative image processing by multiple GPU image processing accelerator cards, thereby improving image processing efficiency and reducing latency. In addition, image processing accelerator cards based on multiple GPU chips are flexible and convenient to use. Only a single PCIe interface is required to connect to the image processing control terminal without adding additional peripherals, thereby greatly improving the user experience. In addition, through the unified scheduling control of the image processing control terminal, the application performance of large-scale image processing can be significantly improved.

[0092] In some specific embodiments, the image processing control terminal includes a CPU and a main memory; the image processing acceleration card also includes a memory, and each GPU chip located in the image processing acceleration card is deployed with a preset number of image processing cores of a single category, and different GPU chips are connected via GPIO.

[0093] In some specific embodiments, the request parsing module 11 may specifically include:

[0094] The first request parsing unit is used to parse the image processing request through the CPU in the image processing control terminal when the image processing control terminal receives an image processing request sent by any user terminal in the distributed user terminals, so as to obtain the original image data to be processed, and save the original image data to the local main memory, and then determine the image processing mode involved in the current image processing request and the image processing parameters related to the image processing mode through the compiler.

[0095] In some specific embodiments, the storage and sending module 12 may specifically include:

[0096] An information acquisition unit, used for acquiring the image processing mode and related image processing parameters from the local main memory through the CPU in the image processing control terminal;

[0097] The information sending unit is used to send the image processing mode and the corresponding image processing parameters to the corresponding storage partition in the memory of the image processing acceleration card through the PCIe bus for storage.

[0098] In some specific embodiments, the image processing module 13 may specifically include:

[0099] A partition determination unit, used to respectively determine a target storage partition corresponding to each GPU chip;

[0100] An information reading unit, used for reading the image processing mode and the corresponding image processing parameters from the target storage partition;

[0101] An information storage unit, used for storing the read image processing mode and the corresponding image processing parameters in a data cache of a corresponding GPU chip;

[0102] The image processing unit is used to sequentially pass through the target GPU chip and other GPU chips and based on the image processing mode and the corresponding image processing parameters in the respective data cache cores, perform corresponding image processing operations on the original image data in a pipeline form to obtain image processing results.

[0103] In some specific embodiments, after the image processing module 13, the following may also be included:

[0104] A sorting unit, used to sort the plurality of image processing results according to a preset order to obtain a sorted processing result;

[0105] A judging unit, used to judge whether the data format of the sorted processing result is a preset data format;

[0106] a format conversion unit, configured to convert the data format of the sorted processed result to obtain a converted processed result if the data format of the sorted processed result is not the preset data format;

[0107] The processing result unit is used to send the converted processing result to the main memory of the image processing control terminal through the PCIe bus for storage.

[0108] In some specific embodiments, the request parsing module 11 may specifically include:

[0109] An identity authentication unit, configured to authenticate the identity of any user terminal when the image processing control terminal receives an image processing request sent by any user terminal in the distributed user terminals;

[0110] A permission acquisition unit, configured to acquire the permission of any user terminal if the identity authentication of any user terminal passes, thereby obtaining the user permission;

[0111] A second request parsing unit is configured to parse the image processing request to obtain the original image data to be processed, and determine the image processing mode involved in the image processing request and image processing parameters related to the image processing mode if the user authority indicates that any user terminal has image processing authority;

[0112] Correspondingly, after the image processing module 13, the following may also be included:

[0113] The result sending unit is used to send the image processing result to the image processing control terminal, and forward the image processing result to any one of the user terminals through the image processing control terminal.

[0114] Furthermore, the present application also discloses an electronic device. Figure 7 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0115] Figure 7The present invention provides a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the image processing method disclosed in any of the above embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0116] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0117] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0118] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and may be Windows Server, Netware, Unix, Linux, etc. In addition to computer programs that can be used to complete the image processing method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 may further include computer programs that can be used to complete other specific tasks.

[0119] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned disclosed image processing method. For the specific steps of the method, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, and no further description will be given here.

[0120] Furthermore, an embodiment of the present application also discloses a computer program product, including a computer program / instructions, which implement the steps of the image processing method disclosed above when executed by a processor.

[0121] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0122] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0123] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0124] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0125] The above is a detailed introduction to an image processing method, device, equipment and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image processing method, characterized in that: An image processing control terminal connected to an image processing accelerator card with multiple GPU chips via a PCIe bus, including: When the image processing control terminal receives an image processing request, it parses the image processing request to obtain the original image data to be processed, and determines the image processing mode involved in the image processing request and image processing parameters related to the image processing mode; Sending the image processing mode and related image processing parameters to the image processing accelerator card for storage through the image processing control terminal, and sending an image processing start instruction containing the original image data to the target GPU chip in the image processing accelerator card; When the target GPU chip receives the image processing start instruction, the target GPU chip and other GPU chips are sequentially used to perform image processing operations on the original image data in a pipeline form based on the image processing mode and the corresponding image processing parameters to obtain an image processing result.

2. The image processing method according to claim 1, wherein: The image processing control terminal includes a CPU and a main memory; the image processing acceleration card also includes a memory, and each GPU chip located in the image processing acceleration card is deployed with a preset number of single-category image processing cores, and different GPU chips are connected via GPIO.

3. The image processing method according to claim 2, wherein: When the image processing control terminal receives an image processing request, it parses the image processing request to obtain original image data to be processed, and determines an image processing mode involved in the image processing request and image processing parameters related to the image processing mode, including: When the image processing control terminal receives an image processing request sent by any user terminal among the distributed user terminals, the CPU in the image processing control terminal parses the image processing request to obtain the original image data to be processed, and saves the original image data to the local main memory. The compiler then determines the image processing mode involved in the current image processing request and the image processing parameters related to the image processing mode.

4. The image processing method according to claim 2, wherein: The sending of the image processing mode and the related image processing parameters to the image processing acceleration card for storage through the image processing control terminal includes: The image processing mode and the related image processing parameters are obtained from the local main memory by the CPU in the image processing control terminal, and the image processing mode and the corresponding image processing parameters are sent to the corresponding storage partition in the memory of the image processing accelerator card through the PCIe bus for storage.

5. The image processing method according to claim 4, characterized in that The step of sequentially performing image processing operations on the original image data in a pipeline manner through the target GPU chip and other GPU chips based on the image processing mode and the corresponding image processing parameters to obtain an image processing result includes: Determining the target storage partition corresponding to each GPU chip respectively, reading the image processing mode and the corresponding image processing parameters from the target storage partition, and then saving the read image processing mode and the corresponding image processing parameters to the data cache of the corresponding GPU chip; The target GPU chip and other GPU chips are sequentially used to perform corresponding image processing operations on the original image data in a pipeline form based on the image processing modes and corresponding image processing parameters in their respective data cache cores to obtain image processing results.

6. The image processing method according to claim 5, characterized in that After performing image processing operations on the original image data in a pipeline form to obtain image processing results, the method further includes: Sorting the plurality of image processing results in a preset order to obtain a sorted processing result; Determining whether the data format of the sorted processing result is a preset data format; If the data format of the sorted processing result is not the preset data format, the data format of the sorted processing result is converted to obtain a converted processing result, and the converted processing result is sent to the main memory of the image processing control terminal via the PCIe bus for storage.

7. The image processing method according to any one of claims 1 to 6, characterized in that: When the image processing control terminal receives an image processing request, it parses the image processing request to obtain original image data to be processed, and determines an image processing mode involved in the image processing request and image processing parameters related to the image processing mode, including: When the image processing control terminal receives an image processing request sent by any user terminal among the distributed user terminals, the image processing control terminal verifies the identity of the user terminal; If the identity authentication of any user terminal is passed, the authority of any user terminal is obtained to obtain the user authority; If the user authority indicates that any of the user terminals has image processing authority, parsing the image processing request to obtain original image data to be processed, and determining an image processing mode involved in the image processing request and image processing parameters related to the image processing mode; Accordingly, after performing image processing operations on the original image data in a pipeline form to obtain image processing results, the method further includes: The image processing result is sent to the image processing control terminal, and the image processing result is forwarded to any one of the user terminals via the image processing control terminal.

8. An image processing device, characterized in that: An image processing control terminal connected to an image processing accelerator card with multiple GPU chips via a PCIe bus, including: a request parsing module configured to parse an image processing request when the image processing control terminal receives the request, to obtain the original image data to be processed, and to determine the image processing mode involved in the request and image processing parameters related to the image processing mode; a storage and sending module, configured to send the image processing mode and related image processing parameters to the image processing accelerator card for storage via the image processing control terminal, and to send an image processing start instruction containing the original image data to the target GPU chip in the image processing accelerator card; The image processing module is used to, when the target GPU chip receives the image processing start instruction, sequentially pass through the target GPU chip and other GPU chips and, based on the image processing mode and the corresponding image processing parameters, perform image processing operations on the original image data in a pipeline form to obtain an image processing result.

9. An electronic device, characterized in that: The apparatus comprises a processor and a memory; wherein, when the processor executes the computer program stored in the memory, the image processing method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that Used to store a computer program; wherein, when the computer program is executed by a processor, the image processing method according to any one of claims 1 to 7 is implemented.

Images