Heterogeneous platform data transmission method for accelerating ultrahigh resolution image detection

Abstract

According to the heterogeneous platform data transmission method for accelerating ultrahigh-resolution image detection, a central processing unit (CPU) serves as a main processor module, one or more neural network processing units (NPU) serve as a detection system of a coprocessor module, and the modules are in data communication through PCIE channels so as to accelerate the target detection process. The method specifically comprises the steps that after an ultrahigh-resolution image is read and subjected to sliding window segmentation in a CPU, all slice image data are sequentially transmitted to an NPU for target detection through a PCIE channel, then target detection data are returned to the CPU through the PCIE channel, and final repeated detection frame elimination work and target detection visualization work are completed in the CPU. Different computing performance characteristics of the CPU and the NPU can be exerted, rapid data transmission between modules is completed through a PCIE channel, and accelerated processing of the ultrahigh-resolution image target detection process is effectively achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent computing, and in particular relates to a heterogeneous platform data transmission method for accelerating ultra-high resolution image detection. Background Art

[0002] Ultra-high-resolution images can provide more detailed original scene details and have a wide range of application scenarios in digital imaging technology, remote sensing technology, and medical image analysis technology. With the rapid development of neural network models and the emergence of neural network processing unit NPU chips, it has become a necessary technical means to achieve ultra-high-resolution image target detection through neural network models in more efficient NPU chips.

[0003] At present, most NPUs need to be used in conjunction with CPUs to form an intelligent computing platform with CPU as the main processor and NPU as a coprocessor. Due to the limited memory and computing power of NPU, and in order to prevent the loss of information of small targets in ultra-high-resolution images, it is first necessary to perform sliding window cropping of ultra-high-resolution images in the CPU to form multiple sub-images, and then perform target detection on each sub-image in the NPU. Finally, all detection data is fused on the original image in the CPU and duplicate detection frames are eliminated. Faced with ultra-high-resolution images with a large amount of input data, and the data cross-device transmission process that is bound to occur during the entire detection process, there is repetitive data reading and handling, which reduces the data transmission efficiency of heterogeneous platforms. Summary of the invention

[0004] In view of this, the heterogeneous platform data transmission method for accelerating ultra-high resolution image detection of the present invention solves the technical problem that the existing method causes the heterogeneous platform data transmission efficiency to be reduced.

[0005] A heterogeneous platform data transmission method for accelerating ultra-high resolution image detection is applicable to the detection of airborne image data. The airborne heterogeneous platform includes a central processing unit and a neural network processor. The central processing unit and multiple neural network processors are connected via a PCIE bus. The heterogeneous platform data transmission method includes the following steps:

[0006] S1: reading the original ultra-high resolution image in the central processing unit and performing image preprocessing, wherein the image preprocessing includes dividing the image into a plurality of slice images according to the pixel positions through a sliding window, wherein the sliding window may define a cropping size and an overlap ratio;

[0007] S2: All the sliced ​​images are sequentially transmitted to a plurality of neural network processors of the same type through the PCIE channel in the order of segmentation, and the neural network processors perform parallel calculations; the PCIE channel can mount a plurality of neural network processors of the same type, and the specific number of neural network processors is determined according to the specifications of the high-resolution image;

[0008] S3: Each neural network processor reads the received slice image and calls the neural network processor to perform target detection on the slice image;

[0009] S4: After detecting the target in the slice image, target detection data is generated, and the target detection data is returned to the memory of the central processing unit through the PCIE channel;

[0010] S5: merging the detection results of all slice images and filtering them using non-maximum suppression processing to eliminate repeated detection frames in the overlapping areas of the slice images;

[0011] S6: After removing the duplicate detection frames, all detection frames are marked in the original ultra-high resolution image to achieve visualization of the target detection results of the ultra-high resolution image, and all the final detection frames are drawn in the original ultra-high resolution image in the central processing unit.

[0012] The technical beneficial effects of the present invention are:

[0013] The present invention integrates the computing characteristics of two different types of processors, CPU and NPU. In an intelligent computing platform with CPU as the main processor and NPU as the coprocessor, on the premise of fully utilizing the computing resources of the neural network processing unit at this stage, the present invention realizes the rapid cross-device transmission of data between the main processor module and the coprocessor module through the PCIE channel, while avoiding the repetitive data handling process within the same device, improving the processing efficiency of the ultra-high resolution image target detection process, realizing the accelerated processing of ultra-high resolution image target detection, and completing the comprehensive and accurate detection of the target. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0015] Figure 1 It is a logical framework diagram of the method of the present invention. DETAILED DESCRIPTION

[0016] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0017] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0018] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0019] like Figure 1 The heterogeneous platform data transmission method for accelerating ultra-high resolution image detection is applicable to the detection of airborne image data. The airborne heterogeneous platform includes a central processing unit (CPU) and a neural network processor (NPU). The central processing unit and multiple neural network processors are connected via a PCIE bus. The heterogeneous platform data transmission method includes the following steps:

[0020] S1: Read the original ultra-high resolution image in the central processing unit and perform image preprocessing. The image preprocessing includes dividing the image into multiple slice images according to the pixel position through a sliding window. The sliding window can define the cropping size and overlap ratio;

[0021] S2: All slice images are transmitted to multiple neural network processors of the same type through the PCIE channel in the order of segmentation, and the neural network processors perform parallel calculations; the PCIE channel can mount multiple neural network processors of the same type, and the specific number of neural network processors is determined according to the specifications of the high-resolution image;

[0022] S3: Each neural network processor reads the slice image and calls the neural network processor to perform target detection on the slice image. The target detection can identify the specific content in the image, for example, the ocean, mountain island, ship or other;

[0023] S4: After detecting the target in the slice image, target detection data is generated and returned to the memory of the central processing unit through the PCIE channel;

[0024] S5: The detection results of all slice images are merged and filtered using non-maximum suppression (NMS) processing to eliminate duplicate detection boxes in the overlapping areas of the slice images, that is, the pixel coordinate conversion of the target position from relative to the slice image to relative to the original super-high resolution image, and the final NMS processing of all target detection data;

[0025] S6: After removing the duplicate detection frames, all detection frames are marked in the original ultra-high resolution image to achieve visualization of the target detection results of the ultra-high resolution image, and all the final detection frames are drawn in the original ultra-high resolution image in the central processing unit, and the visualization results are output.

[0026] A detection system with a central processing unit (CPU) as the main processor module and one or more neural network processing units (NPUs) as coprocessor modules, wherein data communication is performed between modules through a PCIE channel to accelerate the target detection process. Specifically, after reading and sliding window segmentation of ultra-high resolution images in the CPU, all slice image data are sequentially transmitted to the NPU through the PCIE channel for target detection, and then the target detection data is returned to the CPU through the PCIE channel, and the final duplicate detection frame elimination and target detection visualization are completed in the CPU. The present invention can give play to the different computing performance characteristics of the CPU and the NPU, complete the rapid data transmission between modules through the PCIE channel, and effectively realize the accelerated processing of the ultra-high resolution image target detection process.

[0027] As a specific implementation method provided in this case, S2 to S4 include data transmission and processing based on parallel algorithms. In the central processing unit, by establishing a multi-threaded method, ultra-high-resolution image cropping, slice image sending to the neural network processor, and the central processing unit receiving slice image detection data are simultaneously performed; in the neural network processor, by establishing a multi-threaded method, slice image reception, slice image intelligent processing, and slice image detection data are returned simultaneously.

[0028] As a specific implementation method provided in this case, in step S1, OpenCV is used to read the ultra-high resolution image, and the RGB pixel values ​​of the image are stored in the memory of the central processing unit. The effect is: to facilitate the drawing of the target detection frame in the post-processing process, and to avoid multiple readings of the ultra-high resolution image;

[0029] The pixel coordinates of all the segmentation points in the image are stored in the memory of the central processing unit in the form of an array in the order of segmentation (the coordinates are unified as the upper left pixel of the slice image). This is to facilitate the conversion of the target detection frame coordinates in the post-processing process. The cropping size of the sliding window is determined according to the input size of the neural network model, and the overlap ratio is set between 10% and 20%. That is, the ultra-high resolution image is opened using OpenCV, and the RGB pixel values ​​of the image are stored in the CPU memory to facilitate the drawing of the target detection frame in the post-processing process, avoiding multiple loading of the ultra-high resolution image in the memory; at the same time, the pixel coordinates of all the segmentation points in the image are stored in the memory in the form of an array in the order of segmentation (the coordinates are unified as the upper left pixel of the slice image), so as to facilitate the conversion of the target detection frame coordinates in the post-processing process. The cropping size should be determined by the input size of the neural network model used later.

[0030] As a specific implementation method provided in this case, step S2 includes the central processor transmitting the slice image to the neural network processor through the PCIE channel. The PCIE channel can mount multiple neural network processors of the same type, and the specific number of neural network processors is determined according to the specifications of the high-resolution image, further realizing the accelerated processing of ultra-high-resolution image target detection.

[0031] As a specific implementation method provided in this case, step S4 includes:

[0032] The CPU reads the target detection data of the slice image from the specified neural network processor source memory address through the PCIE channel. The target detection data includes the coordinates (x1, y1) of the target detection frame, the width and height of the target detection frame, and the category and confidence of the target. The upper left corner pixel coordinates are used as the coordinate origin, and the coordinates (x1, y1) are used as the relative position relationship between the target position and the coordinate origin. For example, the target ship is at the position x1 east and y1 south of the upper left corner pixel position, and the coordinates of the target ship are (x1, y1);

[0033] As a specific implementation method provided in this case, step S5 includes:

[0034] The target detection data received by the central processor in S4 is converted. The target position is converted from the pixel coordinates relative to the slice image to the original ultra-high resolution image, and the duplicate detection frame is eliminated. For example, if the upper left corner coordinates of the target detection frame on the slice image are (x1, y1), and the upper left corner coordinates of the slice image in the original ultra-high resolution image are (x2, y2), then the upper left corner coordinates of the target detection frame in the original ultra-high resolution image are (x1+x2, y1+y2). The width and height of the target detection frame remain unchanged. After completing the coordinate conversion of all detection frames, the duplicate detection frames are eliminated by the NMS method.

[0035] As a specific implementation method provided in this case, step S6 includes drawing the target detection data obtained in step S5 in the original image read in step S1 and outputting a visualization result.

[0036] Secondly, a system for accelerating ultra-high resolution image target detection is provided, including:

[0037] The main processor module is responsible for the overall scheduling of the ultra-high resolution image detection process, including the preprocessing of ultra-high resolution images, the data transmission of slice images, the reception of target detection data and post-processing, and the visualization of detection results. Specifically, it is used to read the original ultra-high resolution image in the central processor and perform image preprocessing. Image preprocessing includes dividing the image into multiple slice images according to the pixel position through a sliding window. The sliding window can define the cropping size and overlap ratio.

[0038] The detection results of all slice images are merged and filtered using non-maximum suppression (NMS) processing to eliminate repeated detection frames in the overlapping areas of the slice images;

[0039] After removing duplicate detection frames, all detection frames are marked in the original ultra-high resolution image to achieve visualization of the target detection results in the ultra-high resolution image.

[0040] Coprocessor module: used to transfer all slice images to multiple neural network processors of the same type through the PCIE channel in the order of slicing, and the neural network processors perform parallel calculations; the PCIE channel can mount multiple neural network processors of the same type, and the specific number of neural network processors is determined according to the specifications of the high-resolution image;

[0041] Each neural network processor reads the slice image in the memory of the central processing unit and calls the runtime interface of the neural network processor to perform target detection on the slice image; through target detection, it can identify the specific content in the image, and generate target detection data after slice image target detection, and return the target detection data to the memory of the central processing unit through the PCIE channel, that is, it contains one or more NPUs, which are used to accelerate the calculation of the neural network model, and are mainly responsible for receiving slice image data and performing target detection, and at the same time transmitting the slice image target detection data back to the main processor module.

[0042] Example

[0043] See also Figure 1 As shown, all processes within the main processor module and the coprocessor module are covered. The main processor module includes a CPU chip, which is responsible for the pre-processing of ultra-high-resolution images, the scheduling of data transmission processes, and the post-processing of target detection data; the coprocessor module includes an NPU chip, which is responsible for the intelligent computing acceleration of target detection of sliced ​​images, as well as the reception of sliced ​​image data and the return of detection data. In addition, the high-efficiency and high-reliability PCIE channel is an effective way to realize cross-device data transmission between the two modules. Taking the YOLOv5 neural network model for target detection of ultra-high-resolution images of size 70000*8500 as an example, the present invention mainly includes the following steps:

[0044] First, the input size of the YOLOv5 neural network model is generally 640*640, so the sliding window cropping size of the ultra-high resolution image is determined to be 640*640. At the same time, it is known that the minimum target size in the image is 50*50, so the overlap ratio is set to 10%, and the number of slice images obtained is 1830, and the pixel coordinates of the upper left corner of the slice image are [0,0], [576,0], [1152,0], etc. For slice images with a size less than 640*640, such as the pixel coordinates of the last image are [69696,8064], zero extension is used for padding. Then, OpenCV is used to read the entire ultra-high resolution image into the CPU memory and perform sliding window segmentation, and the upper left corner pixel of each slice image is recorded in the memory.

[0045] Secondly, the sliced ​​image data already in the CPU memory is transferred to the NPU through the PCIE channel in the order of segmentation. After the NPU obtains the sliced ​​image data, it performs intelligent calculations at runtime through the internally encapsulated API call to obtain the target detection data of the sliced ​​image, including the coordinates of the upper left corner of the target detection box, the width w and height h of the detection box, and the category and confidence of the target in the detection box.

[0046] Then, the CPU reads the target detection data into the CPU memory through the PCIE channel and performs post-processing of the detection data, mainly including the conversion of pixel coordinates and the elimination of duplicate detection frames. If the upper left corner coordinates of the target detection frame on the slice image are (x1, y1), and the upper left corner coordinates of the slice image in the original ultra-high resolution image read from the memory are (x2, y2), then the upper left corner coordinates of the target detection frame in the original ultra-high resolution image are (x1+x2, y1+y2), and the width w and height h of the target detection frame remain unchanged. After completing the coordinate conversion of all detection frames, the overlapping detection frames in the entire ultra-high resolution image are eliminated by the NMS method, and all detection data are directly drawn in the ultra-high resolution image in the memory, and finally the visual output of the detection results is completed.

[0047] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A heterogeneous platform data transmission method for accelerating ultra-high resolution image detection, suitable for the detection of airborne image data, the airborne heterogeneous platform includes a central processing unit and a neural network processor, characterized in that: The central processing unit and the plurality of neural network processors are connected via a PCIE bus, and the heterogeneous platform data transmission method comprises the following steps: S1: reading the original ultra-high resolution image in the central processing unit and performing image preprocessing, wherein the image preprocessing includes dividing the image into a plurality of slice images according to the pixel positions through a sliding window, wherein the sliding window may define a cropping size and an overlap ratio; S2: All the sliced ​​images are sequentially transmitted to a plurality of neural network processors of the same type through the PCIE channel in the order of segmentation, and the neural network processors perform parallel calculations; the PCIE channel can mount a plurality of neural network processors of the same type, and the specific number of neural network processors is determined according to the specifications of the high-resolution image; S3: Each neural network processor reads the received slice image and calls the neural network processor to perform target detection on the slice image; S4: After detecting the target in the slice image, target detection data is generated, and the target detection data is returned to the memory of the central processing unit through the PCIE channel; S5: merging the detection results of all slice images and filtering them using non-maximum suppression processing to eliminate repeated detection frames in the overlapping areas of the slice images; S6: After removing the duplicate detection frames, all detection frames are marked in the original ultra-high resolution image to achieve visualization of the target detection results of the ultra-high resolution image, and all the final detection frames are drawn in the original ultra-high resolution image in the central processing unit.

2. The heterogeneous platform data transmission method according to claim 1, characterized in that: S2 to S4 include data transmission and processing based on parallel algorithms. By establishing multi-threading in the central processing unit, ultra-high resolution image cutting, slice image sending to the neural network processor and the central processing unit receiving detection data of slice images are performed simultaneously.

3. The heterogeneous platform data transmission method according to claim 1, characterized in that: Step S1 includes using OpenCV to read the ultra-high resolution image and storing the RGB pixel values ​​of the image in the CPU memory; The pixel coordinates of all segmentation points in the image are stored in the memory of the central processing unit in the form of an array according to the segmentation order. This is to facilitate the coordinate conversion of the target detection box in the post-processing process.

4. The heterogeneous platform data transmission method according to claim 3: step S2 includes the central processing unit transmitting the slice image to the neural network processor through the PCIE channel.

5. The heterogeneous platform data transmission method according to claim 4, characterized in that: Step S4 includes the central processing unit reading the target detection data of the slice image from the specified neural network processor source memory address through the PCIE channel, the target detection data including the coordinates (x1, y1) of the target detection frame, the width and height of the target detection frame, and the category and confidence of the target. The upper left corner pixel coordinates are used as the coordinate origin, and the coordinates (x1, y1) are used as the relative position relationship between the target position and the coordinate origin.

6. The heterogeneous platform data transmission method according to claim 5, characterized in that: Step S5 includes: The target detection data received by the central processor in S4 is converted, wherein the target position is converted from pixel coordinates relative to the slice image to pixel coordinates relative to the original ultra-high resolution image, and duplicate detection frames are eliminated.

7. The heterogeneous platform data transmission method according to claim 6, characterized in that: Step S6 includes drawing the target detection data obtained in S5 into the original image read in step S1 and outputting a visualization result.

8. A system for accelerating ultra-high resolution image target detection, characterized in that: include, Main processor module: used to read the original ultra-high resolution image in the central processor and perform image preprocessing, the image preprocessing includes dividing the image into multiple slice images according to the pixel position through a sliding window, and the sliding window can define the cropping size and overlap ratio; The detection results of all slice images are merged and filtered using non-maximum suppression processing to eliminate repeated detection frames in the overlapping areas of the slice images; After removing duplicate detection frames, all detection frames are marked in the original ultra-high resolution image to achieve visualization of the target detection results in the ultra-high resolution image; Coprocessor module: used to transfer all the sliced ​​images to multiple neural network processors of the same type through the PCIE channel in the order of segmentation, and the neural network processors perform parallel calculations; the PCIE channel can mount multiple neural network processors of the same type, and the specific number of neural network processors is determined according to the specifications of the high-resolution image; Each neural network processor reads a slice image in the memory of the central processing unit and calls the neural network processor to perform object detection on the slice image; Through target detection, the specific content in the image can be identified, and target detection data can be generated after slice image target detection, and the target detection data can be returned to the memory of the central processing unit through the PCIE channel.

Images