Intelligent image processing acceleration device for embedded scene

Abstract

The invention provides an intelligent image processing acceleration device for an embedded scene, which belongs to the technical field of embedded digital image processing and specifically comprises an image acquisition unit, an intelligent image preprocessing unit, an intelligent image processing unit, an intelligent image post-processing unit and a data storage unit. The image preprocessing unit decodes the digital image, carries out data type conversion on the decoded data, converts the decoded data into two-dimensional image data, and then carries out preprocessing operation on the two-dimensional image; the intelligent image processing unit is used for performing reasoning operation on the intelligent image according to the detection model and sending a reasoning result to the intelligent image post-processing unit; and the intelligent image post-processing unit analyzes the reasoning result and extracts the position information, the category information and the target confidence information of the target according to the analysis result. Through the processing scheme provided by the invention, the throughput rate of intelligent image processing is improved, and the processing delay and power consumption are reduced.

Description

Technical Field

[0001] The present application relates to the field of embedded digital image processing, and in particular to an intelligent image processing acceleration device for embedded scenarios. Background Art

[0002] Deep learning technology has set off a wave of artificial intelligence and has been widely used in computer vision, large models, autonomous driving, natural language processing and other fields.

[0003] In embedded scenarios, digital image intelligent processing is divided into three parts: image preprocessing, image intelligent reasoning, and image postprocessing. In embedded scenarios, image preprocessing and image postprocessing are generally completed on the embedded processor CPU, and intelligent image reasoning is generally completed on the coprocessor, such as GPU, NPU, TPU, FPGA and other coprocessors. With the development of artificial intelligence technology, artificial intelligence chips have developed rapidly, while embedded processor CPUs have developed relatively slowly, resulting in slow image preprocessing and image postprocessing calculations, and faster intelligent reasoning calculations. The entire image processing calculation bottleneck is stuck on the CPU processor, with problems such as low throughput, high latency, and high power consumption. Summary of the invention

[0004] In view of this, the present application provides an intelligent image processing acceleration device for embedded scenarios, which solves the problems in the prior art, improves the throughput of intelligent image processing and reduces processing delay and power consumption.

[0005] The present application provides an intelligent image processing acceleration device for embedded scenarios using the following technical solutions:

[0006] An intelligent image processing acceleration device for embedded scenes, comprising an image acquisition unit, an intelligent image pre-processing unit, an intelligent image processing unit, an intelligent image post-processing unit and a data storage unit;

[0007] The image acquisition unit is connected to the intelligent image preprocessing unit, and the image acquisition unit is used to convert the real scene into a digital image signal and transmit the digital image signal to the image preprocessing unit;

[0008] The intelligent image preprocessing unit is connected to the intelligent image processing unit, and the image preprocessing unit is used to decode the digital image, perform data type conversion on the decoded data, convert it into two-dimensional image data, and then perform preprocessing operations on the two-dimensional image and send the image to the intelligent image processing unit, and the preprocessing operations include filtering, defogging, and de-jittering;

[0009] The intelligent image processing unit is connected to the intelligent image post-processing unit, and the intelligent image processing unit is used to perform an inference operation of the intelligent image according to the detection model and send the inference result to the intelligent image post-processing unit;

[0010] The intelligent image post-processing unit is connected to the data storage unit, and the intelligent image post-processing unit analyzes the reasoning result and extracts the location information, category information, and target confidence information of the target according to the analysis result;

[0011] The data storage unit is used to store the processing result of the intelligent image post-processing unit.

[0012] Optionally, the image acquisition unit includes a light source, a lens, a camera and an acquisition card.

[0013] Optionally, the image preprocessing unit includes:

[0014] A decoding module, used for decoding a digital image;

[0015] A data type conversion module, used for converting the decoded image data into two-dimensional image data;

[0016] The processing module is used to perform preprocessing operations on the two-dimensional image.

[0017] Optionally, the processing module uses a NEON instruction set to perform parallel acceleration on the two-dimensional image data during a preprocessing operation according to an image processing algorithm.

[0018] Optionally, the processing module uses NEON instruction set for parallel acceleration and OpenMP for parallel acceleration.

[0019] Optionally, the intelligent image processing unit includes an embedded NPU coprocessor, which receives image data sent by the image preprocessing unit processing module, performs intelligent image reasoning operations based on the detection model, and transmits the reasoning results to the processing module and enters the image post-processing unit.

[0020] Optionally, the image post-processing unit uses NMS and a drawing post-processing algorithm to filter the target position information, and at the same time uses OpenMP to perform multi-core parallel acceleration on the NMS and the drawing post-processing algorithm in the vertical direction.

[0021] In summary, this application includes the following beneficial technical effects:

[0022] The application proposes an intelligent image processing acceleration device and acceleration method for embedded scenarios, which completes the inference acceleration of intelligent images on the NPU coprocessor; and completes the parallel acceleration of image pre-processing and post-processing on the embedded processor. The acceleration method proposed in the application solves the computing bottleneck of the CPU processor, uses the NEON instruction set to achieve horizontal parallel acceleration of two-dimensional images during image preprocessing, uses OpenMP to achieve multi-core parallel acceleration of two-dimensional images in the vertical direction, and uses OpenMP to achieve multi-core parallel acceleration of two-dimensional images in the vertical direction during image post-processing. The application can effectively improve the throughput of intelligent image processing, reduce processing delays, and power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 This is a schematic diagram of the structure of the intelligent image processing acceleration device for embedded scenarios in this application;

[0025] Figure 2 This application provides an embedded scene intelligent image processing acceleration method. DETAILED DESCRIPTION

[0026] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0027] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0028] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0029] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The illustrations only show the components related to the present application rather than being drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0030] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0031] An embodiment of the present application provides an intelligent image processing acceleration device for embedded scenarios.

[0032] like Figure 1 and Figure 2 As shown, an intelligent image processing acceleration device for embedded scenarios includes an image acquisition unit, an intelligent image preprocessing unit, an intelligent image processing unit, an intelligent image post-processing unit and a data storage unit.

[0033] The image acquisition unit is connected to the intelligent image preprocessing unit, and the image acquisition unit includes a light source, a lens, a camera and an acquisition card, and these components work together to convert the real scene into a digital image signal and transmit the digital image signal to the image preprocessing unit. Among them, the image acquisition unit is a mature existing technology and will not be described in detail here.

[0034] The intelligent image preprocessing unit is connected to the intelligent image processing unit, and the image preprocessing unit is used to decode the digital image, convert the decoded data into a two-dimensional image data, and then perform a preprocessing operation on the two-dimensional image before sending the image to the intelligent image processing unit. The preprocessing operation includes filtering, defogging, de-jittering and image fusion. Specifically:

[0035] The image preprocessing unit comprises:

[0036] A decoding module, used for decoding a digital image;

[0037] A data type conversion module, used for converting the decoded image data into two-dimensional image data;

[0038] The processing module is used to perform preprocessing operations on the two-dimensional image. The processing module uses the NEON instruction set to perform parallel acceleration on the two-dimensional image data during the preprocessing operation according to the image processing algorithm, so as to achieve parallel acceleration of the two-dimensional image processing in the horizontal direction. The processing module uses the NEON instruction set for parallel acceleration and OpenMP for parallel acceleration at the same time, so as to achieve multi-core parallel acceleration of the two-dimensional image processing in the horizontal and vertical directions respectively. The image preprocessing process completes the calculation on the embedded CPU processor. The processing process can effectively reduce the image preprocessing delay and improve the throughput rate of the device.

[0039] The intelligent image processing unit is connected to the intelligent image post-processing unit, and the intelligent image processing unit is used to perform reasoning operations on intelligent images according to the detection model, and send the reasoning results to the intelligent image post-processing unit; the intelligent image processing unit includes an embedded NPU coprocessor, and the embedded NPU coprocessor receives image data sent by the processing module of the image pre-processing unit, and the embedded NPU coprocessor performs reasoning operations on intelligent images according to the detection model, and the embedded NPU coprocessor transmits the reasoning results to the processing module and enters the image post-processing unit.

[0040] The intelligent image post-processing unit is connected to the data storage unit, and the intelligent image post-processing unit analyzes the inference results and extracts the target's location information, category information, and target confidence information according to the analysis results; specifically, the image post-processing unit uses NMS or drawing post-processing algorithm to filter the target location information. In the embodiment of the present application, the NMS algorithm is used to filter the target location information. Since the NMS algorithm is time-consuming to calculate, the analysis results are accelerated by multi-core parallelism using OpenMP in the vertical direction to shorten the image post-processing time. If it is necessary to render a drawing graphic frame in the original image, OpenMP technology can also be used to realize multi-core parallel acceleration of the frame in the vertical direction of the two-dimensional image, and then the post-processing results are input into the storage management unit.

[0041] The data storage unit is used to store the processing results of the intelligent image post-processing unit. The image storage unit stores the image post-processing results in an unstructured database to facilitate subsequent addition, deletion, modification, and query. The processing results include image information, target location information, target category information, target confidence information, etc. Specifically, the data storage unit includes: an encoding module, which is used to encode and compress the image data processed by the processing module; a data storage module, which is a component for data storage and access.

[0042] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An intelligent image processing acceleration device for embedded scenarios, characterized in that: It includes an image acquisition unit, an intelligent image pre-processing unit, an intelligent image processing unit, an intelligent image post-processing unit and a data storage unit; The image acquisition unit is connected to the intelligent image preprocessing unit, and the image acquisition unit is used to convert the real scene into a digital image signal and transmit the digital image signal to the image preprocessing unit; The intelligent image preprocessing unit is connected to the intelligent image processing unit, and the image preprocessing unit is used to decode the digital image, perform data type conversion on the decoded data, convert it into two-dimensional image data, and then perform preprocessing operations on the two-dimensional image and send the image to the intelligent image processing unit, and the preprocessing operations include filtering, defogging, and de-jittering; The intelligent image processing unit is connected to the intelligent image post-processing unit, and the intelligent image processing unit is used to perform an inference operation of the intelligent image according to the detection model and send the inference result to the intelligent image post-processing unit; The intelligent image post-processing unit is connected to the data storage unit, and the intelligent image post-processing unit analyzes the reasoning result and extracts the location information, category information, and target confidence information of the target according to the analysis result; The data storage unit is used to store the processing result of the intelligent image post-processing unit.

2. The intelligent image processing acceleration device for embedded scenarios according to claim 1, characterized in that: The image acquisition unit includes a light source, a lens, a camera and an acquisition card.

3. The intelligent image processing acceleration device for embedded scenarios according to claim 1, characterized in that: The image preprocessing unit comprises: A decoding module, used for decoding a digital image; A data type conversion module, used for converting the decoded image data into two-dimensional image data; The processing module is used to perform preprocessing operations on the two-dimensional image.

4. The intelligent image processing acceleration device for embedded scenarios according to claim 3 is characterized in that: The processing module uses the NEON instruction set to perform parallel acceleration on the two-dimensional image data during the pre-processing operation according to the image processing algorithm.

5. The intelligent image processing acceleration device for embedded scenarios according to claim 4, characterized in that: The processing module uses the NEON instruction set for parallel acceleration and OpenMP for parallel acceleration.

6. The intelligent image processing acceleration device for embedded scenarios according to claim 1, characterized in that: The intelligent image processing unit includes an embedded NPU coprocessor, which receives image data sent by the image preprocessing unit processing module, performs intelligent image reasoning operations according to the detection model, and transmits the reasoning results to the processing module and enters the image post-processing unit.

7. The intelligent image processing acceleration device for embedded scenarios according to claim 1, characterized in that: The image post-processing unit uses NMS and drawing post-processing algorithms to filter the target position information, and uses OpenMP to perform multi-core parallel acceleration on the NMS and drawing post-processing algorithms in the vertical direction.

Images