End-side intelligent image processing system and method

Abstract

According to the end-side intelligent image processing system and method provided by the invention, through the heterogeneous multi-core architecture design of the end-side intelligent image processing system and the integration of the main control processor and the A I processor, efficient intelligent image processing on a small unmanned aerial vehicle platform is realized, the calculation performance is remarkably improved, and the equipment size and power consumption are reduced. Meanwhile, a double-layer board card and a magnesium alloy shell structure are adopted, the requirement for light weight is met, and the device is suitable for application scenes sensitive to the size and weight. Besides, according to the innovative data protection module design, an ambient light sensor and a contact switch are used for triggering high-voltage discharge, a storage module can be effectively damaged during unauthorized disassembly, and the safety of storage data and an algorithm model is protected.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle technology, and in particular to a terminal-side intelligent image processing system and method. Background Art

[0002] In recent years, drone technology has developed rapidly. As a payload platform, it can carry optical detectors and acquire image data, and its application scope and influence continue to expand.

[0003] In the field of drone detection and processing, traditional on-board online image processing mainly relies on pre-defined rules for simple image processing technology. For large amounts of data and massive image data in complex environments, they are usually transmitted to the back-end server for storage and processing, and cannot be processed on the drone side.

[0004] In addition, the neural network-based deep learning image processing algorithm that has emerged in recent years has good processing effects, but the algorithm operation process is very complex and requires large computing power support, making its application in on-board online image processing not practical.

[0005] At present, the edge-side intelligent image processing system faces many challenges:

[0006] 1) Due to the size and weight constraints of small UAV platforms, the end-side intelligent image processing system must be lightweight and miniaturized to ensure that it can be normally carried on small UAVs without affecting the maneuverability of the UAV;

[0007] 2) In order to ensure that small drones can fly for as long as possible, the end-side intelligent image processing system must be designed with low power consumption to ensure that the energy supply during the flight mission is not affected.

[0008] 3) In order to achieve good image processing effects, the intelligent image processing algorithm needs to be of high complexity, and the end-side intelligent image processing system must have high computing power.

[0009] 4) When processing data on the end side, the deep learning model previously in the cloud must be pre-installed on the end side. The security of end-side data (such as map data, model data, weight data, etc.) is increasingly receiving attention, requiring the end-side intelligent image processing system to have a data protection module to ensure that data is not leaked.

[0010] There are few studies on the terminal-side intelligent image processing system. The UAV and its image processing device disclosed in patent CN114299701A include an image acquisition unit, an image processing unit and a communication unit. Figure 1It can be seen that the device only performs simple fault detection on the end side, mainly transmitting data back to the server side for storage and image recognition through the communication unit. This is because the computing power of the image processing device cannot meet the real-time complex processing requirements of the end side. The multispectral intelligent image processing system for small drone platforms disclosed in patent CN117893391A "Multispectral Intelligent Image Processing System and Method for Small Drone Platforms" is based on Xilinx XC7K410T FPGA and Huawei Ascend At l as200SoC design. The device uses FPGA as the main control chip and has no application-oriented embedded hardware engine. The operation of FPGA resource configuration is complex and cannot quickly realize data preprocessing and processing acceleration. In addition, the device has no data protection module design and cannot ensure that the preset data is not leaked.

[0011] Existing edge-side intelligent image processing systems cannot simultaneously meet the hardware requirements of volume, weight, computing power, and power consumption as well as the functional requirements of rapid implementation, real-time processing, and security protection. Summary of the invention

[0012] A first aspect of the present disclosure provides an end-side intelligent image processing system, including:

[0013] The main control processor is used to connect to the data interface of the UAV front-end optoelectronic equipment to realize system control and communication functions, and pre-process the collected image data, including image encoding and decoding, denoising, contrast enhancement, size adjustment, rotation correction, and cropping;

[0014] The AI ​​processor is used to execute the neural network image processing algorithm to realize functions such as target detection, target tracking and target recognition, and the calculation results are transmitted back to the main control processor through the data interface;

[0015] A storage module, connected to the main control processor and the AI ​​processor respectively, and used to store software codes, convolutional neural network models, image data and corresponding configuration files required for system operation;

[0016] Data communication interface, used for data transmission with various sensors, including Gigabit Ethernet port, 100M Ethernet port, RS422, RS232, RS485, CAN bus, GPIO interface, SPI, and power interface;

[0017] Debug interface, including 100M Ethernet port, SRI0, UART, HDMI, PCIE, USB, JTAG, and Type-C interface for system debugging;

[0018] The data protection module is used to protect the data security of the end-side intelligent image processing system. In the event of external intrusion such as disassembly, it generates high-voltage current to destroy the storage device and protect data security.

[0019] In combination with the first aspect, the main control processor is a Rockchip RK3588J processor with an embedded hardware engine, which has image preprocessing, video encoding and decoding, and image enhancement functions, and provides I NT4 / I NT8 / I NT16 / FP16 mixed operations.

[0020] In combination with the first aspect, the AI ​​processor is the Epoch Times HXAI 100, which includes a 4-core RI SC-V CPU and 4 NNA cores, and is used for accelerated processing of convolutional neural network algorithms, providing parallel task execution and hardware acceleration of deep learning networks.

[0021] In combination with the first aspect, the storage module includes a dynamic memory and a non-volatile memory eMMC, which is used for high-bandwidth data storage.

[0022] In combination with the first aspect, the data communication interface uses a single connector, including a Gigabit Ethernet port, a 100M Ethernet port, RS422, RS232, RS485, a CAN bus, a GPIO interface, an SPI, and a power interface.

[0023] In combination with the first aspect, the data protection module includes a high-voltage destruction mechanism, which triggers the discharge of a large-capacity capacitor through an ambient light sensor or a contact switch to generate a high-voltage current to damage the storage device in the event of unauthorized disassembly.

[0024] A second aspect of the present disclosure provides a terminal-side intelligent image processing method, which specifically includes the following steps:

[0025] The image data collected by the front-end optoelectronic device is encoded and transmitted to the main control processor through the data interface;

[0026] After the main control processor performs preprocessing operations on the image, the data is transmitted to the AI ​​processor through the PC IE interface;

[0027] The AI ​​processor performs intelligent processing on the data, including target detection, tracking and identification, and transmits the processing results back to the main control processor via the PC IE interface;

[0028] The main control processor outputs the processing result to the back-end system through the data communication interface.

[0029] According to a third aspect of the present disclosure, an electronic device is provided, including:

[0030] one or more processors;

[0031] A storage unit is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors can implement the terminal-side intelligent image processing method.

[0032] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the computer program is executed by a processor, the terminal-side intelligent image processing method can be implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a schematic diagram of the interaction of the terminal-side intelligent imaging system according to an embodiment of the present disclosure;

[0034] Figure 2 is a schematic diagram of the structure of a storage module according to an embodiment of the present disclosure;

[0035] Figure 3 It is a structural schematic diagram of a data communication interface according to an embodiment of the present disclosure;

[0036] Figure 4 It is a schematic diagram of the structure of the debugging interface of the embodiment of the present disclosure;

[0037] Figure 5 is a structural diagram of a data protection module according to an embodiment of the present disclosure;

[0038] Figure 6 The figure is a flow chart of the terminal-side intelligent image processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the embodiments of the present disclosure.

[0040] The terms used in the disclosed embodiments are only for the purpose of describing specific embodiments and are not intended to limit the disclosed embodiments. The singular forms of "a", "said" and "the" used in the disclosed embodiments and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0041] The terminal-side intelligent image processing system of the embodiment of the present disclosure includes:

[0042] The main control processor is used to connect to the data interface of the UAV front-end optoelectronic equipment to realize system control and communication functions, and pre-process the collected image data, including image encoding and decoding, denoising, contrast enhancement, size adjustment, rotation correction, and cropping;

[0043] The AI ​​processor is used to execute the neural network image processing algorithm to realize functions such as target detection, target tracking and target recognition, and the calculation results are transmitted back to the main control processor through the data interface;

[0044] A storage module, connected to the main control processor and the AI ​​processor respectively, and used to store software codes, convolutional neural network models, image data and corresponding configuration files required for system operation;

[0045] Data communication interface, used for data transmission with various sensors, including Gigabit Ethernet port, 100M Ethernet port, RS422, RS232, RS485, CAN bus, GPIO interface, SPI, and power interface;

[0046] Debug interface, including 100M Ethernet port, SRI0, UART, HDMI, PCIE, USB, JTAG, and Type-C interface for system debugging;

[0047] The data protection module is used to protect the data security of the end-side intelligent image processing system. In the event of external intrusion such as disassembly, it generates high-voltage current to destroy the storage device and protect data security.

[0048] refer to Figure 1 , is a schematic diagram of the interaction of the above-mentioned terminal-side intelligent imaging system, where:

[0049] The main control processor uses Rockchip's main control chip, and the AI ​​processor uses Epoch AI chip. The two chips are located on the upper and lower boards respectively.

[0050] The upper and lower boards are connected through the inter-board interconnection interface. The upper and lower boards run an operating system respectively, integrating general processor resources, rich interface resources and dedicated neural network computing NNA resources, making the entire system highly versatile, with high computing power and low power consumption.

[0051] The Rockchip main control chip selected is RK3588J, which is responsible for realizing the image acquisition interface, system control communication interface, image preprocessing, image and video encoding and decoding, processing of various sensor data and system management functions.

[0052] Its main technical specifications are the integration of 4 Cortex-A76 and 4 Cortex-A55 and an independent NEON coprocessor, and built-in a variety of powerful embedded hardware engines, including H.264, JPEG and other codecs, 3D GPU, ISP and NPU, etc., support HDR, 3A, LSC, 3DNR, 2DNR, sharpening, dehazing, fisheye correction, gamma correction and other image processing, support INT4 / INT8 / INT16 / FP16 mixed operations, computing power up to 6TOP, support TensorFlow / MXNet / PyTorch / Caffe and other series of framework network model conversion.

[0053] The core era AI chip selected is HXAI100, which is a high-computing neural network coprocessor responsible for image and video target detection, target tracking and recognition and other convolutional neural network algorithms. Its main technical specifications are integrated 4-core RISC-V CPU and 4 NNA cores, support for parallel execution of different tasks, support for hardware acceleration of various common deep learning networks, computing power up to 16TOP, support for OpenCL programming model, support for OpenVX computer vision library.

[0054] The main control processor includes a dynamic memory LPDDR4 and a non-volatile memory eMMC, and is connected to a data communication interface and a debugging interface; the AI ​​processor includes a dynamic memory DDR4 and a non-volatile memory eMMC, and is connected to the debugging interface and a microcontroller unit (MCU), and the microcontroller unit is connected to a contact switch and an ambient light sensor.

[0055] The main control processor and the AI ​​processor are electrically connected and can both be controlled by a clock.

[0056] Furthermore, Figure 2 The schematic diagram of the structure of the storage module of the embodiment of the present disclosure. When designing the storage module, the capacity and bandwidth of the memory are fully considered. Rockchip RK3588 is configured with two 8GB dynamic memories and one 128GB non-volatile memory eMMC. The non-volatile memory eMMC is used to store operating system software code, image data and configuration files.

[0057] The Epoch HXAI100 is equipped with four 8GB dynamic memories and one 128GB non-volatile memory eMMC. The non-volatile memory eMMC is used to store operating system software code and convolutional neural network model parameters and configuration files.

[0058] Figure 3The data communication of the end-side intelligent image processing system designed by the present disclosure takes into account the influence of volume and weight factors, and adopts a single connector as the external data communication connection interface, which is J30J74, including Gigabit Ethernet port, 100M Ethernet port, RS422, RS232, RS485, CAN bus, GPIO interface, SPI, and power interface.

[0059] Figure 4 The schematic diagram of the debugging interface of the embodiment of the present disclosure is as follows. The debugging interface of the end-side intelligent image processing system designed by the present disclosure is also integrated into a single connector, the type of which is J63J, and includes a 100M network port, SRI0, UART, HDMI, PCIE, USB, JTAG, and Type-C interfaces.

[0060] Figure 5 It is a structural diagram of the data protection module of the embodiment of the present disclosure.

[0061] The data protection module includes a high-voltage destruction mechanism. In the case of unauthorized disassembly, the ambient light sensor or the contact switch triggers the discharge of the large-capacity capacitor to generate a high-voltage current to damage the storage device.

[0062] Specifically, the microcontroller unit (MCU) is powered by the power chip and is respectively connected to the RK3588J board and the housing contact switch, the RK3588J board and the housing ambient light sensor, the HXAI100 board and the housing contact switch, and the HXAI100 board ambient light sensor.

[0063] When the end-side intelligent image processing system is disassembled and the casing is opened, the ambient light sensor or contact switch will be triggered, and the MCU will control the large-capacity capacitor to discharge quickly and instantly. After passing through the voltage conversion chip, it is converted into high voltage. The instantaneous high voltage causes the eMMC storage module in the system to break down and be damaged, and the data cannot be read or restored, thus achieving the data protection effect.

[0064] The ambient light sensor is used to detect external environmental conditions. If the device is disassembled or improperly operated, the sensor will sense the change. The contact switch is used when the device is physically disassembled. The contact switch will be activated and trigger the protection mechanism.

[0065] The entire design is designed to prevent data leakage, especially for devices storing sensitive information. In the event of any unauthorized disassembly, the system can react quickly to protect data security.

[0066] Optionally, the mechanical housing of the end-side intelligent image processing system is made of magnesium alloy material, with dimensions of 100mm×80mm×25mm, a weight of 200g, and a 14-layer PCB.

[0067] The terminal-side intelligent image processing system of the disclosed embodiment has two layers of hardware boards that communicate through the board interconnection interface. The typical operating power consumption is 15W, the peak computing power is 22TOPS@INT8, the shell is made of magnesium alloy, and the processing chip and the shell are in hard contact with silicone grease to dissipate heat. The operating temperature range is -40°C to 55°C.

[0068] The hardware system adopts a heterogeneous multi-core architecture, with the characteristics of rich external interfaces, large computing power and low power consumption, providing the possibility for intelligent image processing on the terminal side of small UAV platforms.

[0069] Figure 6 The following is a flow chart of the intelligent image processing method on the terminal side according to an embodiment of the present disclosure. The method specifically includes the following steps:

[0070] The image data collected by the front-end optoelectronic device is encoded and transmitted to the main control processor through the data interface;

[0071] After the main control processor performs preprocessing operations on the image, the data is transmitted to the AI ​​processor through the PCIE interface;

[0072] The AI ​​processor performs intelligent processing on the data, including target detection, tracking and identification, and transmits the processing results back to the main control processor through the PCIE interface;

[0073] The main control processor outputs the processing result to the back-end system through the data communication interface.

[0074] Specifically, the front-end multi-channel image signals and multi-channel video signals are encoded and compressed and input through a Gigabit network cable or a 100M network cable; the position signal, attitude signal, control signal and other signal signals of the front-end detector are input through RS232, RS484, RS422 serial port and other interfaces;

[0075] The main processor RK3588J pre-processes the input image and video according to the control signal, including optional operations such as image / video decoding, geometric correction, blind pixel replacement, non-uniformity correction and image blur removal.

[0076] The main processor RK3588J transmits the pre-processed image video to the coprocessor HXAI100 through the PCIE interface;

[0077] The coprocessor HXAI100 has been started in advance and is waiting to receive data. After receiving the video frame / image, it directly performs intelligent image processing, including target detection, target tracking and recognition;

[0078] The coprocessor HXAI100 transmits the processed image results back to the main processor RK3588J through PCIE;

[0079] The main processor RK3588J solves the image processing results to obtain the final processing results; and transmits the end-side intelligent image processing system to the back end through the data communication interface.

[0080] Beneficial effects:

[0081] The heterogeneous multi-core architecture design makes the entire system small in size, light in weight, powerful in computing, and rich in interfaces, making it suitable for intelligent image processing scenarios on the drone side.

[0082] Adopt high-computing AI processor to support fast reasoning of complex neural network intelligent algorithms;

[0083] The system's multiple network ports can simultaneously transmit and process multiple channels of video simultaneously, improving the system's real-time performance.

[0084] The system's data protection function can ensure the data security needs of small drones.

[0085] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.

Claims

1. A terminal-side intelligent image processing system, characterized in that: include: The main control processor is used to connect to the data interface of the UAV front-end optoelectronic equipment to realize system control and communication functions, and pre-process the collected image data, including image encoding and decoding, denoising, contrast enhancement, size adjustment, rotation correction, and cropping; The AI ​​processor is used to execute the neural network image processing algorithm to realize functions such as target detection, target tracking and target recognition, and the calculation results are transmitted back to the main control processor through the data interface; A storage module, connected to the main control processor and the AI ​​processor respectively, and used to store software codes, convolutional neural network models, image data and corresponding configuration files required for system operation; Data communication interface, used for data transmission with various sensors, including Gigabit Ethernet port, 100M Ethernet port, RS422, RS232, RS485, CAN bus, GPIO interface, SPI, and power interface; Debug interface, including 100M Ethernet port, SRI0, UART, HDMI, PCIE, USB, JTAG, and Type-C interface for system debugging; The data protection module is used to protect the data security of the end-side intelligent image processing system. In the event of external intrusion such as disassembly, it generates high-voltage current to destroy the storage device and protect data security.

2. The terminal-side intelligent image processing system according to claim 1, characterized in that: The main control processor is a Rockchip RK3588J processor with an embedded hardware engine, which has image preprocessing, video encoding and decoding, and image enhancement functions, and provides INT4 / INT8 / INT16 / FP16 mixed operations.

3. The terminal-side intelligent image processing system according to claim 1, characterized in that: The AI ​​processor is the Epoch Times HXAI 100, which includes a 4-core RISC-V CPU and 4 NNA cores. It is used to accelerate the processing of convolutional neural network algorithms and provide parallel task execution and hardware acceleration of deep learning networks.

4. The terminal-side intelligent image processing system according to claim 1, characterized in that: The storage module includes a dynamic memory and a non-volatile memory eMMC, which is used for high-bandwidth data storage.

5. The terminal-side intelligent image processing system according to claim 1, characterized in that: The data communication interface adopts a single connector, including a Gigabit Ethernet port, a 100M Ethernet port, RS422, RS232, RS485, CAN bus, GPIO interface, SPI, and a power supply interface.

6. The terminal-side intelligent image processing system according to claim 1, characterized in that: The data protection module includes a high-voltage destruction mechanism. In the case of unauthorized disassembly, the ambient light sensor or contact switch triggers the discharge of a large-capacity capacitor to generate a high-voltage current to damage the storage device.

7. A terminal-side intelligent image processing method, characterized in that: include: The image data collected by the front-end optoelectronic device is encoded and transmitted to the main control processor through the data interface; After the main control processor performs preprocessing operations on the image, the data is transmitted to the AI ​​processor through the PCIE interface; The AI ​​processor performs intelligent processing on the data, including target detection, tracking and identification, and transmits the processing results back to the main control processor through the PCIE interface; The main control processor outputs the processing result to the back-end system through the data communication interface.

8. An electronic device, characterized in that: include: one or more processors; A storage unit is used to store one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement the end-side intelligent image processing method according to claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can implement the terminal-side intelligent image processing method according to claim 7.

Images