Image intelligent processing method, system and device based on FPGA and AI

Abstract

The invention provides an intelligent image processing method, system and device based on an FPGA and an AI, and particularly relates to the technical field of image processing. The method comprises the following steps: acquiring data to be processed through an FPGA (Field Programmable Gate Array), carrying out preprocessing and image enhancement on the data to be processed, sending the data subjected to preprocessing and image enhancement to an AI module to carry out automatic image detection, recognition and calculation, sending the data subjected to automatic image detection, recognition and calculation to the FPGA to carry out image post-processing to obtain image post-processing data, and sending the image post-processing data to the AI module to carry out image detection, recognition and calculation. And finally, encoding the image post-processing data through an AI module to obtain the image post-intelligent processing data. According to the method, the FPGA is combined with the AI module, and the FPGA is used for image preprocessing and image post-processing, so that the intelligent image processing efficiency is remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image intelligent processing method, system and device based on FPGA and AI. Background Art

[0002] With the progress and development of science and technology, the application of camera-based image monitoring and real-time visual detection and recognition technology in the security industry has made a leap forward. However, it also puts forward new requirements for image tracking stability, recognition rate of automatic target detection and recognition, pixel size of the target to be recognized, and processing time; however, the existing automatic image detection and recognition architecture has many shortcomings: the design architecture of the existing automatic image detection and recognition system makes the recognition processing time too long, and the computing resources are too large, which cannot meet the application requirements.

[0003] In the related art, in order to solve the above problems, the usual approach is to improve CPU performance by upgrading memory, replacing the motherboard, optimizing the system, etc., so as to improve the efficiency of automatic image recognition. However, this method is not effective in improving the efficiency of automatic image recognition and is costly.

[0004] In view of this, how to improve the efficiency of automatic image detection and recognition processing is a problem that needs to be solved in this technical field. Summary of the invention

[0005] The embodiments of the present application provide an image intelligent processing method, system and device based on FPGA and AI, so as to achieve the effect of improving the efficiency of image intelligent processing.

[0006] In a first aspect, an embodiment of the present application provides an image intelligent processing method based on FPGA and AI, which is applied to an image processing system, including:

[0007] Acquire the data to be processed through the FPGA, and obtain first processed data after preprocessing and image enhancement of the data to be processed through the FPGA;

[0008] Sending the first processed data to an AI module through the FPGA, and obtaining second processed data after being processed by the AI ​​module;

[0009] The second processed data is sent to the FPGA, and the FPGA performs image post-processing on the second processed data to obtain third processed data;

[0010] Encoding the third processed data through the AI ​​module to obtain encoded data and a SEI packet;

[0011] The encoded data and the SEI packet are synchronously output through a network port.

[0012] Optionally, the acquiring the data to be processed by using the FPGA, and performing preprocessing and image enhancement on the data to be processed by using the FPGA to obtain first processed data, includes:

[0013] The data to be processed is transmitted to the FPGA through a peripheral module using a high-speed transmission line SDI, wherein the peripheral module includes: a camera and a data transmission system;

[0014] The FPGA performs preprocessing and image enhancement on the data to be processed to obtain first processed data.

[0015] Optionally, the preprocessing and image enhancement of the data to be processed by the FPGA to obtain the first processed data includes:

[0016] Decoding the data to be processed;

[0017] Converting the color format of the decoded image into RGB format by the FPGA to obtain an RGB image, wherein the format conversion includes RAW to RGB, YUV420SP to RGB, and YUV422 to RGB;

[0018] Performing image enhancement on the RGB image through FPGA to obtain an enhanced image, wherein the image enhancement includes: image defogging, image denoising, image edge enhancement, image low illumination enhancement, image stitching, and image fusion;

[0019] The enhanced image is scaled in equal proportion by using FPGA, and the scaled image is filled to obtain first processed data.

[0020] Optionally, sending the first processed data to an AI module through the FPGA, and obtaining second processed data after processing by the AI ​​module, includes:

[0021] Sending the first processed data to the AI ​​module through the FPGA using a PCIE interface;

[0022] Decoding the first processed data by the AI ​​module to obtain decoded data;

[0023] Model inference is performed on the decoded data according to a target tracking algorithm or a target detection and recognition algorithm to obtain second processed data.

[0024] Optionally, sending the second processed data to an FPGA, and performing image post-processing on the second processed data by the FPGA to obtain third processed data includes:

[0025] Obtain the second processed data, and send it to the FPGA through the PCLE interface;

[0026] The second processed data is subjected to image post-processing by FPGA, and the third processed data is outputted through a transmission interface, wherein the transmission interface comprises: an SDI interface, a CamLink interface, and a PAL interface, and the image post-processing comprises: image OSD and image color mode conversion.

[0027] In a second aspect, an embodiment of the present application provides an image intelligent processing system based on FPGA and AI, including: an AI module, an FPGA module, and a peripheral module;

[0028] The AI ​​module provides a basic function support package for realizing intelligent image processing. The AI ​​module is based on general intelligent hardware and transfers the computing function of current automatic image detection and recognition to the AI ​​module;

[0029] The FPGA module integrates programmable resources that can be flexibly configured and combined to implement input and output interfaces, general digital logic, memory, digital signal processing, clock management functions, and also provides dedicated clock and wiring resources;

[0030] The peripheral module communicates with the AI ​​module via high-speed Ethernet, the peripheral module includes a camera, the peripheral module can be connected to the FPGA module via a high-speed transmission line SDI, the peripheral module can be connected to the FPGA module via CamLink, and the peripheral module can be connected to the FPGA module via a PAL interface.

[0031] Optionally, the AI ​​module further includes:

[0032] The AI ​​module is connected to the onboard storage module through the EMMC interface. The large-capacity onboard storage module facilitates users to store information locally. The AI ​​module has a built-in RGMII port. In a specific mode, the RGMII port communicates with the PHY chip, and the PHY chip communicates with the high-speed Ethernet.

[0033] The AI ​​module integrates a PCIE Gen3 X4 controller, supports up to PCIe 3.0 protocol, is backward compatible with PCIe 2.0 / 1.0, and has a working rate of up to Gpbs. The AI ​​module and FPGA module can be connected through the PCIe interface;

[0034] Optionally, the FPGA module further includes:

[0035] The FPGA module includes a UART module and a CAN module; the UART module includes a UART chip and a UART interface, and the CAN module includes a CAN chip and a CAN interface; the UART module and the CAN module can communicate with the peripherals through the UART interface and the CAN interface, and the CAN module realizes CAN communication with the peripherals;

[0036] The FPGA module communicates with the extended DDR, SDI interface, UART interface and CAN interface respectively; the FPGA module is connected to the extended DDR through the JEDEC interface and is used to execute system programs and cache data when the system is working; the DDR controller in the FPGA module supports access control to the dynamic memory;

[0037] The FPGA module extends the SDI module, the SDI input module includes an SDI equalizer and an SDI input interface; the SDI output module includes an SDI driver and an SDI output interface; the SDI input module and the SDI output module can receive data through the SDI input interface and send data through the SDI output interface.

[0038] In a third aspect, an embodiment of the present application provides an image intelligent processing device based on FPGA and AI, which is applied to an image processing system, and the device includes:

[0039] Acquisition module, acquires the data to be processed through FPGA;

[0040] A processing module, which performs preprocessing and image enhancement on the data to be processed through the FPGA to obtain first processed data;

[0041] The processing module is further used to send the first processed data to the AI ​​module through the FPGA, and obtain second processed data after being processed by the AI ​​module;

[0042] The processing module is further used to send the second processed data to the FPGA, and perform image post-processing on the second processed data through the FPGA to obtain third processed data;

[0043] The processing module is further used to encode the third processed data through the AI ​​module to obtain encoded data and a SEI packet;

[0044] The output module synchronously outputs the encoded data and the SEI packet through the network port.

[0045] Optionally, the processing module is specifically used to transmit the data to be processed to the FPGA via a peripheral module using a high-speed transmission line SDI, and the peripheral module includes: a camera and a data transmission system, and the first processed data is obtained after preprocessing and image enhancement of the data to be processed by the FPGA.

[0046] Optionally, the processing module is specifically used to decode the data to be processed; convert the color format of the decoded image into RGB format through the FPGA to obtain an RGB image, wherein the format conversion includes RAW to RGB, YUV420SP to RGB, and YUV422 to RGB; perform image enhancement on the RGB image through the FPGA to obtain an enhanced image, wherein the image enhancement includes: image defogging, image denoising, image edge enhancement, image low-illumination enhancement, image stitching, and image fusion; scale the enhanced image proportionally through the FPGA, and fill the scaled image to obtain the first processed data.

[0047] Optionally, the processing module is specifically used to send the first processed data to the AI ​​module through the FPGA using the PCIE interface, decode the first processed data through the AI ​​module to obtain decoded data, and perform model inference on the decoded data according to a target tracking algorithm or a target detection and recognition algorithm to obtain second processed data.

[0048] Optionally, the processing module is specifically used to obtain the second processed data, send it to the FPGA through the PCLE interface, perform image post-processing on the second processed data through the FPGA, and output third processed data through the transmission interface, wherein the transmission interface includes: SDI interface, CamLink interface, PAL interface, and the image post-processing includes: image OSD and image color mode conversion.

[0049] In a fourth aspect, an embodiment of the present application provides an image intelligent processing device based on FPGA and AI, including: a memory, a processor;

[0050] The memory stores computer-executable instructions;

[0051] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the first aspect and / or various possible implementations of the first aspect as described above, or the processor executes the second aspect and / or various possible implementations of the second aspect as described above.

[0052] The FPGA and AI-based image intelligent processing method, system and device provided in the embodiment of the present application obtain the data to be processed through FPGA, pre-process and image enhance the data to be processed, and then send the pre-processed and image-enhanced data to the AI ​​module for automatic image detection and recognition calculation, and then send the data after the automatic image detection and recognition calculation to FPGA for image post-processing to obtain image post-processing data, and finally encode the image post-processing data through the AI ​​module to obtain the image intelligently processed data. This method combines FPGA with AI module, and uses FPGA for image pre-processing and image post-processing, so as to achieve the effect of improving the efficiency of image intelligent processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0054] Figure 1 Schematic diagram of the process of the image intelligent processing method based on FPGA and AI provided in this application Figure 1 ;

[0055] Figure 2 Schematic diagram of the process of the image intelligent processing method based on FPGA and AI provided in this application Figure 2 ;

[0056] Figure 3 A schematic diagram of the structure of the image intelligent processing system based on FPGA and AI provided in this application;

[0057] Figure 4 A schematic diagram of the programmable resources of the JFM7 module provided in this application;

[0058] Figure 5 A schematic diagram of the AI ​​module interface definition provided for this application;

[0059] Figure 6 The connection diagram between the AI ​​module and the onboard EMMC provided for this application;

[0060] Figure 7 Schematic diagram of the connection between the AI ​​module provided in this application and high-speed Ethernet;

[0061] Figure 8 Schematic diagram of the connection between the AI ​​module and the FPGA module provided in this application through the PCLe interface;

[0062] Fig. 9 A schematic diagram showing the connection between the AI ​​module and the FPGA module provided in this application via a UART interface;

[0063] Fig.10 A schematic diagram showing the connection between the AI ​​module and the FPGA module provided in this application via the SPI interface;

[0064] Fig.11 A schematic diagram of the connection between the FPGA module and the extended DDR provided in this application;

[0065] Fig.12 Schematic diagram of the connection between the FPGA module and the CAN peripheral provided in this application;

[0066] Fig.13 A schematic diagram of the connection between the FPGA module and the RS422 connector provided in this application;

[0067] Fig.14 A schematic diagram of the connection between the FPGA module and the SDI input module provided in this application;

[0068] Fig.15 A schematic diagram of the connection between the FPGA module and the SDI output module provided in this application;

[0069] Fig.16 A schematic diagram of the processing flow of SDI interface input and display based on the image processing system provided in this application;

[0070] Fig.17 A schematic diagram of the processing flow of SDI interface input and network upload based on the image processing system provided in this application;

[0071] Fig.18 A schematic diagram of the processing flow of SDI interface input, display and network upload based on the image processing system provided in this application;

[0072] Fig.19 A schematic diagram of the structure of the image intelligent processing device based on FPGA and AI provided in this application;

[0073] Fig. 20 A schematic diagram of the structure of the image intelligent processing device based on FPGA and AI provided in this application.

[0074] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0076] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein.

[0077] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0078] With the progress and development of science and technology, the application of camera-based image monitoring and real-time visual detection and recognition technology in the security industry has made a leap forward. However, it also puts forward new requirements for image tracking stability, recognition rate of automatic target detection and recognition, pixel size of the target to be recognized, and processing time; however, the existing automatic image detection and recognition architecture has many shortcomings: the design architecture of the existing automatic image detection and recognition system makes the recognition processing time too long, and the computing resources are too large, which cannot meet the application requirements.

[0079] In the related art, in order to solve the above problems, the usual approach is to improve CPU performance by upgrading memory, replacing the motherboard, optimizing the system, etc., so as to improve the efficiency of automatic image recognition. However, this method is not effective in improving the efficiency of automatic image recognition and is costly.

[0080] In view of this, how to improve the efficiency of automatic image detection and recognition processing is a problem that needs to be solved in this technical field.

[0081] In response to the above problems, this application proposes an image intelligent processing method based on FPGA and AI, which obtains the data to be processed through FPGA, pre-processes and enhances the image of the data to be processed, and then sends the pre-processed and image-enhanced data to the AI ​​module for automatic image detection and recognition calculation, and then sends the data after the automatic image detection and recognition calculation to FPGA for image post-processing to obtain image post-processing data, and finally encodes the image post-processing data through the AI ​​module to obtain image intelligently processed data. This method solves the technical problem of low efficiency of image automatic detection and recognition processing.

[0082] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0083] Figure 1 Schematic diagram of the process of the image intelligent processing method based on FPGA and AI provided in this application Figure 1 The execution subject of this embodiment is an image processing system, such as Figure 1 As shown, this embodiment provides an image intelligent processing method based on FPGA and AI, the method comprising:

[0084] S101: Acquire data to be processed through FPGA, and obtain first processed data after preprocessing and image enhancement on the data to be processed through FPGA.

[0085] Among them, the above-mentioned FPGA refers to a field programmable gate array, and FPGA can be used to improve the performance of AI chips. FPGA can obtain the data to be processed through the data transmission interface, store the obtained data to be processed, query the data to be processed from the database according to actual business needs, and integrate flexibly configurable programmable resources through FPGA according to the data to be processed and actual needs. Through the input and output interfaces, general digital logic, memory, digital signal processing, dedicated clock and wiring resources, and clock management functions in the programmable resources, the first processed data is obtained after preprocessing and image enhancement of the data to be processed.

[0086] It can be understood that the purpose of preprocessing and image enhancement of the data to be processed is to pre-process the format, size, etc. of the data to be processed into a data format compatible with the AI ​​module, and then pass it to the AI ​​module for processing, thereby greatly improving the performance of the system.

[0087] S102: Send the first processed data to the AI ​​module through the FPGA, and obtain the second processed data after being processed by the AI ​​module.

[0088] Among them, the FPGA module can send the first processed data to the AI ​​module through the PCIe interface, the UART interface and the SPI interface. The AI ​​module uses the ATC model conversion tool to convert the improved Yolov5 model into an .om file adapted to the AI ​​processor, initialize the model resources and load the model from the *.om file through the interface, and pre-process the input video stream size through data pre-processing to make the incoming video width and height conform to the width and height of the target detection model as the input of the model reasoning, and create the input data of the model reasoning, which is used to describe the input data during the model reasoning, the input memory address, and the memory size. The improved Yolov5 network model uses Bi-FPN to replace PAN-Net, has better feature fusion capability, can better fuse feature information at different levels, can effectively prevent the loss of small target information, and improve the overall detection performance. The lightweight network MobileNetV3 is used for feature extraction, which can further reduce the model size and calculation amount, and improve the model reasoning speed and efficiency. By executing model reasoning, after the model reasoning is completed, the result of the model reasoning is processed in the data post-processing stage to obtain the second processed data.

[0089] It can be understood that the image processing system can transfer the computing function of target recognition and detection to the AI ​​module, rely on the powerful computing power of the AI ​​module, use vision-based detection methods to complete target recognition and detection calculations, thereby utilizing the powerful computing power of the AI ​​module to complete target recognition and detection calculations in the shortest time, thereby improving the recognition efficiency of the system.

[0090] S103: Send the second processed data to the FPGA, and perform image post-processing on the second processed data through the FPGA to obtain third processed data.

[0091] Among them, FPGA obtains the second processed data obtained after processing by the AI ​​module. FPGA realizes image post-processing by integrating input and output interfaces, general digital logic, memory, digital signal processing, dedicated clock and wiring resources, and clock management functions in programmable resources that can be flexibly configured and combined.

[0092] In this step, image post-processing is performed on the second processed data through FPGA so as to convert the second processed data into data required by the adaptable AI module.

[0093] S104: Encode the third processed data through the AI ​​module to obtain encoded data and a SEI packet.

[0094] SEI refers to supplementary enhancement information, which provides a method for adding information to the video bitstream and is one of the features of the H.264 / H.265 video compression standard. After obtaining the third processed data, the AI ​​module encodes the third processed data and obtains the encoded data and SEI package after processing the encoded data.

[0095] The purpose of this step is to encode the data after FPGA image post-processing to obtain coded data and SEI packets, so that the coded data and SEI packets can be further analyzed and processed in subsequent processes.

[0096] S105: Synchronously outputting the encoded data and the SEI packet through a network port.

[0097] Among them, the network port, that is, the network interface, refers to various interfaces of network devices, which are used for connection and communication between computers and other devices. After the data to be processed is processed by the FPGA and AI modules, the encoded data and the SEI package are obtained, and the encoded data and the SEI package are synchronously output for further processing in subsequent programs.

[0098] The FPGA and AI-based image intelligent processing method provided in the embodiment of the present application obtains the data to be processed through the FPGA, pre-processes and enhances the image of the data to be processed, and then sends the pre-processed and image-enhanced data to the AI ​​module for automatic image detection and recognition calculation, and then sends the data after the automatic image detection and recognition calculation to the FPGA for image post-processing to obtain image post-processing data, and finally encodes the image post-processing data through the AI ​​module to obtain the image intelligently processed data. This method significantly improves the efficiency of image intelligent processing by combining FPGA with AI module and using FPGA for image pre-processing and image post-processing.

[0099] Figure 2 Schematic diagram of the process of the image intelligent processing method based on FPGA and AI provided in this application Figure 2 ,like Figure 2 As shown, in this embodiment Figure 1 Based on the embodiment, the image intelligent processing method based on FPGA and AI is described in detail, and the method includes:

[0100] S201: Transmitting the data to be processed to the FPGA via a peripheral module using a high-speed transmission line SDI.

[0101] The peripheral module includes a camera and a data transmission system. SDI is a serial digital interface. The image processing system transmits the video information to the FPGA by receiving the video information from the SDI interface, wherein the video information is a type of data to be processed and includes visible light data and infrared data.

[0102] It is understandable that using a high-speed transmission line SDI to transmit the data to be processed through a peripheral module can improve the transmission efficiency of the data to be processed, thereby improving the overall processing efficiency of the image processing system.

[0103] S202: Decode the data to be processed, and convert the color format of the decoded image into RGB format through the FPGA to obtain an RGB image.

[0104] Among them, the format conversion includes RAW to RGB, YUV420SP to RGB and YUV422 to RGB. In this embodiment, the received data to be processed is first decoded by the data decoding module. The data to be processed may exist in a variety of encoding formats, such as H.264, H.265, JPEG, etc. The decoding process involves restoring the encoded data to the original image pixel data, which may exist in formats such as RAW, YUV420SP or YUV422. The FPGA color conversion module is the core part of the present invention. It receives the decoded image data and executes the corresponding conversion algorithm according to the original color format of the data to generate an image in RGB format.

[0105] RAW to RGB: For RAW format data, FPGA first identifies its arrangement (such as Bayer format), and then applies a demosaicing algorithm to convert the RAW data into RGB data. The demosaicing algorithm may involve interpolation, filtering and other steps to restore the RGB value of each pixel.

[0106] YUV420SP to RGB: YUV420SP (also known as the planar interleaved version of YUV420P, where chrominance information U and V are stored interleaved) is a commonly used color space format for compressing and transmitting image data. The FPGA first parses the YUV420SP data to separate the brightness component Y and the chrominance components U and V. Then, the YUV to RGB conversion formula is applied to convert the YUV value of each pixel into an RGB value.

[0107] YUV422 to RGB: In the YUV422 format, each pixel has a brightness component Y and two chrominance components (U and V appear alternately, and every two pixels share a pair of chrominance values). FPGA also parses the YUV422 data first, and then applies the YUV to RGB conversion formula for conversion. It should be noted that due to the higher chrominance resolution in YUV422 (relative to YUV420), the converted RGB image may be more delicate in color performance.

[0108] S203: Performing image enhancement on the RGB image through FPGA to obtain an enhanced image, scaling the enhanced image in equal proportion through FPGA, and filling the scaled image to obtain first processed data.

[0109] Among them, the image enhancement includes: image defogging, image denoising, image edge enhancement, image low-light enhancement, image stitching, and image fusion.

[0110] In specific implementation, the image processing system of this embodiment mainly includes an FPGA processing unit, an image input module, an image output module and a storage unit. Among them, the FPGA processing unit is the core, responsible for performing image enhancement, scaling and filling and other processing tasks; the image input module is used to receive the original RGB image data; the image output module is responsible for outputting the processed image data; the storage unit is used to temporarily store intermediate data and final results.

[0111] 1. Image enhancement processing

[0112] Within the FPGA processing unit, the image enhancement processing module is responsible for performing the following enhancement operations:

[0113] Image defogging: Using the dark channel prior principle, the dark channel map of the image is calculated to estimate the atmospheric light value, thereby restoring a clear image. FPGA accelerates this process through parallel computing to achieve real-time defogging.

[0114] Image denoising: Use algorithms such as median filtering, Gaussian filtering or bilateral filtering to suppress noise in the image. FPGA optimizes the filtering algorithm to achieve efficient denoising while retaining image details.

[0115] Image edge enhancement: Use algorithms such as the Sobel operator and Canny edge detection to identify edge information in the image and enhance the image clarity by enhancing edge contrast. FPGA accelerates the edge detection and enhancement process through parallel processing.

[0116] Image low-light enhancement: Use histogram equalization, Retinex theory or adaptive contrast stretching to improve image quality under low-light conditions and increase image brightness and contrast. FPGA optimizes the algorithm to achieve real-time enhancement of low-light images.

[0117] Image stitching: For multiple images with overlapping areas, image stitching is achieved through steps such as feature point matching, image registration, and image fusion. FPGA accelerates the feature point matching and image fusion process through parallel processing, improving stitching efficiency.

[0118] Image fusion: For multi-source images (such as infrared and visible light images), image fusion is achieved through multi-scale decomposition, weight distribution and image reconstruction. FPGA improves the quality and visual effect of fused images by optimizing the fusion algorithm.

[0119] 2. Proportional Scaling and Filling

[0120] After the image enhancement process, the scaling module in the FPGA processing unit is responsible for scaling the enhanced image in equal proportion. During the scaling process, the FPGA uses algorithms such as bilinear interpolation, nearest neighbor interpolation, or cubic interpolation to ensure the quality of the scaled image.

[0121] After the scaling is completed, the filling module is responsible for filling the border or background of the scaled image to obtain the first processed data that meets the output requirements. The filling content can be black, white, gray or other specified colors, or it can be a mirror image or repeated edge of the image.

[0122] 3. Storage and Output

[0123] The processed image data is stored in the storage unit and can be displayed, transmitted or further processed through the image output module. The image output module supports multiple interface standards, such as HDMI, DVI, USB, etc., to meet the needs of different application scenarios.

[0124] It can be understood that this embodiment realizes the enhancement, scaling and filling processing of RGB images through FPGA, thereby improving the quality, clarity and applicability of the image. At the same time, the parallel processing capability of FPGA ensures the high efficiency and real-time performance of the processing, and meets the performance requirements of the image processing system. In addition, this embodiment also provides a flexible image processing algorithm selection, and the type and degree of image enhancement, as well as the scaling and filling parameters can be adjusted according to actual needs to adapt to different application scenarios and needs.

[0125] In summary, this embodiment provides a clear, detailed, standardized and fully disclosed image processing method and system, which has broad application prospects and practical value.

[0126] S204: Send the first processed data to the AI ​​module via the FPGA using the PCIE interface.

[0127] Among them, the PCIE interface is a high-speed serial point-to-point dual-channel high-bandwidth transmission interface protocol. The first processed data can be quickly sent to the AI ​​module through the PCIE interface, thereby improving the data transmission efficiency.

[0128] S205: Decoding the first processed data by the AI ​​module to obtain decoded data.

[0129] Among them, the decoding method can be hard decoding and soft decoding. Hard decoding refers to the use of a dedicated hardware decoder to decode video or audio. The characteristics of hard decoding are fast speed and high efficiency, because it directly uses the dedicated capabilities of the hardware to handle the decoding task, reducing the burden on the CPU; soft decoding is decoding through software, which relies on the CPU to perform the decoding task. The advantage of soft decoding is that it provides high flexibility and configurability, because all calculations can be completed on the CPU. This method may occupy more CPU resources, especially when processing high-resolution or high-frame-rate videos. These two decoding methods have their own characteristics and applicable scenarios. In development, which method to choose depends on the specific hardware configuration, performance requirements, and application scenarios. For example, in a resource-constrained environment, such as a mobile device or a low-performance computer, hard decoding may be more suitable because it can provide better performance and lower power consumption. In cases where flexibility and configurability are required, such as when multiple formats need to be supported or complex post-processing needs to be performed, soft decoding is more suitable. The specific decoding method can be determined according to business needs, so it is not limited here.

[0130] The purpose of this step is to further process the decoded data using relevant algorithm programs based on the decoded data.

[0131] S206: Performing model inference on the decoded data according to a target tracking algorithm or a target detection and recognition algorithm to obtain second processed data.

[0132] The target tracking algorithm is a computer vision technology used to identify and track specific targets in videos or continuous image frames. The target tracking algorithm includes a KCF tracking algorithm and an improved KCF tracking algorithm with CN features added.

[0133] This embodiment includes a model reasoning module, which is the core part of the present invention. The model reasoning module performs reasoning on the preprocessed data according to the target tracking algorithm or the target detection and recognition algorithm.

[0134] Target tracking algorithm: If the target tracking algorithm is used, the model inference module first loads the pre-trained target tracking model, which may be built based on a deep learning framework (such as TensorFlow, PyTorch) and trained with a large amount of labeled data. Then, the module inputs the pre-processed image data into the model, and the model outputs the location information of the target in the current frame (such as bounding box coordinates) through steps such as feature extraction and similarity calculation.

[0135] Target detection and recognition algorithm: If the target detection and recognition algorithm is used, the model inference module also loads the pre-trained target detection and recognition model. The model can detect multiple targets in the image at the same time and identify their categories. During the model inference process, the module inputs the pre-processed image data into the model, and the model outputs the target's location information (bounding box coordinates) and category information through steps such as feature extraction, candidate region generation, classification and regression.

[0136] Optionally, the present embodiment further includes a post-processing module, which further processes the output results of the model reasoning module. For target tracking algorithms, post-processing may include coordinate smoothing, trajectory optimization, etc.; for target detection and recognition algorithms, post-processing may include non-maximum suppression (NMS), category probability adjustment, etc., to remove duplicate detections and improve detection accuracy.

[0137] This step uses a target tracking algorithm or a target detection and recognition algorithm to perform model inference on the decoded data. This can process the decoded data in a targeted manner, thereby obtaining second processed data that can be used in subsequent programs.

[0138] S207: Obtain the second processed data, and send it to the FPGA through the PCLE interface.

[0139] After acquiring the second processed data, the second processed data can be quickly sent to the FPGA through the PCIE interface, thereby improving data transmission efficiency.

[0140] S208: Perform image post-processing on the second processed data through the FPGA, and output third processed data through the transmission interface.

[0141] Among them, image post-processing refers to the ability to modify the appearance of an image by performing additional processing steps, and the image post-processing includes: image OSD and image color mode conversion. For example, the color of the image can be modified by image OSD and image color mode conversion or new elements can be superimposed on the image using post-processing. The image post-processed data is output through a transmission interface to obtain third processed data. The transmission interface includes: SDI, CamLink and PAL interfaces.

[0142] Image OSD (On Screen Display) and image color mode conversion are two common image post-processing technologies. OSD is a screen display technology used to display characters, graphics and images on a display terminal. Image color mode conversion refers to converting an image from one color mode to another to adapt to different output devices or achieve specific visual effects.

[0143] In this step, image post-processing can make the visual effect of the image clearer, so as to ensure that the image after image processing has higher clarity.

[0144] S209: Encode the third processed data through the AI ​​module to obtain encoded data and a SEI packet.

[0145] S210: Synchronously outputting the encoded data and the SEI packet through a network port.

[0146] Among them, steps S209-S210 are similar to steps S104-S105, so they are not repeated here.

[0147] The FPGA and AI-based image intelligent processing method provided in the embodiment of the present application transmits the data to be processed to the FPGA through a high-speed transmission line SDI through a peripheral module, integrates programmable resources that can be flexibly configured and combined according to the FPGA, performs preprocessing and image enhancement on the data to be processed through the programmable resources to obtain first processed data, sends the first processed data to the AI ​​module through the FPGA using a PCIE interface, decodes the first processed data through the AI ​​module to obtain decoded data, performs model inference on the decoded data according to a target tracking algorithm or a target detection and recognition algorithm to obtain second processed data, obtains the second processed data, sends it to the FPGA through a PCLE interface, performs image post-processing on the second processed data through the FPGA, and outputs third processed data through a transmission interface, encodes the third processed data through the AI ​​module to obtain encoded data and SEI packets, and synchronously outputs the encoded data and the SEI packets through a network port. The method combines FPGA with an AI module and uses FPGA for image pre-processing and image post-processing, thereby significantly improving the efficiency of image intelligent processing.

[0148] Figure 3 This is a schematic diagram of the structure of the image intelligent processing system based on FPGA and AI provided in this application. Figure 3As shown, this embodiment provides an image intelligent processing system based on FPGA and AI, which includes: an AI module 302, an FPGA module 301, and a peripheral module 303; wherein the AI ​​module 302 provides a basic function support package for realizing image intelligent processing, and the AI ​​module 302 is based on general intelligent hardware, and the computing function of the current image automatic detection and recognition is transferred to the AI ​​module 302; the AI ​​module 302 is connected to the onboard storage module through the EMMC interface, and the large-capacity onboard storage module facilitates users to realize local information storage, and the AI ​​module 302 has a built-in RGMII port, which communicates with the PHY chip through the RGMII port in a specific mode, and then the PHY chip communicates with the high-speed Ethernet; the AI ​​module 302 integrates a PCIE Gen3 X4 controller, which supports the PCIe 3.0 protocol at the highest, is backward compatible with PCIe2.0 / 1.0, and has a working rate of up to Gpbs level, and the AI ​​module 302 and the FPGA module 301 can be connected through the PCIe interface;

[0149] The FPGA module 301 integrates programmable resources that can be flexibly configured and combined, and is used to realize input and output interfaces, general digital logic, memory, digital signal processing, and clock management functions, and also provides dedicated clock and wiring resources; the FPGA module 301 includes a UART module and a CAN module; the UART module includes a UART chip and a UART interface, and the CAN module includes a CAN chip and a CAN interface; the UART module and the CAN module can communicate with peripherals through the UART interface and the CAN interface, and the CAN module realizes CAN communication with peripherals; the FPGA module 301 communicates with the extended DDR, SDI interface, UART interface and CAN interface respectively; the FPGA module 301 is connected to the extended DDR through the JEDEC interface, and is used to execute system programs and cache data when the system is working; the DDR controller in the FPGA module 301 supports access control to the dynamic memory; the FPGA module 301 extends the SDI module, and the SDI input module includes an SDI equalizer and an SDI input interface; the SDI output module includes an SDI driver and an SDI output interface; the SDI input module and the SDI output module receive data through the SDI input interface and send data through the SDI output interface.

[0150] The peripheral module 303 communicates with the AI ​​module 302 via high-speed Ethernet. The peripheral module 303 includes a camera. The peripheral module 303 can be connected to the FPGA module 301 via a high-speed transmission line SDI. The peripheral module 303 can be connected to the FPGA module 301 via CamLink. The peripheral module 303 can be connected to the FPGA module 301 via a PAL interface.

[0151] Figure 4A schematic diagram of the programmable resources of the JFM7 module provided for this application; the FPGA module can select a domestic high-performance FPGA. Taking the JFM7 series as an example, the JFM7 module integrates programmable resources that can be flexibly configured and combined to realize multiple functions such as input and output interfaces, general digital logic, memory, digital signal processing, clock management, etc. It also provides dedicated clock and wiring resources. Its programming resources are as follows Figure 4 shown.

[0152] Figure 5 Schematic diagram of the AI ​​module interface definition provided in this application; the AI ​​module is built with an AI310 processor, and its AI module has a computing power of up to 22TFLOPS INT8, which enables algorithms such as Yolov5 to be implemented; the AI310 processor includes 2 Da Vinci AI cores and 8 A55 Arm Cores (maximum main frequency 1.6GHz). The AI310 processor can communicate with the outside world through a Gigabit Ethernet interface, which is defined as follows Figure 5 shown.

[0153] Figure 6 The schematic diagram of the connection between the AI ​​module and the onboard EMMC provided in this application; the AI ​​module is connected to the onboard storage module through the EMMC interface, and the large-capacity onboard storage module facilitates users to store information locally; the specific connection method between the AI ​​module and the EMMC is as follows Figure 6 shown.

[0154] Figure 7 The schematic diagram of the connection between the AI ​​module and the high-speed Ethernet provided in the present application; the AI ​​module has a built-in RGMII port, which supports working in 10Mbps, 100Mbps and 1000Mbps modes; the AI ​​module communicates with the PHY chip through the RGMII port, and then the PHY chip communicates with the high-speed Ethernet; preferably, the model of the PHY chip is: YT8511H-DA; the RGMII port (i.e., the high-speed Ethernet interface) sends the extracted channel data to the host computer in the order of the channels at the same sampling time; the specific connection method between the AI ​​module and the high-speed Ethernet is as follows Figure 7 shown.

[0155] Figure 8 Schematic diagram of the AI ​​module and FPGA module connected through the PCLe interface provided in this application; the AI ​​module integrates a PCIE Gen3 X4 controller, supports the PCIe 3.0 protocol at the highest, is backward compatible with PCIe 2.0 / 1.0, and has a working rate of up to Gpbs level. The AI ​​module and FPGA module can be connected through the PCIe interface; the specific connection method of PCIe is as follows Figure 8 shown.

[0156] Fig. 9 Schematic diagram of the AI ​​module and FPGA module provided in this application connected via a UART interface; the UART communication interface is used for command signal transmission and detection information feedback; the specific connection method of the UART interface is as follows Fig. 9 shown.

[0157] Fig.10 The schematic diagram of the AI ​​module and FPGA module provided in this application are connected through the SPI interface; the AI ​​module integrates the SPI communication protocol, and the SPI transmission mode can be set by setting the relevant parameters of the control register; the SPI interface is used for command signal transmission and detection information feedback and as a redundant interface backup. The specific connection method of the SPI interface is as follows Fig.10 shown.

[0158] Fig.11 The schematic diagram of the connection between the FPGA module and the extended DDR provided in this application; the FPGA module communicates with the extended DDR3, SDI interface, UART interface and CAN interface respectively; the FPGA module is connected to the extended DDR3 through the JEDEC interface, and is used to execute system programs and cache data when the system is working; the DDR3 controller in the FPGA module supports access control to the dynamic memory DDR3 SDRAM; the specific connection method between the FPGA module and the extended DDR is as follows Fig.11 shown.

[0159] Fig.12 The schematic diagram of the connection between the FPGA module and the CAN peripheral provided in this application; the FPGA module is an important part of the system, and the FPGA module includes a UART module and a CAN module; the UART module includes a UART chip and a UART interface, and the CAN module includes a CAN chip and a CAN interface; the UART module and the CAN module can communicate with the peripherals through the UART interface and the CAN interface; the CAN module and the peripherals (i.e., CAN node devices or units) realize CAN communication, and the specific connection method between the FPGA module and the CAN node device is as follows Fig.12 shown.

[0160] Fig.13 The connection diagram of the FPGA module and the RS422 connector provided in this application; the FPGA module communicates with the RS422 transceiver through UART_RX and UART_TX, and the RS422 transceiver communicates with the RS422 interface through RS422_RXN / P and RS422_TXN / P. The specific connection method is as follows Fig.13 shown.

[0161] Fig.14 A schematic diagram of the connection between the FPGA module and the SDI input module provided in this application; Fig.15 The connection diagram of the FPGA module and the SDI output module provided in this application; the FPGA module extends the SDI module, the SDI input module includes an SDI equalizer and an SDI input interface; the SDI output module includes an SDI driver and an SDI output interface; the SDI input module and the SDI output module can receive data through the SDI input interface and send data through the SDI output interface. The specific connection method between the FPGA module and the SDI interface is as follows Fig.14 and Fig.15 shown.

[0162] Fig.16 The schematic diagram of the processing flow of SDI interface input and display based on the image processing system provided in this application mainly shows the processing flow of SDI interface input and display in the image processing system. The specific process is as follows Fig.16 shown.

[0163] Fig.17 The schematic diagram of the processing flow of SDI interface input and network upload based on the image processing system provided in this application mainly shows the processing flow of SDI interface input and network upload in the image processing system. Fig.17 shown.

[0164] Fig.18 The schematic diagram of the processing flow of SDI interface input, display and network upload based on the image processing system provided in this application mainly shows the processing flow of SDI interface input, display and network upload in the image processing system. The specific process is as follows Fig.18 shown.

[0165] Fig.19 The schematic diagram of the structure of the image intelligent processing device based on FPGA and AI provided in this application is as follows: Fig.19 As shown, the image intelligent processing device 1900 based on FPGA and AI provided in this embodiment includes:

[0166] An acquisition module 1901 acquires data to be processed through FPGA;

[0167] The processing module 1902 performs preprocessing and image enhancement on the data to be processed through the FPGA to obtain first processed data;

[0168] The processing module 1902 is further configured to send the first processed data to the AI ​​module through the FPGA, and obtain second processed data after being processed by the AI ​​module;

[0169] The processing module 1902 is further configured to send the second processed data to the FPGA, and perform image post-processing on the second processed data through the FPGA to obtain third processed data;

[0170] The processing module 1902 is further configured to encode the third processed data through the AI ​​module to obtain encoded data and a SEI packet;

[0171] The output module 1903 synchronously outputs the encoded data and the SEI packet through the network port.

[0172] Optionally, the processing module 1902 is specifically used to transmit the data to be processed to the FPGA via a peripheral module using a high-speed transmission line SDI, and the peripheral module includes: a camera and a data transmission system, and the first processed data is obtained after preprocessing and image enhancement of the data to be processed by the FPGA.

[0173] Optionally, the processing module 1902 is specifically configured to integrate programmable resources that can be flexibly configured and combined according to the FPGA, and obtain first processed data after preprocessing and image enhancement of the data to be processed through the programmable resources.

[0174] Optionally, the processing module 1902 is specifically used to send the first processed data to the AI ​​module through the FPGA using the PCIE interface, decode the first processed data through the AI ​​module to obtain decoded data, and perform model inference on the decoded data according to a target tracking algorithm or a target detection and recognition algorithm to obtain second processed data.

[0175] Optionally, the processing module 1902 is specifically used to obtain the second processed data, send it to the FPGA through the PCLE interface, perform image post-processing on the second processed data through the FPGA, and output third processed data through the transmission interface, wherein the transmission interface includes: SDI interface, CamLink interface, PAL interface, and the image post-processing includes: image OSD and image color mode conversion.

[0176] The image intelligent processing device based on FPGA and AI provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0177] Fig. 20 This is a schematic diagram of the structure of the image intelligent processing device based on FPGA and AI provided in this application. Fig. 20 As shown, the present application provides an image intelligent processing device 2000 based on FPGA and AI, including:

[0178] Memory 2002;

[0179] Processor 2001;

[0180] Wherein, the memory 2002 stores computer-executable instructions;

[0181] The processor 2001 executes the computer execution instructions stored in the memory to implement the image intelligent processing method based on FPGA and AI as shown in the above embodiment.

[0182] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0183] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0184] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An image intelligent processing method based on FPGA and AI, characterized in that: Applied to image processing systems, including: Acquire the data to be processed through the FPGA, and obtain first processed data after preprocessing and image enhancement of the data to be processed through the FPGA; Sending the first processed data to an AI module through the FPGA, and obtaining second processed data after being processed by the AI ​​module; The second processed data is sent to the FPGA, and the FPGA performs image post-processing on the second processed data to obtain third processed data; The third processed data is encoded by the AI ​​module to obtain encoded data and a SEI packet; and the encoded data and the SEI packet are synchronously output through a network port.

2. The method according to claim 1, characterized in that The method of acquiring the data to be processed by using the FPGA, and performing preprocessing and image enhancement on the data to be processed by using the FPGA to obtain first processed data includes: The data to be processed is transmitted to the FPGA through a peripheral module using a high-speed transmission line SDI, wherein the peripheral module includes: a camera and a data transmission system; The FPGA performs preprocessing and image enhancement on the data to be processed to obtain first processed data.

3. The method according to claim 2, characterized in that The method of obtaining first processed data by preprocessing and image enhancing the data to be processed by the FPGA includes: Decoding the data to be processed; Converting the color format of the decoded image into RGB format by the FPGA to obtain an RGB image, wherein the format conversion includes RAW to RGB, YUV420SP to RGB, and YUV422 to RGB; Performing image enhancement on the RGB image through FPGA to obtain an enhanced image, wherein the image enhancement includes: image defogging, image denoising, image edge enhancement, image low illumination enhancement, image stitching, and image fusion; The enhanced image is scaled in equal proportion by using FPGA, and the scaled image is filled to obtain first processed data.

4. The method according to claim 1, characterized in that: The step of sending the first processed data to the AI ​​module through the FPGA, and obtaining the second processed data after processing by the AI ​​module, includes: The FPGA sends the first processed data to the AI ​​module using a PCIE interface; the AI ​​module decodes the first processed data to obtain decoded data; Model inference is performed on the decoded data according to a target tracking algorithm or a target detection and recognition algorithm to obtain second processed data.

5. The method according to claim 1, characterized in that The sending the second processed data to the FPGA, and performing image post-processing on the second processed data by the FPGA to obtain third processed data, includes: Obtain the second processed data, and send it to the FPGA through the PCLE interface; The second processed data is subjected to image post-processing by FPGA, and the third processed data is outputted through a transmission interface, wherein the transmission interface comprises: an SDI interface, a CamLink interface, and a PAL interface, and the image post-processing comprises: image OSD and image color mode conversion.

6. An image intelligent processing system based on FPGA and AI, characterized in that: include: AI module, FPGA module, peripheral module; The AI ​​module provides a basic function support package for realizing intelligent image processing. The AI ​​module is based on general intelligent hardware and transfers the computing function of current automatic image detection and recognition to the AI ​​module; The FPGA module integrates programmable resources that can be flexibly configured and combined to implement input and output interfaces, general digital logic, memory, digital signal processing, clock management functions, and also provides dedicated clock and wiring resources; The peripheral module communicates with the AI ​​module via high-speed Ethernet, the peripheral module includes a camera, the peripheral module can be connected to the FPGA module via a high-speed transmission line SDI, the peripheral module can be connected to the FPGA module via CamLink, and the peripheral module can be connected to the FPGA module via a PAL interface.

7. The system according to claim 6, characterized in that The AI ​​module further includes: the AI ​​module is connected to the onboard storage module through the EMMC interface, the large-capacity onboard storage module is convenient for users to store information locally, the AI ​​module has a built-in RGMII port, in a specific mode, the RGMII port communicates with the PHY chip, and the PHY chip communicates with the high-speed Ethernet; The AI ​​module integrates a PCIE Gen3 X4 controller, supports up to PCIe 3.0 protocol, is backward compatible with PCIe 2.0 / 1.0, and has a working speed of up to Gpbs level. The AI ​​module and FPGA module can be connected through the PCIe interface.

8. The system according to claim 6, characterized in that The FPGA module further includes: The FPGA module includes a UART module and a CAN module; the UART module includes a UART chip and a UART interface, and the CAN module includes a CAN chip and a CAN interface; the UART module and the CAN module can communicate with the peripherals through the UART interface and the CAN interface, and the CAN module realizes CAN communication with the peripherals; The FPGA module communicates with the extended DDR, SDI interface, UART interface and CAN interface respectively; the FPGA module is connected to the extended DDR through the JEDEC interface and is used to execute system programs and cache data when the system is working; the DDR controller in the FPGA module supports access control to the dynamic memory; The FPGA module extends the SDI module, the SDI input module includes an SDI equalizer and an SDI input interface; the SDI output module includes an SDI driver and an SDI output interface; the SDI input module and the SDI output module can receive data through the SDI input interface and send data through the SDI output interface.

9. An image intelligent processing device based on FPGA and AI, characterized in that: Applied to an image processing system, the device comprises: Acquisition module, acquires the data to be processed through FPGA; A processing module, which performs preprocessing and image enhancement on the data to be processed through the FPGA to obtain first processed data; The processing module is further used to send the first processed data to the AI ​​module through the FPGA, and obtain second processed data after being processed by the AI ​​module; The processing module is further used to send the second processed data to the FPGA, and perform image post-processing on the second processed data through the FPGA to obtain third processed data; The processing module is further used to encode the third processed data through the AI ​​module to obtain encoded data and a SEI packet; The output module synchronously outputs the encoded data and the SEI packet through the network port.

10. An image intelligent processing device based on FPGA and AI, characterized in that: include: Memory; processor; Wherein, the memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the FPGA and AI-based image intelligent processing method as described in any one of claims 1 to 5.

Images