Image processing system

Abstract

The invention relates to an image processing system. The system comprises a sensor unit, an FPGA unit, a display unit and an upper computer unit, the sensor unit is used for collecting original image data and transmitting the collected data to the FPGA unit; the FPGA unit is used for dividing original image data into two same paths of data, carrying out ISP processing on one path of data, then carrying out image enhancement on the data subjected to ISP processing to obtain first path of image data, and sending the first path of image data to the display unit; the FPGA unit is also used for sending the other path of image data as a second path of image data to the upper computer unit; the upper computer unit is used for determining a target network model corresponding to a current application scene and issuing the target network model to the FPGA unit; and the display unit is used for displaying the received first path of image data. By adopting the system, the problems that the application environment is limited and the requirements of different application environments cannot be met when the FPGA is used as the accelerator of the AI ISP in the existing image processing system can be solved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing system. Background Technology

[0002] With the development of data processing technology, digital image processing technology has also developed rapidly. In the process of digital image processing, the IPS (Image Signal Processor), as an important component of the camera device, improves the quality of the original image by performing operations such as color correction, white balance adjustment, and noise reduction on the raw image data acquired by the camera device.

[0003] Traditional IPS modules suffer from poor coupling, requiring adjustments based on preset parameters and leading to inter-module interference. Furthermore, traditional IPS struggles to adapt to complex and changing environments, failing to meet the demands for detail restoration and image enhancement in complex scenes. To address these issues, existing technologies have proposed AI ISPs (Artificial Intelligence Image Signal Processors) that combine deep learning and computer vision. AI ISPs can better restore image details, increase dynamic range, and improve image quality. However, compared to traditional ISPs, AI ISPs require significantly more computation, necessitating the use of accelerators to avoid latency issues.

[0004] In existing image processing systems, AI ISP accelerators are generally implemented using FPGAs. When an FPGA is used as an AIISP accelerator, due to the limitations of the FPGA's computing power resources, it is impossible to deploy large-scale networks or deploy multiple small-scale networks at the same time, resulting in limited applicable environments and failing to meet the requirements of different application environments.

[0005] For existing image processing systems, when FPGAs are used as accelerators for AI ISPs, there is a limitation in their applicable environment, making it impossible to meet the requirements of different application environments. Currently, no effective solution has been proposed. Summary of the Invention

[0006] Therefore, it is necessary to provide an image processing system to address the aforementioned technical problems.

[0007] In a first aspect, this application provides an image processing system. The system includes: a sensor unit, an FPGA unit, a display unit, and a host computer unit;

[0008] The sensor unit is connected to the FPGA unit and is used to acquire raw image data and transmit the acquired raw image data to the FPGA unit.

[0009] The FPGA unit, connected to the host computer unit and the display unit, is used to divide the received raw image data into two identical data streams. One stream is first processed by ISP (Image Signal Processing), and then the loaded target network model is used to perform image enhancement on the ISP-processed data to obtain the first image data stream. This first image data stream is then sent to the display unit. The FPGA unit is also used to send the other of the two data streams as the second image data stream to the host computer unit.

[0010] The host computer unit is used to determine the target network model corresponding to the current application scenario based on the received second-channel image data, and send the target network model to the FPGA unit for loading by the FPGA unit;

[0011] The display unit is used to display the received first channel image data.

[0012] In one embodiment, the FPGA unit includes an image segmentation subunit, an image preprocessing subunit, and a network model subunit;

[0013] The image segmentation subunit is connected to the sensor unit, the image preprocessing subunit, and the host computer unit. It is used to receive the raw image data sent by the sensor unit, divide the received raw image data into two identical data streams, send one of the data streams to the image preprocessing subunit, and send the other data stream as the second image data stream to the host computer unit.

[0014] The image preprocessing subunit is connected to the network model subunit and is used to perform ISP processing on the received data to obtain ISP-processed data, and send the ISP-processed data to the network model subunit.

[0015] The network model subunit is connected to the host computer unit and the display unit. It is used to load the target network model provided by the host computer unit, and use the loaded target network model to perform image enhancement on the received data processed by the ISP to obtain the first channel image data, and transmit the first channel image data to the display unit.

[0016] In one embodiment, the image segmentation subunit is further configured to copy the received original image data and divide the copied data into two identical data streams.

[0017] In one embodiment, the network model subunit is further configured to update the loaded target network model in response to the parameters of the trained target network model sent by the host computer unit.

[0018] In one embodiment, the FPGA unit further includes an image post-processing subunit;

[0019] The image post-processing subunit is connected to the network model subunit and the display unit, and is used to receive the first channel image data sent by the network model subunit; configure the display of the received first channel image data to obtain the configured first channel image data; and send the configured first channel image data to the display unit.

[0020] In one embodiment, the host computer unit includes: a host computer subunit and a cloud subunit;

[0021] The host computer subunit, connected to the image segmentation subunit, the network model subunit, and the cloud subunit, is used to receive the second-channel image data sent by the image segmentation subunit; determine the target network model corresponding to the current application scenario based on the received second-channel image data; and send the target network model to the network model subunit for loading; the host computer subunit is also used to send the received second-channel image data and the determined model number of the target network model to the cloud subunit;

[0022] The cloud subunit is used to determine the network model to be trained based on the received model number; the target network model to be trained is the same as the target network model; the cloud subunit is also used to train the network model to be trained based on the received second image data to obtain the parameters of the trained target network model, and send the parameters of the trained target network model to the host computer subunit.

[0023] The host computer subunit is also used to send the received parameters of the trained target network model to the network model subunit.

[0024] In one embodiment, the host computer subunit is further configured to determine the current application scenario based on the received second-channel image data; and to determine the target network model corresponding to the current application scenario based on the current application scenario.

[0025] In one embodiment, the host computer subunit includes: a data and parameter management component, a configuration storage component, and a configuration control component;

[0026] The data and parameter management component, connected to the cloud subunit, the image segmentation subunit, and the configuration storage component, is used to receive the second-channel image data sent by the image segmentation subunit and send the received second-channel image data to the cloud subunit. The data and parameter management component is also used to determine the current application scenario based on the received second-channel image data; determine the target network model corresponding to the current application scenario based on the current application scenario; determine the model number of the target network model based on the target network model corresponding to the current application scenario, and send the model number to the configuration storage component and the cloud subunit respectively. The data and parameter management component is also used to receive the parameters of the trained target network model sent by the cloud subunit and send the received parameters of the trained target network model to the network model subunit.

[0027] The configuration storage component, connected to the configuration control component, is used to determine the target network model corresponding to the received model number, and send the target network model to the configuration control component.

[0028] The configuration control component is connected to the network model subunit and is used to send the received target network model to the network model subunit for loading.

[0029] In one embodiment, the FPGA unit further includes an interface configuration subunit;

[0030] The interface configuration subunit connects the image segmentation subunit and the sensor unit, and is used to send preset interface configuration information to the sensor unit and send the received raw image data to the image segmentation subunit.

[0031] In one embodiment, the sensor unit is further configured to determine image acquisition configuration parameters based on the received interface configuration information; the configuration parameters include at least one of acquisition frame rate, resolution, or exposure parameters; and acquire the raw image data based on the image acquisition configuration parameters.

[0032] The aforementioned image processing system includes a sensor unit for acquiring raw image data and transmitting the acquired raw image data to an FPGA unit; an FPGA unit for dividing the received raw image data into two identical data streams, performing ISP processing on one stream, and then using a loaded target network model to perform image enhancement on the ISP-processed data to obtain the first image data stream, which is then sent to a display unit; the FPGA unit also sends the other stream of the two data streams as the second image data stream to a host computer unit; the host computer unit determines the target network model corresponding to the current application scenario based on the received second image data stream and sends the target network model to the FPGA unit for loading; and a display unit for displaying the received first image data stream. It divides image data into two identical data streams, processes and displays the first stream, and selects a model using the second stream. The selected target network model is then sent to the FPGA unit for image processing of the first stream image data. By matching the host computer network model with the environment, it avoids the computational burden of matching the FPGA with the environment. This not only realizes the accelerator function of the FPGA but also ensures that the target network model equipped on the FPGA is adaptable to the environment. This solves the problem that existing image processing systems, when using the FPGA as an accelerator for an AI ISP, have limited applicable environments and cannot meet the requirements of different application environments.

[0033] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0034] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0035] Figure 1 This is a first structural block diagram of an image processing system provided in an embodiment of this application;

[0036] Figure 2 This is a second structural block diagram of an image processing system provided in an embodiment of this application;

[0037] Figure 3 This is a third structural block diagram of an image processing system provided in an embodiment of this application;

[0038] Figure 4 This is a fourth structural block diagram of an image processing system provided in an embodiment of this application;

[0039] Figure 5This is a fifth structural block diagram of an image processing system provided in an embodiment of this application. Detailed Implementation

[0040] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0041] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0042] This embodiment provides an image processing system. Figure 1 This is a first structural block diagram of the image processing system provided in this embodiment, such as... Figure 1As shown, the system includes: a sensor unit 110, an FPGA unit 120, a display unit 130, and a host computer unit 140; the sensor unit 110, connected to the FPGA unit 120, is used to acquire raw image data and transmit the acquired raw image data to the FPGA unit 120; the FPGA unit 120, connected to the host computer unit 140 and the display unit 130, is used to divide the received raw image data into two identical data streams, perform ISP processing on one stream, and then use the loaded target network model to perform image enhancement on the ISP-processed data to obtain the first image data stream, which is then sent to the display unit 130; the FPGA unit 120 is also used to send the other stream of the two data streams as the second image data stream to the host computer unit 140; the host computer unit 140 is used to determine the target network model corresponding to the current application scenario based on the received second image data stream, and send the target network model to the FPGA unit 120 for loading; the display unit 130 is used to display the received first image data stream.

[0043] The aforementioned sensor unit 110 includes at least a sensor for image acquisition. The type of sensor can be varied depending on the application scenario. In some cases, the sensor unit 110 only needs one sensor, while in others, it needs multiple sensors. Specifically, the sensor can be one or more of the following: charge-coupled device (CCD), complementary metal-oxide-semiconductor (CMOS), infrared sensor, ultraviolet sensor, multispectral or hyperspectral sensor, 3D image sensor, fiber optic sensor, polarization sensor, etc. It should be noted that the number and type of sensors in the sensor unit 110 can be specifically configured according to specific needs. This embodiment does not impose specific limitations, as long as the sensors in the sensor unit 110 can be used to acquire raw image data.

[0044] In this embodiment, the sensor unit 110 is connected to the FPGA unit 120 and transmits data to the FPGA (Field-Programmable Gate Array) unit through a target interface. The target interface can be a MIPI (Mobile Industry Processor Interface) interface or a USB (Universal Serial Bus) interface. It should be noted that the target interface can also be other types of interfaces capable of transmitting image data; this embodiment does not impose specific limitations, as long as the target interface enables the transmission of raw image data acquired by the sensor unit 110 to the FPGA unit 120.

[0045] The aforementioned FPGA unit 120 can be a hardware circuit system with specific functions built around a Field-Programmable Gate Array (FPGA). An FPGA is a semiconductor device whose logic functions and interconnections can be configured by the user through a hardware description language, offering high flexibility and reconfigurability. The FPGA unit 120 integrates at least an image preprocessing subunit for performing ISP processing on the data, and a network model subunit for image enhancement of the ISP-processed data using a loaded target network model. The aforementioned ISP (Image Signal Processing) processing can include one or more operations such as denoising, depigmentation, white balance, color correction, shadow correction, and distortion correction. It should be noted that different image sensors differ in design and manufacturing; some sensors may have some ISP processing functions built-in at the factory, such as depigmentation and white balance. Therefore, in practical applications, these sensors may only require a few specific preprocessing steps. For example, some high-performance CMOS (Complementary Metal-Oxide-Semiconductor) sensors may have built-in advanced demosaic algorithms, thus reducing the need for a demosaic module in the preprocessing stage. Therefore, during ISP processing, preprocessing modules can be flexibly selected and configured according to the specific sensor type and application requirements. For instance, in applications with extremely high image quality requirements, all preprocessing modules may need to be enabled to ensure image accuracy and realism; while in applications with high real-time requirements, the preprocessing flow may need to be simplified to improve system processing speed. Furthermore, the purpose of performing ISP processing on one of the two data streams is to restore the raw image data captured by the sensor to the real scene as accurately as possible, providing a high-quality data foundation for subsequent image processing and analysis.

[0046] The image enhancement methods described above can include one or more of the following: denoising (removing noise from an image, such as Gaussian noise, salt-and-pepper noise, etc., to make the image clearer), super-resolution (increasing the resolution of an image), contrast enhancement (improving the contrast of an image, making details in the image clearer), color enhancement (adjusting the color saturation, hue, etc. of an image to make the image colors more vibrant and natural), brightness adjustment (improving the brightness of an image so that the image maintains good visual effects under different lighting conditions), dynamic range compression (compressing a high dynamic range image into a low dynamic range image while preserving as much detail as possible), sharpening (enhancing the edges and details of an image to make the image look clearer), deblurring (removing blur from an image to restore image clarity), image inpainting (repairing missing or damaged parts of an image to make the image complete), image deshading, image demoiring, and image deblocking.

[0047] Furthermore, the aforementioned host computer unit 140 can be a hardware platform built on a high-performance computer or server, possessing powerful computing capabilities, storage capacity, and abundant interface resources. The hardware platform of the host computer unit 140 can be configured according to specific requirements; this embodiment does not impose specific limitations, as long as the host computer unit 140 can determine the target network model corresponding to the current application scenario based on the received second-channel image data, and then send the target network model to the FPGA unit 120 for loading. These specific requirements may include one or more of the following: processing speed, data throughput, and scalability.

[0048] In this embodiment, the host computer unit 140 determines the current application scenario by analyzing the image content based on the received second-channel image data. Different application scenarios may require different image enhancement strategies and network models. After determining the application scenario, the host computer unit 140 selects the corresponding target network model and sends it to the FPGA unit 120 for loading and use. This allows the system to flexibly adjust the image enhancement strategy according to different application scenarios, improving the adaptability and intelligence level of the image processing system.

[0049] The aforementioned display unit 130 includes at least a display screen for displaying the received first-channel image data. The display screen can be one of a liquid crystal display, an organic light-emitting diode display, or a cathode ray tube display, etc.

[0050] In this embodiment, the image processing system divides the image data into two identical data streams. The first stream is processed and displayed, while the second stream is used to select a model. The selected target network model is then sent to the FPGA unit 120 for image processing of the first stream image data. By matching the host computer network model with the environment, the computational burden of matching the FPGA with the environment is avoided. This system can realize the accelerator function of the FPGA and ensure that the target network model equipped on the FPGA is adapted to the environment. This solves the problem that existing image processing systems, when the FPGA is used as an accelerator for AIISP, have limited applicable environments and cannot meet the requirements of different application environments.

[0051] Figure 2 This is a second structural block diagram of the image processing system provided in this embodiment, as shown below. Figure 2 As shown, the FPGA unit 120 includes an image segmentation subunit 122, an image preprocessing subunit 124, and a network model subunit 126. The image segmentation subunit 122 connects the sensor unit 110, the image preprocessing subunit 124, and the host computer unit 140. It is used to receive the raw image data sent by the sensor unit 110, divide the received raw image data into two identical data streams, send one stream to the image preprocessing subunit 124, and send the other stream as the second image data stream to the host computer unit 140. The image preprocessing subunit 126... The processing subunit 124 is connected to the network model subunit 126 and is used to perform ISP processing on the received data to obtain the ISP-processed data, and send the ISP-processed data to the network model subunit 126. The network model subunit 126 is connected to the host computer unit 140 and the display unit 130 and is used to load the target network model provided by the host computer unit 140, and use the loaded target network model to perform image enhancement on the received ISP-processed data to obtain the first channel image data, and transmit the first channel image data to the display unit 130.

[0052] The target network model described above can be a convolutional neural network model, such as a super-resolution convolutional neural network model or a deep convolutional neural network model. It should be noted that the target network model described above can also be one or more of the following: generative adversarial network model (e.g., super-resolution generative adversarial network or recurrent generative adversarial network), recurrent neural network (e.g., convolutional long short-term memory network), attention mechanism network (e.g., residual channel attention network).

[0053] Specifically, in one embodiment, the image segmentation subunit 122 is also used to copy the received original image data and divide the copied data into two identical data streams.

[0054] In this embodiment, after receiving the original image data sent by the sensor unit 110, the image segmentation unit copies the original image data to obtain copied data. The original image data and the copied data have the same data content. The original image data and the copied data can be used as two data streams. One data stream (first data stream) is used for image processing and image display, and the other data stream (second data stream) is used for model training to train the neural network model used by the first data stream. This allows the parameters of the image processing model to be updated based on the received data without interrupting the display of the first data stream.

[0055] In another embodiment, the network model subunit 126 is also used to update the loaded target network model in response to the parameters of the trained target network model sent by the host computer unit 140.

[0056] In this embodiment, the FPGA unit 120 loads the target network model from the host computer unit 140 to perform image enhancement on the data processed by the ISP. When the target network model in the FPGA unit 120 performs image enhancement on the data processed by the ISP, if the current parameters of the target network model are not compatible with the current environment, or if the current environment changes, it is not necessary to stop the image enhancement process of the target network model. Instead, the trained target network model parameters sent by the host computer unit 140 can be received to update the parameters of the target network model to be more compatible with the current environment without stopping the operation of the target network model. The aforementioned trained target network model parameters are obtained by training a network model identical to the target network model using the same second-channel data as the data processed by the target network model. These parameters are adapted to the target network model, enabling the target network model to continuously upgrade the image enhancement effect.

[0057] Figure 3 This is a third structural block diagram of the image processing system provided in this embodiment, such as... Figure 3 As shown, the FPGA unit 120 also includes an image post-processing subunit 128; the image post-processing subunit 128 is connected to the network model subunit 126 and the display unit 130, and is used to receive the first channel image data sent by the network model subunit 126; perform display configuration on the received first channel image data to obtain the configured first channel image data; and send the configured first channel image data to the display unit 130.

[0058] Generally, the first-channel image data output by the network model subunit 126 does not meet the transmission or display format of the backend display. Therefore, the image post-processing subunit 128 is needed to configure the display of the first-channel image data so that the configured first-channel image data can meet the interface requirements of the backend display. For example, if the backend display interface is a different interface such as HDMI (High-Definition Multimedia Interface) or DP (DisplayPort), the first-channel image data needs to be configured differently so that the configured first-channel image data can be adapted to the display of the display unit 130. The above display configuration may involve adjusting the format of the first-channel image data so that the configured first-channel image data can be sent to the display unit 130 for normal display. The content of the above display configuration can be determined based on the parameters of the display of the display unit 130.

[0059] in addition, Figure 4 This is the fourth structural block diagram of the image processing system provided in this embodiment, as shown below. Figure 4 As shown, the host computer unit 140 includes: a host computer subunit 142 and a cloud subunit 144; the host computer subunit 142 is connected to the image segmentation subunit 122, the network model subunit 126, and the cloud subunit 144, and is used to receive the second-channel image data sent by the image segmentation subunit 122; based on the received second-channel image data, determine the target network model corresponding to the current application scenario; and send the target network model to the network model subunit 126 for loading; the host computer subunit 142 is also used to transfer the received second-channel image data and the determined target network model to the cloud subunit 144. The model number of the target network model is sent to the cloud subunit 144; the cloud subunit 144 is used to determine the network model to be trained based on the received model number; the target network model to be trained is the same as the target network model; the cloud subunit 144 is also used to train the network model to be trained based on the received second-channel image data, obtain the parameters of the trained target network model, and send the parameters of the trained target network model to the host computer subunit 142; the host computer subunit 142 is also used to send the received parameters of the trained target network model to the network model subunit 126.

[0060] In this embodiment, the cloud subunit 144 pre-stores a network model library to provide network models to be trained. This allows the network model to be trained based on the received second-channel image data, obtaining the parameters of the trained target network model. The trained target network model parameters are then sent to the host computer subunit 142. The host computer subunit 142's configuration storage component also pre-stores a network model library. Based on the received model number, it determines the target network model corresponding to that model number and sends it to the configuration control component. This enables the target network model to be distributed to the FPGA unit 120 for image enhancement of the data processed by the ISP. The model number of the network model can be a unique identifier assigned to each network model during network model setup, allowing for differentiation between different network models.

[0061] It should be noted that, because the model number is a unique identifier for the network model, when the model number is determined, the network model to be trained determined in the cloud sub-unit 144 and the target network model determined in the configuration storage component are the same model. Furthermore, since the second-channel image data is the same data as the data processed by ISP in the FPGA unit 120, the parameters obtained by training the network model to be trained using the second-channel image data are also applicable to the target network model and can be used as parameters for the trained target network model.

[0062] Furthermore, in one embodiment, the host computer subunit 142 is also configured to determine the current application scenario based on the received second-channel image data; and determine the target network model corresponding to the current application scenario based on the current application scenario.

[0063] In this embodiment, the host computer unit 140 receives second-channel image data from the FPGA unit 120. This data is the raw image data without enhancement processing, preserving the true features and information of the image. The determination of the current application scenario based on the received second-channel image data can be achieved by pre-storing parameters corresponding to different application scenarios (a preset application scenario feature library). Based on the received second-channel image data, feature extraction and analysis are performed to obtain the parameters of the second-channel image data. Based on the parameters of the second-channel image data and the pre-stored parameters corresponding to different application scenarios, the application scenario corresponding to the second-channel image data is determined, i.e., the current application scenario is determined. Specifically, before performing feature extraction and analysis on the second-channel image data, necessary preprocessing operations, such as noise reduction, scaling, and normalization, are required to improve the accuracy and efficiency of subsequent processing. The purpose of preprocessing is to eliminate irrelevant information in the image, highlight the image features, and make it more suitable for analysis and recognition. The aforementioned feature extraction and analysis of the second-channel image data can utilize image processing algorithms and deep learning models to extract key feature information, such as color, texture, shape, and spatial relationship features. Then, the extracted features are analyzed to identify objects, scenes, behaviors, and other elements within the second-channel image data. For example, in medical imaging scenarios, feature analysis might include identifying organs and lesion areas in the second-channel image data; in security monitoring scenarios, it might involve identifying people, vehicles, and actions. Finally, the image features of the second-channel image data are matched against a pre-defined application scenario feature library to calculate the similarity or matching degree between the image features of the second-channel image data and the features of each scene. The scene with the highest matching degree is the application scenario corresponding to the second-channel image data. This pre-defined application scenario feature library stores typical feature information for various application scenarios, used for comparison and matching with real-time image features.

[0064] The above-mentioned determination of the target network model corresponding to the current application scenario can be based on a pre-stored network model library. There is a correspondence between each network model and the application scenario. Therefore, the target network model corresponding to the current application scenario can be directly determined through the current application scenario.

[0065] Figure 5 This is the fifth structural block diagram of the image processing system provided in this embodiment, as shown below. Figure 5As shown, the host computer subunit 142 includes: a data and parameter management component 1422, a configuration storage component 1424, and a configuration control component 1426. The data and parameter management component 1422 is connected to the cloud subunit 144, the image segmentation subunit 122, and the configuration storage component 1424. It is used to receive the second-channel image data sent by the image segmentation subunit 122 and send the received second-channel image data to the cloud subunit 144. The data and parameter management component 1422 is also used to determine the current application scenario based on the received second-channel image data; determine the target network model corresponding to the current application scenario based on the current application scenario; and determine the model of the target network model based on the target network model corresponding to the current application scenario. The model is assigned a model number and sent to the configuration storage component 1424 and the cloud subunit 144 respectively. The data and parameter management component 1422 is also used to receive the parameters of the trained target network model sent by the cloud subunit 144 and send the received parameters of the trained target network model to the network model subunit 126. The configuration storage component 1424 is connected to the configuration control component 1426 and is used to determine the target network model corresponding to the received model number and send the target network model to the configuration control component 1426. The configuration control component 1426 is connected to the network model subunit 126 and is used to distribute the received target network model to the network model subunit 126 for loading.

[0066] It should be noted that in order for the network model subunit 126 to load the target network model, a configuration interface component can be set in the network model subunit 126. The target network model in the configuration control component 1426 can be loaded through the configuration interface component to achieve normal loading of the target network model.

[0067] In one embodiment, the FPGA unit 120 further includes an interface configuration subunit; the interface configuration subunit is connected to the image segmentation subunit 122 and the sensor unit 110, and is used to send preset interface configuration information to the sensor unit 110 and send the received raw image data to the image segmentation subunit 122.

[0068] The aforementioned preset interface configuration information can be used to set relevant parameters when the sensor unit 110 acquires image data, ensuring that image data can be acquired and transmitted according to system requirements. The preset interface configuration information includes at least one of the following: frame rate, resolution, and exposure parameters. This interface configuration information corresponds to the image acquisition configuration parameters, determining which parameters are included and the specific content of each parameter. The aforementioned interface configuration subunit can send the preset interface configuration information to the sensor unit 110 via the IIC (Inter-Integrated Circuit) interface.

[0069] In another embodiment, the sensor unit 110 is also configured to determine image acquisition configuration parameters based on the received interface configuration information; the configuration parameters include at least one of acquisition frame rate, resolution, or exposure parameters; and acquire raw image data based on the image acquisition configuration parameters.

[0070] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0071] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0072] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0073] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image processing system, characterized by, The system comprises a sensor unit, an FPGA unit, a display unit and an upper computer unit; The sensor unit is connected to the FPGA unit and is configured to collect original image data and transmit the collected original image data to the FPGA unit; The FPGA unit is connected to the upper computer unit and the display unit and is configured to divide the received original image data into two same data, perform ISP processing on one of the two data, perform image enhancement on the data after ISP processing by using a loaded target network model to obtain first image data, and send the first image data to the display unit; the FPGA unit is further configured to send the other of the two data as second image data to the upper computer unit; The upper computer unit is configured to determine a target network model corresponding to a current application scenario based on the received second image data, and distribute the target network model to the FPGA unit for loading by the FPGA unit; The display unit is configured to display the received first image data.

2. The image processing system of claim 1, wherein, The FPGA unit comprises an image division subunit, an image preprocessing subunit and a network model subunit; The image division subunit is connected to the sensor unit, the image preprocessing subunit and the upper computer unit, configured to receive the original image data sent by the sensor unit, divide the received original image data into two same data, send one of the two data to the image preprocessing subunit, and send the other of the two data as the second image data to the upper computer unit; The image preprocessing subunit is connected to the network model subunit and is configured to perform ISP processing on the received data to obtain data after ISP processing, and send the data after ISP processing to the network model subunit; The network model subunit is connected to the upper computer unit and the display unit, configured to load the target network model provided by the upper computer unit, perform image enhancement on the received data after ISP processing by using the loaded target network model to obtain the first image data, and transmit the first image data to the display unit.

3. The image processing system of claim 2, wherein, The image division subunit is further configured to copy the received original image data and divide the copied data into two same data.

4. The image processing system of claim 2, wherein, The network model subunit is further configured to update the loaded target network model in response to the received parameters of the trained target network model sent by the upper computer unit.

5. The image processing system of claim 2, wherein, The FPGA unit further comprises an image post-processing subunit; The image post-processing subunit is connected to the network model subunit and the display unit and is configured to receive the first image data sent by the network model subunit. The first image data received is configured to obtain configured first image data; and the configured first image data is sent to the display unit.

6. The image processing system of claim 4, wherein, The upper computer unit comprises an upper computer subunit and a cloud subunit. The upper computer subunit is connected with the image division subunit, the network model subunit and the cloud subunit, receives the second image data sent by the image division subunit, determines a target network model corresponding to a current application scenario based on the received second image data, and sends the target network model to the network model subunit for loading. The cloud subunit determines a network model to be trained based on the received model number, and trains the network model to be trained based on the received second image data to obtain parameters of a trained target network model and sends the parameters to the upper computer subunit. The upper computer subunit further sends the received parameters of the trained target network model to the network model subunit.

7. The image processing system of claim 6, wherein, The upper computer subunit determines the current application scenario based on the received second image data, and determines the target network model corresponding to the current application scenario based on the current application scenario.

8. The image processing system of claim 7, wherein, The upper computer subunit comprises a data and parameter management component, a configuration storage component and a configuration control component. The data and parameter management component is connected with the cloud subunit, the image division subunit and the configuration storage component, receives the second image data sent by the image division subunit, and sends the received second image data to the cloud subunit. The configuration storage component is connected with the configuration control component, determines the target network model corresponding to the model number based on the received model number, and sends the target network model to the configuration control component. The configuration control component is connected with the network model subunit, receives the target network model sent by the configuration storage component, and sends the target network model to the network model subunit for loading. The configuration control component is connected with the network model subunit and is configured to distribute the received target network model to the network model subunit for loading by the network model subunit.

9. The image processing system of claim 2, wherein, The FPGA unit further includes an interface configuration subunit. The interface configuration subunit is connected with the image division subunit and the sensor unit, and is configured to send preset interface configuration information to the sensor unit and send the received original image data to the image division subunit.

10. The image processing system of claim 9, wherein, The sensor unit is further configured to determine configuration parameters of image acquisition based on the received interface configuration information, and the configuration parameters include at least one of acquisition frame rate, resolution, or exposure parameter. The original image data is acquired based on the configuration parameters of image acquisition.

Images