Image processing system and image processing method

Abstract

The invention provides an image processing system and an image processing method, and the system comprises an image preprocessing module, a distributed processing module, and a video stream generation module. Wherein the image preprocessing module acquires original image data from the image acquisition equipment, decodes the original image data to generate original video data and sends the original video data; the distributed processing module comprises at least two cache processing modules with different cache requirements and delay requirements, and the two cache processing modules sequentially perform image processing on the original video data to generate cached video data; the video stream generation module performs framing coding processing on the cached video data to generate video stream data, and sends the video stream data to an external display device; and the image preprocessing module, the distributed processing module, the video stream generation module and each cache processing module in the distributed processing module perform parallel processing on the received data. According to the scheme, time delay can be reduced, and the display effect of the external display equipment cannot be reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly relates to an image processing system and an image processing method. Background Art

[0002] With the development of technology, people have more needs for "face-to-face" or the feeling of being on-site. Therefore, real-time video transmission and image processing technologies have developed rapidly. In the scenario of real-time video transmission (such as video conferencing, live streaming, video teaching, and even the streaming rearview mirror of a vehicle), the video is acquired by an image acquisition device, encoded, and then uploaded to a processor, and then sent by the processor to a display. The display receives the video frame forwarded from the processor, thereby realizing the real-time video function.

[0003] However, the existing real-time video transmission technologies all have the common problem of excessive delay. In order to reduce the processing delay, in the prior art, generally, the timestamp of the data frame is detected, and after being compared with the system timestamp, offset value compensation and data frame discarding are performed to control the system delay within the expected range. For example, as mentioned in the patent document CN108540855B, in the live streaming scenario, for audio playback, frame discarding is performed according to the chasing frame threshold range, and video synchronization discards packets before decoding, thereby reducing the delay. However, randomly discarding frames will cause a sense of visual discontinuity and affect the display effect.

[0004] Therefore, the existing image processing systems have the problems of long delay and poor display effect easily caused by using the method of discarding frames to reduce the delay. Summary of the Invention

[0005] The purpose of the present invention is to solve the problems in the prior art that the existing image processing systems have long delay and poor display effect easily caused by using the method of discarding frames to reduce the delay.

[0006] To solve the above problems, an embodiment of the present invention discloses an image processing system, including: an image preprocessing module, which obtains original image data from an image acquisition device, decodes the original image data to generate original video data, and sends the original video data; a distributed processing module, which is communicatively connected to the image preprocessing module and includes at least two cache processing modules with different cache requirements and latency requirements. The two cache processing modules sequentially perform image processing on the original video data from the image preprocessing module to generate cached video data; a video stream generation module, which is communicatively connected to the distributed processing module, obtains the cached video data from the distributed processing module, performs frame encoding processing on the cached video data to generate video stream data, and sends the video stream data to an external display device. Moreover, the image preprocessing module, the distributed processing module, and the video stream generation module all perform parallel processing on the received original image data, original video data, and cached video data; each cache processing module in the distributed processing module performs parallel processing on the received data.

[0007] With the above solution, each processing module processes data in parallel, saving the time for solving large and complex problems during the processing, and effectively reducing the latency of the image processing system. Moreover, the distributed processing module is divided into multiple cache processing modules according to cache requirements and latency requirements, effectively realizing the hierarchical management of processing latency and further reducing the latency of the image processing system. Furthermore, the image processing system fundamentally reduces the processing latency through parallel processing, effectively avoiding problems such as visual discomfort caused by dropped frames and poor display effects.

[0008] According to another specific embodiment of the present invention, in the image processing system disclosed in the embodiment of the present invention, at least two cache processing modules include a pixel-level image processing module and a line buffer type image processing module; wherein, the pixel-level image processing module is communicatively connected to the image preprocessing module and the line buffer type image processing module respectively. The pixel-level image processing module performs primary image processing on the original video data from the image preprocessing module to generate primary video data, and sends the primary video data to the line buffer type image processing module. The line buffer type image processing module performs secondary image processing on the received primary video data to generate secondary video data, and sends the secondary video data as cached video data to the video stream generation module; wherein, the primary image processing includes at least one of color interpolation processing, black level correction processing, white balance processing, color space conversion processing, and gray scale correction processing; the secondary image processing includes at least one of 2D denoising processing, edge enhancement processing, and image scaling processing.

[0009] With the above solution, the video display effect can be effectively enhanced through primary image processing and secondary image processing.

[0010] According to another specific embodiment of the present invention, in the image processing system disclosed in the embodiment of the present invention, at least two cache processing modules further include a frame buffer type image processing module; wherein, the frame buffer type image processing module is respectively communicatively connected to the line buffer type image processing module and the video stream generation module. The frame buffer type image processing module receives the secondary video data from the line buffer type image processing module, performs tertiary image processing on the secondary video data to generate tertiary video data, and sends the tertiary video data as cached video data to the video stream generation module; wherein, the tertiary image processing includes histogram enhancement processing, 3D denoising processing, and high dynamic range imaging processing. Moreover, the cache requirements and latency requirements of the pixel-level image processing module, the line buffer type image processing module, and the frame buffer type image processing module increase in sequence.

[0011] By adopting the above solution, the distributed processing module is further subdivided into a pixel-level image processing module with extremely small cache and extremely low latency, a line buffer type image processing module with relatively small cache and relatively low latency, and a frame buffer type image processing module with relatively large cache and relatively high latency, effectively realizing hierarchical management of processing latency and providing a basis and conditions for realizing real-time video transmission with extremely low latency.

[0012] According to another specific embodiment of the present invention, the image processing system disclosed in the embodiment of the present invention further includes an embedded processing module; the embedded processing module is respectively communicatively connected to the image acquisition device, the external display device, the image preprocessing module, the distributed processing module, and the video stream generation module; wherein, the embedded processing module receives the image device status information sent by the image acquisition device and performs fault monitoring on the image acquisition device according to the image device status information. The embedded processing module receives the display device status information sent by the external display device and performs status monitoring on the external display device according to the display device status information. The embedded processing module sends the pre-stored processing parameter data to the distributed processing module at a predetermined interval and performs parameter setting on the distributed processing module, and sends the pre-stored latency threshold to the video stream generation module and performs latency control on the video stream generation module. Moreover, the embedded processing module is further communicatively connected to the environmental data acquisition device, obtains the external environmental data from the environmental data acquisition device, generates scene information according to the external environmental data, and sends the scene information to the distributed processing module, and the distributed processing module determines the steps of performing image processing on the original video data according to the scene information.

[0013] By adopting the above solution, the embedded processing module is used to detect possible faults during the image processing process and perform more precise control on the latency, and the data acquired by the image acquisition device can be displayed on the external display device with relatively low latency and relatively high display effect.

[0014] According to another specific embodiment of the present invention, in the image processing system disclosed in the embodiments of the present invention, the environmental data acquisition device includes a speed sensor and a light sensor; the external environmental data includes the moving speed data of the image acquisition device obtained by the speed sensor and the light intensity information of the external environment obtained by the light sensor; the scene information includes a high-speed daytime scene, a high-speed nighttime scene, a low-speed daytime scene, and a low-speed nighttime scene. Moreover, in the high-speed daytime scene, the distributed processing module omits the steps of the three-level image processing of the frame buffer type image processing module and directly sends the secondary video data to the video stream generation module as the cached video data; in the high-speed nighttime scene, the distributed processing module only retains the histogram enhancement processing in the three-level image processing and sends the generated three-level video data to the video stream generation module as the cached video data; in the low-speed daytime scene and the low-speed nighttime scene, the distributed processing module retains the histogram enhancement processing in the three-level image processing and retains at least one of the 3D denoising processing and the high-dynamic range imaging processing, and sends the three-level video data generated by the frame buffer type image processing module to the video stream generation module as the cached video data. Moreover, the image device state information includes at least one of the acceleration change rate and the motion direction of the image acquisition device; the display device state information includes the frame interval between two adjacent frames of the video displayed by the external display device.

[0015] Adopting the above solution, the specific scenes of the image processing system are classified according to the speed and brightness that can have a major impact on the user's visual perception, and the corresponding steps of image processing are omitted or retained in a targeted manner in different scenes, which can meet the different requirements for latency and visual effects in different scenes.

[0016] According to another specific embodiment of the present invention, in the image processing system disclosed in the embodiments of the present invention, the image preprocessing module, the pixel-level image processing module, the line buffer type image processing module, the frame buffer type image processing module, the video stream generation module, and the embedded processing module are all integrated on the same integrated chip; moreover, a register and several static random access memory modules are also provided on the integrated chip, wherein the frame buffer type image processing module and the video stream generation module are both connected to the register, and the register is connected to the external dynamic random access memory module; the pixel-level image processing module and the line buffer type image processing module are each connected to at least one static random access memory module.

[0017] With the above solution, the cache requirements of the pixel-level image processing module with extremely small cache and extremely low latency, as well as the line buffer type image processing module with relatively small cache and relatively low latency, are met by the static random access memory module inside the integrated chip, without the need to connect to an external storage module, reducing the occupied space of the image processing system. The frame buffer type image processing module with relatively large cache and relatively high latency, as well as the video stream generation module, are connected to the external dynamic random access memory module via the registers inside the integrated chip, which can improve the processing efficiency, reduce the latency, and meet the relatively high storage requirements.

[0018] According to another specific embodiment of the present invention, for the image processing system disclosed in the embodiment of the present invention, the image acquisition device is a vehicle-mounted camera; the external display device is the streaming media rearview mirror of the vehicle; and, the embedded processing module is further communicatively connected to the vehicle's vehicle controller.

[0019] The embodiment of the present invention discloses an image processing method for an image processing system, including:

[0020] S1: Obtain the original image data, and perform decoding processing on the original image data to generate the original video data;

[0021] S2: Obtain the external environment data, generate the scene information according to the external environment data, and perform image processing on the original video data according to the scene information to generate the cached video data; wherein, the image processing includes at least primary image processing and secondary image processing;

[0022] S3: Perform frame encoding processing on the cached video data to generate the video stream data.

[0023] According to another specific embodiment of the present invention, for the image processing method disclosed in the embodiment of the present invention, in step S2, the primary image processing includes at least one of color interpolation processing, black level correction processing, white balance processing, color space conversion processing, and gray scale correction processing; the secondary image processing includes at least one of 2D denoising processing, edge enhancement processing, and image scaling processing; and, the image processing further includes tertiary image processing, wherein, the tertiary image processing includes histogram enhancement processing, 3D denoising processing, and high dynamic range imaging processing; and, the external environment data includes the moving speed data of the image acquisition device for obtaining the original image data, and the light intensity information of the external environment; the scene information includes high-speed daytime scenes, high-speed nighttime scenes, low-speed daytime scenes, and low-speed nighttime scenes.

[0024] According to another specific embodiment of the present invention, in the image processing method disclosed in the embodiment of the present invention, in step S2, image processing is performed on the original video data according to the scene information to generate cached video data, including: in the high-speed daytime scene, performing primary image processing and secondary image processing, and omitting the step of tertiary image processing; in the high-speed nighttime scene, performing primary image processing and secondary image processing, and only retaining the histogram enhancement processing in the tertiary image processing; in the low-speed daytime scene and the low-speed nighttime scene, performing primary image processing and secondary image processing, and retaining at least one of the histogram enhancement processing, 3D denoising processing, and high-dynamic range imaging processing in the tertiary image processing.

[0025] With the above solution, the specific scenes of the image processing system are classified according to the speed and brightness that can have a major impact on the user's visual perception, and the corresponding steps of image processing are omitted or retained in a targeted manner in different scenes, which can meet the control requirements of extremely low latency in high-speed scenes, while in low-speed scenes, the display effect can be improved as much as possible on the basis of meeting the basic latency requirements.

[0026] The beneficial effects of the present invention are:

[0027] The image processing system provided by the present application uses distributed processing technology. On the one hand, it uses a modular pipeline method to complete the processing of the original image data acquired by the image acquisition device, and each processing module works in full parallel in a cascaded manner to ensure the real-time performance of signal processing; on the other hand, it classifies the cache requirements and latency requirements of the processing modules, and tries to use distributed storage resources to implement different levels of data cache processing, so as to effectively reduce the reuse requirements for a single memory and thus effectively reduce the waiting time for data access, thereby effectively reducing the latency of image processing. Further, a flexible distributed image processing allocation technology is introduced, and the processing parameters of the distributed processing module for image processing are dynamically adjusted according to the vehicle speed and light information. In particular, in high-speed scenes, the extremely low latency effect under extreme conditions can be effectively improved by omitting high-latency frame buffer-like processing, and in low-speed scenes, frame buffer-like processing is retained to effectively improve the display effect, which can better meet the needs of the actual scene. Furthermore, compared with the frame dropping and packet dropping control methods in the prior art, the present application fundamentally reduces the processing latency through parallel processing and processing algorithm optimization control, not only effectively avoiding the visual discomfort caused by frame dropping, but also achieving a lower processing latency. Description of the Drawings

[0028] Figure 1 is a schematic structural diagram of the image processing system provided by the embodiment of the present invention;

[0029] Figure 2 is another schematic structural diagram of the image processing system provided by the embodiment of the present invention;

[0030] Figure 3 is another structural schematic diagram of the image processing system provided by the embodiments of the present invention;

[0031] Figure 4 is a schematic flowchart of the image processing method provided by the embodiments of the present invention.

[0032] Description of reference numerals:

[0033] 1. Image preprocessing module; 2. Distributed processing module; 21. Pixel-level image processing module; 22. Row buffer type image processing module; 23. Frame buffer type image processing module; 3. Video stream generation module; 4. Embedded processing module; 5. Image acquisition device; 6. External display device; 7. Environmental data acquisition device. Detailed implementation manners

[0034] Embodiment 1:

[0035] To solve the problems in the prior art that the image processing system has a long delay, and reducing the delay by the method of dropping frames is likely to result in a poor display effect, this embodiment provides an image processing system. Specifically, referring to Figure 1 , the image processing system includes an image preprocessing module 1, a distributed processing module 2, and a video stream generation module 3. Among them, the distributed processing module 2 includes at least two cache processing modules with different cache requirements and delay requirements.

[0036] Further, the image preprocessing module 1 obtains the original image data from the image acquisition device 5, decodes the original image data to generate the original video data, and sends the original video data. Specifically, the image acquisition device 5 can be a camera whose output result is an image or a video camera whose output result is a video. Therefore, the original image data can be image data or video data. The image preprocessing module 1 can obtain the original image data from the image acquisition device 5 through high-speed transmission protocols such as LVDS, MIPI, SPD-LINK, GMSL, etc. The decoding process can convert the image or video output by the image acquisition device 5 into a data form readable by the processor. Specifically, a specific interface of the image acquisition device 5 (such as a GMSL interface) can deserialize the image or video data through a decoder (such as a MAX96706 decoder) and then send it to the distributed processing module 2 through a specific data transmission bus interface (such as AXI, AHB, or other custom video stream buses). It should be noted that the data delay of the image preprocessing module 1 is relatively low. More specifically, regardless of whether the data type obtained by the image preprocessing module 1 from the image acquisition device 5 is an image or a video, the image preprocessing module 1 can convert the image or video into video data. Among them, the image preprocessing module 1 synthesizes the image data to form video data and decodes the video data, or the image preprocessing module 1 directly decodes the video data.

[0037] The distributed processing module 2 is communicatively connected to the image preprocessing module 1, and at least two cache processing modules included in the distributed processing module 2 sequentially perform image processing on the original video data from the image preprocessing module 1 to generate cached video data. Specifically, the distributed processing module 2 is mainly used to perform image processing such as correction, denoising, scaling, enhancement, etc. on the original video data. The distributed processing module 2 occupies the main processing and storage resources of the image processing system, so it also causes the main delay of the external display device 6. And at least two cache processing modules can sequentially perform pixel-level processing, pixel-block-level processing, or even frame-level processing on the original video data. It should be noted that when performing the above processing, the original video data needs to be classified according to the storage requirements and processing levels first. Generally speaking, for pixel-level processing, pixel-block-level processing, and frame-level processing, the required cache requirements and delay requirements are relatively large. Among them, the cache requirement refers to the requirement for cache resources, and the delay requirement refers to the time length of data processing delay. The greater the requirement of data for cache resources, the greater the corresponding processing delay, and more data processing delay needs to be introduced.

[0038] The video stream generation module 3 is communicatively connected to the distributed processing module 2, obtains the cached video data from the distributed processing module 2, performs frame encoding processing on the cached video data to generate video stream data, and sends the video stream data to the external display device 6. Specifically, the video stream generation module 3 is used to collect the video data processed by the distributed processing module 2, and perform image data framing and encoding output processing according to specified conditions (such as the limit conditions of system delay and frame interval) to generate video stream data, and send it to the external display device 6 in real time. More specifically, the encoding processing can be performed using high-speed transmission protocols such as LVDS, MIPI, SPD-LINK, GMSL, etc. and sent to the external display device 6 via a specified interface. The external display device 6 is generally a display terminal. It should be noted that the cache required by the video stream generation module 3 is related to the delay generated by the required control.

[0039] Furthermore, referring to Figure 1 , the image preprocessing module 1, the distributed processing module 2, and the video stream generation module 3 all perform parallel processing on the received original image data, original video data, and cached video data; and each cache processing module in the distributed processing module 2 performs parallel processing on the received data. That is to say, each processing module of this image system can work simultaneously to process data. It should be noted that the image preprocessing module 1, the distributed processing module 2, and the video stream generation module 3 can all be integrated into a distributed processor, such as an FPGA chip of Xilinx, for implementation. As for the specific image preprocessing module 1, distributed processing module 2, and video stream generation module 3, they are all encapsulated into IP protocols that can perform parallel processing inside the FPGA chip. And for the processing of small caches with low latency, the required cache resources can be implemented using internal BRAM in the FPGA; for the processing of large caches with high latency, the required cache resources can be implemented by calling an external memory module (such as a DDR3 memory chip).

[0040] With such a structure, by dividing the entire process of real-time video transmission and allocating different parts to different processing modules, each processing module can process data in parallel, saving the time to solve large and complex problems during the processing, and effectively reducing the latency of the image processing system. And the distributed processing module 2 is divided into multiple cache processing modules according to the cache requirements and latency requirements, and the data processing is divided into processing with small caches and low latency, and processing with large caches and high latency, effectively realizing the hierarchical management of processing latency and further reducing the latency of the image processing system. And this image processing system fundamentally reduces the processing latency through parallel processing, effectively avoiding problems such as visual discomfort caused by dropped frames and poor display effects.

[0041] Further, in the image processing system according to the present invention, referring to Figure 1 , at least two cache processing modules include a pixel-level image processing module 21 and a line cache type image processing module 22. Among them, the pixel-level image processing module 21 processes the received data at the pixel level. During the processing, only several pixel points need to be cached, so the cache requirement and latency requirement are both very small. The line cache type image processing module 22 processes the received data at the pixel block level. During the processing, the data needs to be processed in units of several arranged data within a pixel block or the adjacent area of pixel points. The cache requirement and latency requirement of this line cache type image processing module 22 are both in units of lines, generally not exceeding 0.5 frames. Therefore, the cache requirement and latency requirement are greater than those of the pixel-level image processing module 21.

[0042] Specifically, the pixel-level image processing module 21 is communicatively connected to the image preprocessing module 1 and the line cache type image processing module 22 respectively. The pixel-level image processing module 21 performs primary image processing on the original video data from the image preprocessing module 1 to generate primary video data, and sends the primary video data to the line cache type image processing module 22. The line cache type image processing module 22 performs secondary image processing on the received primary video data to generate secondary video data, and sends the secondary video data as cached video data to the video stream generation module 3.

[0043] More specifically, the primary image processing includes at least one of color interpolation processing, black level correction processing, white balance processing, color space conversion processing, and gray scale correction processing. Among them, color interpolation processing (DEMOSAIC) is to convert the Bayer data sensed by an image sensor (CMOS image sensor, CIS) into a complete true color image (RGB data format) that can be seen by the human eye using the nearest neighbor interpolation method. The black level correction processing is to obtain the level when the CIS senses all black using the fixed value correction method, and eliminate its influence through re-quantization. The white balance processing generally uses an automatic method and adjusts the colors of the captured photos to be exactly the same as the colors of the scenery seen by the human eye under various lighting conditions by adjusting the R and B gains in RGB. The color space conversion processing is to convert the image between different color spaces by transforming between the RGB and YUV domains. The gray scale correction processing (GAMMA correction) is a non-linear operation on the gray scale values of the input image by using a low GAMMA value correction to enhance the contrast in the dark areas of the image, so that the gray scale values of the output image and the input image are in an exponential relationship.

[0044] Secondary image processing includes at least one of 2D denoising processing, edge enhancement processing, and image scaling processing. Among them, 2D denoising processing generally uses improved Gaussian filtering to achieve image noise reduction, thereby suppressing the noise of digital images within a single-frame image. Edge enhancement processing generally uses a first-order gradient operator (SOBEL operator) to achieve edge enhancement, so as to enhance the sharpness of the image by enhancing the contrast of the image edges. Image scaling processing generally can use bilinear interpolation to achieve image scaling, so as to adjust the size of the image by interpolation.

[0045] Furthermore, in this image processing system according to the present invention, refer to Figure 1 , at least two cache processing modules further include a frame buffer type image processing module 23. The frame buffer type image processing module 23 is mainly used to process the video stream in units of frames. The image data of several adjacent frames required in the processing process needs to be in units of frames. Therefore, the cache requirement and delay requirement of the frame buffer type image processing module 23 are higher than those of the line buffer type image processing module 22. It should be noted that the frame buffer type image processing module 23 will introduce data processing delays of multiple frames, and the cache resources it requires are often provided by a centralized synchronous dynamic random access memory (SDRAM) outside the integrated chip. In addition, registers (such as VDMA) inside the integrated chip are also required for assistance.

[0046] Among them, the frame buffer type image processing module 23 is respectively communicatively connected to the line buffer type image processing module 22 and the video stream generation module 3. The frame buffer type image processing module 23 receives the secondary video data from the line buffer type image processing module 22, performs tertiary image processing on the secondary video data to generate tertiary video data, and sends the tertiary video data as cached video data to the video stream generation module 3.

[0047] Specifically, the tertiary image processing includes histogram enhancement processing, 3D denoising processing, and high dynamic range imaging processing. Among them, histogram enhancement processing generally uses an improved histogram equalization method to enhance the detail display effect in strong light and weak light environments of the image, so as to achieve image enhancement through appropriate equalization based on obtaining the histogram statistical characteristics of the image. 3D denoising processing generally uses a time-domain denoising method to achieve image noise reduction, so as to suppress the noise of digital images by integrating the information of multiple adjacent frames. High dynamic range imaging processing (HDR processing) generally uses the enhancement processing of a single-exposure image to expand the dynamic range of the image, so as to expand the dynamic range of the image by means of multiple-exposure fusion or the enhancement processing of a single-exposure image.

[0048] It should be noted that the cache requirements and latency requirements of the pixel-level image processing module 21, the line buffer-based image processing module 22, and the frame buffer-based image processing module 23 increase in sequence. In this way, the distributed processing module 2 is further divided into the pixel-level image processing module 21 with extremely small cache and extremely low latency, the line buffer-based image processing module 22 with relatively small cache and relatively low latency, and the frame buffer-based image processing module 23 with relatively large cache and relatively high latency, effectively realizing the hierarchical management of processing latency and providing the basis and conditions for realizing real-time video transmission with extremely low latency.

[0049] Furthermore, in this image processing system according to the present invention, referring to Figure 1 , the system further includes an embedded processing module 4. The embedded processing module 4 is integrated inside the integrated chip together with the image preprocessing module 1, the distributed processing module 2, and the video stream generation module 3, and is a sequential processor embedded in the integrated chip, which is mainly used to undertake system control with low latency requirements and to monitor system failures or errors. Among them, system control is, for example, the setting of image signal processor (ISP) parameters in the distributed processing module 2 and the latency control of the video stream generation module 3.

[0050] The embedded processing module 4 is communicatively connected to the image acquisition device 5, the external display device 6, the image preprocessing module 1, the distributed processing module 2, and the video stream generation module 3 respectively. Among them, the embedded processing module 4 receives the image device status information sent by the image acquisition device 5 and monitors the failure of the image acquisition device 5 according to the image device status information. That is, the system failure or error monitoring mentioned above. The failure monitoring of the image acquisition device 5 generally includes monitoring whether the image acquisition device 5 has fallen off. Therefore, the image device status information includes at least one of the acceleration change rate and the movement direction of the image acquisition device 5. Among them, the acceleration change rate can be measured by an accelerometer, and the movement direction can be measured by a position sensor. When the acceleration change rate is higher than a certain threshold, for example, the acceleration change rate exceeds 50 m / s³; or the movement direction changes suddenly, for example, the angle with the original movement direction exceeds 50°, it indicates that the image device may have fallen off. Of course, it is also possible to make a comprehensive judgment by combining the acceleration change rate and the movement direction.

[0051] Further, the embedded processing module 4 can also receive the display device status information sent by the external display device 6 and monitor the status of the external display device 6 according to the display device status information. The status monitoring of the external display device 6 generally includes monitoring whether there is a problem of display frame stop in the external display device 6. Moreover, the display device status information includes the frame interval between two adjacent frames of the video displayed by the external display device 6. When the frame interval is greater than a predetermined threshold, such as 50 ms, there is a problem of display frame stop in the external display device 6. In addition, it is also possible to obtain multiple frames of images and compare the similarity of the multiple frames of images. If the images within a period of time maintain a high similarity and do not change, it is determined that the display frame of the external display device 6 stops.

[0052] Furthermore, the embedded processing module 4 can also perform parameter and enable control on processing modules such as the image preprocessing module 1, the distributed processing module 2, and the video stream generation module 3. Specifically, the embedded processing module 4 can also send the pre-stored processing parameter data to the distributed processing module 2 at a predetermined interval and perform parameter setting on the distributed processing module 2, that is, the setting of the image processing parameters mentioned above. The image processing parameters are the parameters corresponding to each process in the above-mentioned primary, secondary, and tertiary image processing.

[0053] Moreover, the embedded processing module 4 can also send the pre-stored delay threshold to the video stream generation module 3 and perform delay control on the video stream generation module 3, that is, the delay control of the video stream generation module 3 mentioned above. It should be noted that the delay control of the video stream generation module 3 is essentially to control the video stream generation module 3 to perform frame encoding processing on the image data according to the specified system delay and frame interval. It should also be noted that the specified system delay is the system delay value of different working conditions calibrated during product design. For example, it is agreed that the system delay value for the high-speed daytime motion scenario is 20 ms, the system delay value for the high-speed nighttime motion scenario is 40 ms, and the system delay value for the low-speed motion scenario is 80 ms, etc. The specified frame interval is the same as the frame interval of the received video stream. For example, if the video frame rate received from the CIS is 50 Hz, representing a 20 ms frame interval, then the frame interval generated by the video stream generation sub-module is also 20 ms. The frame interval is not applicable to scene switching.

[0054] With such a structure, by using the embedded processing module 4 to perform corresponding image processing parameter settings, delay control of the video stream generation module 3, and fault detection of the image acquisition device 5 and the external display device 6, etc., it is possible to detect faults that may occur during the image processing process and perform more accurate control of the delay. Furthermore, the data acquired by the image acquisition device 5 can be displayed on the external display device 6 with lower delay and higher display effect.

[0055] Further, in the image processing system according to the present invention, with reference to Figure 1 , the embedded processing module 4 is also communicatively connected to the environmental data acquisition device 7, obtains external environmental data from the environmental data acquisition device 7, generates scene information based on the external environmental data, and sends the scene information to the distributed processing module 2. The distributed processing module 2 determines the steps for the distributed processing module 2 to perform image processing on the original video data according to the scene information.

[0056] Furthermore, in the image processing system according to the present invention, the environmental data acquisition device 7 includes a speed sensor and a light sensor. And the external environmental data includes the moving speed data of the image acquisition device 5 obtained by the speed sensor and the light intensity information of the external environment obtained by the light sensor. The scene information includes a high-speed daytime scene, a high-speed nighttime scene, a low-speed daytime scene, and a low-speed nighttime scene.

[0057] Moreover, when in the high-speed daytime scene, the distributed processing module 2 omits the steps of the three-level image processing of the frame buffer type image processing module 23 and directly sends the secondary video data as the cached video data to the video stream generation module 3. With such steps, when in the high-speed daytime scene, since the scene changes relatively fast and is relatively easy to be perceived by the naked eye, in order to ensure the display effect and make the delay not easily perceived by the user, it is necessary to reduce the delay as much as possible. Therefore, in this scene, all the steps of the three-level image processing of the frame buffer type image processing module 23 are omitted. By omitting the image processing steps with relatively large cache requirements and delay requirements, part of the image processing performance is sacrificed in exchange for an extremely low delay, improving the display effect of the external display device 6 in the high-speed daytime scene.

[0058] When in the high-speed nighttime scene, the distributed processing module 2 only retains the histogram enhancement processing in the three-level image processing and sends the generated three-level video data as the cached video data to the video stream generation module 3. With such steps, when in the high-speed nighttime scene, although the scene changes relatively fast, it is not easily perceived by the naked eye, and the delay can be appropriately reduced. Therefore, in this scene, only the histogram enhancement processing is retained to ensure the detail display effect of the image in strong light and weak light environments on the premise of minimizing the delay as much as possible.

[0059] When the distributed processing module 2 is in the low-speed daytime scenario and the low-speed nighttime scenario, it retains the histogram enhancement processing in the three-level image processing, and retains at least one of the 3D denoising processing and the high-dynamic range imaging processing, and sends the three-level video data generated by the frame buffer type image processing module 23 to the video stream generation module 3 as the cached video data. With such steps, in the low-speed scenario, the scene changes slowly, so a relatively large delay will not have a great impact on the display effect. At this time, most or all of the steps in the three-level image processing with relatively high cache requirements and delay requirements can be selected to be retained to improve the display effect.

[0060] In such a way, the specific scenarios of the image processing system are classified according to the speed and brightness that can have a major impact on the user's visual perception, and the corresponding steps of the image processing are omitted or retained in a targeted manner in different scenarios, which can meet the different requirements for delay and visual effects in different scenarios.

[0061] Furthermore, in this image processing system according to the present invention, referring to Figure 1 , the image preprocessing module 1, the pixel-level image processing module 21, the line buffer type image processing module 22, the frame buffer type image processing module 23, the video stream generation module 3, and the embedded processing module 4 are all integrated on the same integrated chip. This integrated chip is generally an FPGA integrated chip.

[0062] Furthermore, in this image processing system according to the present invention, a register (VDMA) and several static random access memory modules (SRAM) are also provided on the integrated chip. Specifically, the frame buffer type image processing module 23 and the video stream generation module 3 are both connected to the register, and the register is connected to the external dynamic random access memory module (SDRAM). The pixel-level image processing module 21 and the line buffer type image processing module 22 are each connected to at least one static random access memory module. The read and write speed of the static random access memory module is faster than that of the external dynamic random access memory module, but the capacity is smaller, and it can only support the processing of data with relatively low cache requirements. The external dynamic random access memory module has a larger capacity, and the row and column addresses of the external dynamic random access memory module are multiplexed, and the processing efficiency is higher. Therefore, when there are relatively high cache requirements and processing efficiency requirements, the external dynamic random access memory module can be selected. In such a way, the cache requirements of the pixel-level image processing module 21 with extremely low cache and extremely low delay and the line buffer type image processing module 22 with relatively small cache and relatively low delay are met by using the static random access memory modules inside the integrated chip, without connecting to an external storage module, reducing the occupied space of the image processing system. Connecting the frame buffer type image processing module 23 with relatively large cache and relatively high delay and the video stream generation module 3 to the external dynamic random access memory module via the register inside the integrated chip can improve the processing efficiency, reduce the delay, and meet the relatively high storage requirements.

[0063] Example 2:

[0064] In fact, the image acquisition device and the external display device can be set according to the actual scenario. Next, several specific real-time video transmission scenarios are provided in this embodiment.

[0065] The first scenario is the streaming media rearview mirror of a vehicle. In this scenario, the image acquisition device is an in-vehicle camera, and the external display device is the streaming media rearview mirror or the central control display screen of the vehicle. Moreover, the embedded processing module is also communicatively connected to the vehicle's vehicle controller. Refer to Figure 2 , all processing modules of this image processing system are integrated on the same integrated chip and are respectively connected to the in-vehicle camera, the streaming media rearview mirror or the central control display screen of the vehicle.

[0066] The in-vehicle camera acquires the environmental video around the vehicle as the original image data and transmits it to the image preprocessing module (i.e., the video stream receiving module). After that, the image preprocessing module decodes the original image data to generate and send the original video data. The distributed processing module includes a pixel-level image processing module for performing pixel-level image processing, a line buffer type image processing module for performing line buffer type image processing, and a frame buffer type image processing module for performing frame buffer type image processing. Among them, the cache requirements of the pixel-level image processing module and the line buffer type image processing module are provided by one or more distributed SRAMs. The cache requirement of the frame buffer type image processing module is realized by a centralized SDRAM connected to the registers in the integrated chip. The distributed processing module sequentially performs primary image processing, secondary image processing, and tertiary image processing on the original video data by using the pixel-level image processing module, the line buffer type image processing module, and the frame buffer type image processing module, and then sends the cached video data to the video stream generation module.

[0067] The video stream generation module performs frame encoding processing on the cached video data to generate video stream data and sends the video stream data to the streaming media rearview mirror or the central control display screen of the vehicle. The cache requirement of the video stream generation module is also realized by a centralized SDRAM connected to the registers in the integrated chip.

[0068] An embedded processing module (i.e., an embedded processor) is also integrated in this integrated chip. This embedded processing module is used for error detection such as camera detachment and error detection such as display frame stagnation. Moreover, relevant preset parameters, delay parameters, etc. are also stored in the embedded processing module, and the distributed processing module can be controlled according to these parameters.

[0069] In addition, the embedded processing module also obtains information such as vehicle speed and light intensity from the vehicle controller, determines specific driving scenarios (high-speed daytime scenario, high-speed nighttime scenario, low-speed daytime scenario, and low-speed nighttime scenario) based on the vehicle speed and light intensity information, and controls the processing process of the frame buffer type image processing module according to different driving scenarios. Specifically, in the high-speed daytime scenario, since the scene changes rapidly, in order to ensure driving safety, it is necessary to minimize the delay as much as possible. Therefore, in this scenario, all steps of the three-level image processing of the frame buffer type image processing module are omitted, so as to minimize the delay to ensure driving safety. In the high-speed nighttime scenario, in order to ensure driving safety, a relatively low delay is required, and there are certain requirements for the display effect to make up for the impact of the night environment on the passengers' line of sight. Therefore, in this scenario, only the histogram enhancement processing in the three-level image processing is retained, so that the display effect can be ensured under the condition of minimizing the delay as much as possible. In the low-speed scenario, the vehicle speed is slow, and the surrounding conditions change relatively frequently, and there are high requirements for the display effect. Therefore, in this scenario, all the processing in the three-level image is retained to ensure the display effect.

[0070] Reference Figure 3 , more specifically, the image acquisition device is a camera installed on the vehicle body, and the streaming media rearview mirror includes two outer rearview mirror channels on the left and right, and the camera frame rate is set to 50 frames per second. Each processing module of the image processing system is implemented by selecting the A7 series automotive-grade FPGA of Xilinx (model XA7A100T).

[0071] The data of the outer rearview mirror camera is deserialized by the MAX96706 deserializer through the GMSL interface of the FPGA and then input to the image preprocessing module, and the output video is directly connected to the two display screens on the left and right.

[0072] The image preprocessing module, pixel-level image processing module, line buffer type image processing module, frame buffer type image processing module, and video stream generation module in the FPGA are all encapsulated into IPs that can be processed in parallel. The distributed SRAM required by the pixel-level image processing module and the line buffer type image processing module is implemented using the internal partial BRAM resources of the FPGA; the large-scale caches required by the frame buffer type image processing module and the video stream generation module are implemented using off-chip DDR3 memory particles. At the same time, in the FPGA, the RISC-V soft core is used as a dedicated embedded processing module, and this embedded processing module is responsible for the parameter and enable control of each processing module in the chip, as well as the status monitoring of the camera and the display screen. In addition, the embedded processing module is also responsible for interacting with the in-vehicle host, external ambient light sensor, control buttons, etc. Among them, the interaction with the in-vehicle host is carried out through the in-vehicle host interface, and the vehicle speed information can be obtained. The interaction with the external ambient light sensor can obtain the light intensity information. The interaction with the control buttons can turn on or off the streaming media rearview mirror.

[0073] The FPGA is powered by a vehicle power supply and has an internal independent clock signal.

[0074] The second scenario is a video conferencing scenario. In this scenario, the image acquisition device can be a shooting terminal held by the conference host, and the external display device can be a display terminal held by the participants; the image preprocessing module, the distributed processing module, the video stream generation module, and the embedded processing module can be a cloud server.

[0075] The third scenario is a live broadcast or video teaching scenario. In this scenario, the image acquisition device can be a shooting terminal held by the anchor or teacher, and the external display device can be a display terminal held by the users watching the live broadcast or video; the image preprocessing module, the distributed processing module, the video stream generation module, and the embedded processing module can be a cloud server.

[0076] The fourth scenario is a scenario of real-time video transmission using a drone aerial camera. In this scenario, the image acquisition device is the camera of the drone, and the external display device can be the image receiving and display device corresponding to the drone aerial camera. The image preprocessing module, the distributed processing module, the video stream generation module, and the embedded processing module can be the processing chips integrated in the aerial camera.

[0077] It should be noted that the specific implementation of the processing module in the second to fourth scenarios can refer to the module selection and connection in the first scenario.

[0078] It should also be noted that regardless of the above scenarios, the image preprocessing module, the distributed processing module, the video stream generation module, and the embedded processing module can all perform parallel processing on data. Moreover, each cache processing module in the distributed processing module also performs parallel processing on data, so as to minimize the display delay caused by data processing.

[0079] Embodiment 3:

[0080] Based on the above image processing system, this embodiment provides an image processing method for the image processing system, referring to Figure 4 , including the following steps:

[0081] S1: Obtain the original image data and perform decoding processing on the original image data to generate the original video data;

[0082] S2: Obtain the external environment data, generate scene information according to the external environment data, and perform image processing on the original video data according to the scene information to generate cached video data; wherein, the image processing includes at least primary image processing and secondary image processing;

[0083] S3: Perform framed encoding on the cached video data to generate video stream data.

[0084] Specifically, step S1 is executed by the image preprocessing module in the above image processing system. The steps of obtaining external environment data and generating scene information in step S2 are executed by the embedded processing module; the steps of processing the original video data are executed by the distributed processing module; and step S3 is executed by the video stream generation module.

[0085] More specifically, the external environment data includes the moving speed data of the image acquisition device for obtaining the original image data and the light intensity information of the external environment; the scene information includes high-speed day scenes, high-speed night scenes, low-speed day scenes, and low-speed night scenes. In step S2, obtaining the external environment data and generating scene information based on the external environment data includes: obtaining the moving speed data of the image acquisition device and the light intensity information of the external environment, and judging the current scene based on the moving speed data and the light intensity information. Among them, when the moving speed data is higher than the first speed threshold for a period of time, it is determined that the high-speed scene is entered; when the moving speed data is lower than the second speed threshold for a period of time, it is determined that the low-speed scene is entered. When the average light intensity within a period of time is greater than the first light intensity threshold, it is determined that the day scene is entered; when the average light intensity within a period of time is less than the second light intensity threshold, the night scene is entered. The values of the first speed threshold and the second speed threshold can be determined according to different usage scenarios. For example, when the image acquisition system is applied to the streaming media rearview mirror of a vehicle, the first speed threshold is generally set at 80 Km / h, and the second speed threshold is generally set at 70 Km / h; when the image acquisition system is applied to video conferencing scenarios, live broadcasts, or video teaching scenarios, the first speed threshold is generally set at 10 Km / h, and the second speed threshold is generally set at 5 Km / h; when the image acquisition system is applied to an unmanned aerial vehicle aerial camera, the first speed threshold is generally set at 30 Km / h, and the second speed threshold is generally set at 20 Km / h. And, the first light intensity threshold and the second light intensity threshold have the same value in any scenario. Generally, the first light intensity threshold is set at 100 lux, and the second light intensity threshold is set at 50 lux. It should be noted that the above moving speed data and light intensity information are only used as preset conditions for scene switching. When the above conditions are not met, the current scene remains unchanged.

[0086] As for the specific methods of obtaining other data, the generation methods of video data, etc., they are all specifically described in Embodiment 1, and will not be elaborated in this embodiment.

[0087] Further, in the image processing method according to the present invention, in step S2, the primary image processing includes at least one of color interpolation processing, black level correction processing, white balance processing, color space conversion processing, and gray scale correction processing. The secondary image processing includes at least one of 2D denoising processing, edge enhancement processing, and image scaling processing. Moreover, the image processing further includes tertiary image processing, wherein the tertiary image processing includes histogram enhancement processing, 3D denoising processing, and high dynamic range imaging processing. The cache requirements and latency requirements for the primary, secondary, and tertiary image processing increase in sequence. Preferably, the primary, secondary, and tertiary image processing all execute the specific processing steps mentioned above. As for the specific content of each processing step in the primary, secondary, and tertiary image processing, it is described in the embodiments, and will not be elaborated in this embodiment.

[0088] Further, in the image processing method according to the present invention, in step S2, performing image processing on the original video data according to the scene information and generating cached video data includes: in a high-speed daytime scene, performing primary image processing and secondary image processing and omitting the steps of tertiary image processing; in a high-speed nighttime scene, performing primary image processing and secondary image processing and only retaining histogram enhancement processing in the tertiary image processing; in a low-speed daytime scene and a low-speed nighttime scene, performing primary image processing and secondary image processing and retaining histogram enhancement processing in the tertiary image processing, and retaining at least one of 3D denoising processing and high dynamic range imaging processing.

[0089] According to a large number of experimental results, after the above steps, in a high-speed daytime scene, the processing latency can be controlled within 20 ms; in a high-speed nighttime scene, the processing latency can be controlled within 40 ms; in a low-speed motion scene, the processing latency can be controlled at around 80 ms. Therefore, in this embodiment, the specific scenes of the image processing system are classified according to the speed and brightness that can have a major impact on the user's visual perception, and the corresponding steps of image processing are omitted or retained in a targeted manner in different scenes, which can meet the extremely low latency control requirements in high-speed scenes, while in low-speed scenes, the display effect can be improved as much as possible on the basis of meeting the basic latency requirements.

[0090] Although the present invention has been illustrated and described by referring to some preferred embodiments of the present invention, those of ordinary skill in the art should understand that the above content is a further detailed description of the present invention in combination with specific embodiments, and it cannot be determined that the specific implementation of the present invention is only limited to these descriptions. Those skilled in the art can make various changes in form and details, including making several simple deductions or substitutions, without departing from the spirit and scope of the present invention.

Claims

1. An image processing system, characterized in that, Including: An image preprocessing module, which obtains original image data from an image acquisition device, decodes the original image data to generate original video data, and sends the original video data; A distributed processing module, which is communicatively connected to the image preprocessing module and includes at least two cache processing modules with different cache requirements and latency requirements. The at least two cache processing modules sequentially perform image processing on the original video data from the image preprocessing module to generate cached video data; A video stream generation module, which is communicatively connected to the distributed processing module, obtains the cached video data from the distributed processing module, performs frame encoding processing on the cached video data to generate video stream data, and sends the video stream data to an external display device; and The image preprocessing module, the distributed processing module, and the video stream generation module all perform parallel processing on the received original image data, the original video data, and the cached video data; Each of the cache processing modules in the distributed processing module performs parallel processing on the received data.

2. The image processing system according to claim 1, characterized in that, The at least two cache processing modules include a pixel-level image processing module and a line cache type image processing module; where The pixel-level image processing module is communicatively connected to the image preprocessing module and the line cache type image processing module respectively. The pixel-level image processing module performs primary image processing on the original video data from the image preprocessing module to generate primary video data, and sends the primary video data to the line cache type image processing module. The line cache type image processing module performs secondary image processing on the received primary video data to generate secondary video data, and sends the secondary video data as the cached video data to the video stream generation module; Where The primary image processing includes at least one of color interpolation processing, black level correction processing, white balance processing, color space conversion processing, and gray scale correction processing; The secondary image processing includes at least one of 2D denoising processing, edge enhancement processing, and image scaling processing.

3. The image processing system according to claim 2, wherein The at least two cache processing modules further include a frame cache type image processing module; where The frame cache type image processing module is communicatively connected to the line cache type image processing module and the video stream generation module respectively. The frame cache type image processing module receives the secondary video data from the line cache type image processing module, performs tertiary image processing on the secondary video data to generate tertiary video data, and sends the tertiary video data as the cached video data to the video stream generation module; Where The tertiary image processing includes histogram enhancement processing, 3D denoising processing, and high dynamic range imaging processing; and The cache requirements and the latency requirements of the pixel-level image processing module, the line cache type image processing module, and the frame cache type image processing module increase sequentially.

4. The image processing system according to claim 3, wherein It further includes an embedded processing module; The embedded processing module is communicatively connected to the image acquisition device, the external display device, the image preprocessing module, the distributed processing module, and the video stream generation module respectively; Wherein the embedded processing module receives the image device status information sent by the image acquisition device, and performs fault monitoring on the image acquisition device according to the image device status information; the embedded processing module receives the display device status information sent by the external display device, and performs status monitoring on the external display device according to the display device status information; the embedded processing module sends the pre-stored processing parameter data to the distributed processing module at a predetermined interval and sets parameters for the distributed processing module, and sends the pre-stored delay threshold to the video stream generation module and performs delay control on the video stream generation module; and the embedded processing module is also communicatively connected to an environmental data acquisition device, obtains external environmental data from the environmental data acquisition device, generates scene information according to the external environmental data, and sends the scene information to the distributed processing module. The distributed processing module determines the steps for the distributed processing module to perform image processing on the original video data according to the scene information.

5. The image processing system according to claim 4, wherein, The environmental data acquisition device includes a speed sensor and a light sensor; The external environmental data includes the moving speed data of the image acquisition device obtained by the speed sensor and the light intensity information of the external environment obtained by the light sensor; The scene information includes a high-speed daytime scene, a high-speed nighttime scene, a low-speed daytime scene, and a low-speed nighttime scene; and When the distributed processing module is in the high-speed daytime scene, it omits the three-level image processing steps of the frame buffer type image processing module, and directly sends the secondary video data as the cached video data to the video stream generation module; When the distributed processing module is in the high-speed nighttime scene, it only retains the histogram enhancement processing in the three-level image processing, and sends the generated three-level video data as the cached video data to the video stream generation module; When the distributed processing module is in the low-speed daytime scene and the low-speed nighttime scene, it retains the histogram enhancement processing in the three-level image processing, and retains at least one of the 3D denoising processing and the high dynamic range imaging processing, and sends the three-level video data generated by the frame buffer type image processing module as the cached video data to the video stream generation module.

6. The image processing system according to claim 5, wherein The image preprocessing module, the pixel-level image processing module, the line buffer type image processing module, the frame buffer type image processing module, the video stream generation module, and the embedded processing module are all integrated on the same integrated chip; and A register and a number of static random storage modules are also provided on the integrated chip, wherein the frame buffer type image processing module and the video stream generation module are both connected to the register, and the register is connected to an external dynamic random storage module; The pixel-level image processing module and the line buffer type image processing module are each correspondingly connected to at least one of the static random access memory modules.

7. The image processing system according to claim 6, wherein The image acquisition device is an in-vehicle camera; The external display device is a streaming rearview mirror of the vehicle; and The embedded processing module is also communicatively connected to the vehicle's vehicle control unit.

8. An image processing method for an image processing system according to any one of claims 1-7, characterized in that, It includes: S1: Acquire original image data, and perform decoding processing on the original image data to generate original video data; S2: Acquire external environment data, generate scene information according to the external environment data, and perform image processing on the original video data according to the scene information to generate cached video data; wherein, the image processing includes at least primary image processing and secondary image processing; S3: Perform frame encoding processing on the cached video data to generate video stream data.

9. The image processing method according to claim 8, wherein In step S2, the primary image processing includes at least one of color interpolation processing, black level correction processing, white balance processing, color space conversion processing, and gray scale correction processing; The secondary image processing includes at least one of 2D denoising processing, edge enhancement processing, and image scaling processing; and The image processing further includes tertiary image processing, wherein the tertiary image processing includes histogram enhancement processing, 3D denoising processing, and high dynamic range imaging processing; and The external environment data includes the moving speed data of the image acquisition device for acquiring the original image data and the light intensity information of the external environment; The scene information includes a high-speed daytime scene, a high-speed nighttime scene, a low-speed daytime scene, and a low-speed nighttime scene.

10. The image processing method according to claim 9, wherein In step S2, performing image processing on the original video data according to the scene information and generating cached video data includes: In the high-speed daytime scene, perform the primary image processing and the secondary image processing, and omit the step of the tertiary image processing; In the high-speed nighttime scene, perform the primary image processing and the secondary image processing, and only retain the histogram enhancement processing in the tertiary image processing; In the low-speed daytime scene and the low-speed nighttime scene, perform the primary image processing and the secondary image processing, and retain the histogram enhancement processing in the tertiary image processing, and retain at least one of the 3D denoising processing and the high dynamic range imaging processing.

Images