Binocular camera image processing system and image processing method

Abstract

The invention discloses a binocular camera image processing system and an image processing method. The system comprises an image acquisition module, a left path image processing module, a right path image processing module and a stereo matching module. The image acquisition module is used for acquiring a to-be-processed left path image and a to-be-processed right path image; the left-path image processing module is used for performing multi-stage processing on the left-path image through a pipeline architecture to obtain a target left-path image, and sending the target left-path image to the stereo matching module; the right-path image processing module is used for performing multi-stage processing on the right-path image through the same pipeline architecture to obtain a target right-path image, and sending the target right-path image to the stereo matching module; and the stereo matching module is used for carrying out stereo matching on the target left path image and the target right path image to obtain a target image, and sending the target image to the target image output module for output. According to the invention, binocular stereo vision and light supplement technologies are fused, and the image processing precision of the binocular camera is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a binocular camera image processing system and image processing method, a terminal and a computer-readable storage medium. Background Art

[0002] The common traditional processor architectures include CPU, GPU, DSP, etc. CPU is usually used for general computing tasks, GPU is mainly used for graphics processing, and DSP focuses on digital signal processing tasks. These processors have their own specific architecture and optimization methods to adapt to their specific application scenarios. They usually need to take out an instruction from the memory to analyze the nature and function of the instruction, and complete the corresponding operation according to the function of the instruction, and finally store the operation result in the memory or other registers. The processing process is cumbersome and power-intensive.

[0003] Traditional processor systems have some obvious disadvantages when processing binocular stereo matching algorithms, such as high computational complexity: binocular stereo matching algorithms involve a large number of pixel comparison and search operations, especially when processing high-resolution images, the amount of data calculated is huge. Although the general computing unit of the traditional processor architecture can complete these calculations, its design focuses more on generality rather than optimization specifically for image processing tasks, resulting in relatively low execution efficiency. Lack of parallelism: It is difficult for traditional processor architectures to fully utilize parallelism when executing binocular stereo matching algorithms. Stereo matching involves comparing and searching for each pixel in the image, and these operations have natural parallelism. However, the general computing unit of traditional processors does not support large-scale data parallel operations and cannot efficiently utilize hardware resources. Memory access bottleneck: The binocular stereo matching algorithm has random access to image data and requires frequent memory access. The memory access model of traditional processors may not be able to efficiently cope with this randomness, resulting in performance bottlenecks.

[0004] The above shortcomings are mainly related to the computational complexity and parallel nature of the binocular stereo matching algorithm. It can be seen that the current binocular camera image processing system has low computational performance and efficiency, and lacks parallel technology, and the results obtained by binocular camera image processing are not accurate enough.

[0005] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0006] The main purpose of the present invention is to provide a binocular camera image processing system and an image processing method, a terminal and a computer-readable storage medium, aiming to solve the problems in the prior art that the binocular camera image processing system has low computing performance and efficiency, lacks parallel technology, and the image results obtained by processing are not accurate enough.

[0007] To achieve the above object, the present invention provides a binocular camera image processing system, the binocular camera image processing system comprising: an image acquisition module, a left image processing module, a right image processing module and a stereo matching module; the image acquisition module is connected to the left image processing module and the right image processing module, the left image processing module and the right image processing module are connected to the stereo matching module, and the stereo matching module is connected to a target image output module;

[0008] The image acquisition module is used to acquire a left image and a right image to be processed, send the left image to the left image processing module, and send the right image to the right image processing module;

[0009] The left-channel image processing module is used to perform multi-stage processing on the left-channel image through a pipeline architecture to obtain a target left-channel image, and send the target left-channel image to the stereo matching module;

[0010] The right image processing module is used to perform multi-stage processing on the right image through the same pipeline architecture to obtain a target right image, and send the target right image to the stereo matching module;

[0011] The stereo matching module is used to perform stereo matching on the target left image and the target right image to obtain a target image, and send the target image to a target image output module for output.

[0012] Optionally, the binocular camera image processing system, wherein the left image processing module and the right image processing module include black level correction, white balance, demosaicing, color correction, denoising, sharpening, exposure control, autofocus, epipolar correction, lens shading correction, saturation adjustment, tone mapping, gamma correction and high dynamic processing units composed of the same pipeline architecture.

[0013] Optionally, the binocular camera image processing system further comprises a laser fill light module;

[0014] The laser fill light module is arranged between the image acquisition module and the left image processing module and the right image processing module;

[0015] The laser fill light module is used to obtain the light intensity of the current environment, compare the light intensity with a preset threshold, and if the light intensity is lower than the preset threshold, use diffraction infrared light fill light technology to increase the light intensity of the environment to the preset intensity.

[0016] Optionally, in the binocular camera image processing system, the black level correction, white balance, demosaicing, color correction, denoising, sharpening, exposure control, autofocus, epipolar correction, lens shading correction, saturation adjustment, tone mapping, gamma correction and storage structure of the high dynamic processing unit are all customized storage structures optimized according to the data processing size, which are used to alleviate the memory access bottleneck.

[0017] Optionally, in the binocular camera image processing system, a clock synchronization module is further provided between the left image processing module and the right image processing module;

[0018] The clock synchronization module is used to keep the output time synchronization of the same unit in the left image processing module and the right image.

[0019] In addition, to achieve the above object, the present invention further provides an image processing method, wherein the image processing method comprises:

[0020] Obtaining the left image to be processed, the right image and the light intensity of the current environment, and comparing the light intensity with a preset threshold;

[0021] If the light intensity is lower than the preset threshold, the diffraction infrared light fill-in technology is used to fill-in the left image and the right image to obtain a filled-in left image and a filled-in right image;

[0022] The left image after fill-in light and the right image after fill-in light are respectively processed in parallel in multiple stages through two identical pipeline architectures to obtain a target left image and a target right image;

[0023] Stereo matching is performed on the target left image and the target right image to obtain a complete target image and output it.

[0024] Optionally, the image processing method, wherein the multi-stage parallel processing is performed on the left image after fill light and the right image after fill light by two identical pipeline architectures to obtain the target left image and the target right image, specifically comprising:

[0025] The left image after fill light is subjected to black level correction, white balance, demosaicing, color correction, denoising, sharpening, exposure control, autofocus, epipolar correction, lens shading correction, saturation adjustment, tone mapping, gamma correction and high dynamic processing in sequence through the first pipeline architecture to obtain a target left image;

[0026] Through the second pipeline architecture, the right image after fill light is subjected to black level correction operation, white balance operation, demosaicing operation, color correction operation, denoising operation, sharpening operation, exposure control operation, autofocus operation, epipolar line correction operation, lens shading correction operation, saturation adjustment operation, tone mapping operation, gamma correction operation and high dynamic processing operation in sequence to obtain the target right image.

[0027] Optionally, the image processing method further comprises:

[0028] When any one of the black level correction operation, white balance operation, demosaicing operation, color correction operation, denoising operation, sharpening operation, exposure control operation, autofocus operation, epipolar correction operation, lens shading correction operation, saturation adjustment operation, tone mapping operation, gamma correction operation and high dynamic processing operation of any one of the images is completed, the completion status of the corresponding operation of the other image is obtained;

[0029] If the completion status of the operation corresponding to the other image is incomplete, the completion time required to complete the operation corresponding to the other image is calculated, and when the completion time is over, the two images are controlled to enter the next operation at the same time.

[0030] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and an image processing program stored in the memory and executable on the processor, and when the image processing program is executed by the processor, the steps of the image processing method described above are implemented.

[0031] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an image processing program, and when the image processing program is executed by a processor, the steps of the image processing method as described above are implemented.

[0032] In the present invention, the binocular camera image processing system includes: an image acquisition module, a left-channel image processing module, a right-channel image processing module and a stereo matching module; the image acquisition module is used to acquire the left-channel image and the right-channel image to be processed, and send the left-channel image to the left-channel image processing module, and send the right-channel image to the right-channel image processing module; the left-channel image processing module is used to perform multi-stage processing on the left-channel image through a pipeline architecture to obtain a target left-channel image, and send the target left-channel image to the stereo matching module; the right-channel image processing module is used to perform multi-stage processing on the right-channel image through the same pipeline architecture to obtain a target right-channel image, and send the target right-channel image to the stereo matching module; the stereo matching module is used to perform stereo matching on the target left-channel image and the target right-channel image to obtain a target image, and send the target image to the target image output module for output. The present invention integrates binocular stereo vision and fill light technology, and improves the accuracy of binocular camera image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a structural diagram of a preferred embodiment of the binocular camera image processing system of the present invention;

[0034] Figure 2 is a flow chart of a preferred embodiment of the image processing method of the present invention;

[0035] Figure 3 It is a schematic diagram of the principle of the image processing method of the present invention;

[0036] Figure 4 Schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0037] The present application provides an image processing method and related equipment. To make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0038] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0039] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0040] To solve the problems in the prior art, this embodiment provides a binocular camera image processing system, such as Figure 1 As shown, the binocular camera image processing system includes: an image acquisition module, a left image processing module, a right image processing module and a stereo matching module.

[0041] The image acquisition module is connected to the left image processing module and the right image processing module, the left image processing module and the right image processing module are connected to the stereo matching module, and the stereo matching module is connected to the target image output module.

[0042] The image acquisition module is used to acquire a left image and a right image to be processed, send the left image to the left image processing module, and send the right image to the right image processing module.

[0043] Specifically, the binocular camera of the binocular camera acquires a left image and a right image to be processed, wherein the spectral range of the binocular camera covers visible light and infrared light, and the image acquisition module receives the left image and the right image to be processed sent by the binocular camera, and after performing simple processing on the left image and the right image, the image acquisition module sends the left image to the left image processing module, and synchronously sends the right image to the right image processing module.

[0044] The left-channel image processing module is used to perform multi-stage processing on the left-channel image through a pipeline architecture to obtain a target left-channel image, and send the target left-channel image to the stereo matching module.

[0045] In this embodiment, the image processing module adopts a pipeline architecture, dividing the ISP (image signal processing, a unit mainly used to process the output signal of the front-end image sensor to match the image sensors of different manufacturers) and the different stages in the binocular stereo matching algorithm into independent pipeline stages. Each stage performs a specific task and passes the result to the next stage. Such a design can realize parallel processing and improve the overall processing speed.

[0046] Specifically, the left image processing module includes black level correction, white balance, demosaicing, color correction, denoising, sharpening, exposure control, autofocus, epipolar correction, lens shading correction, saturation adjustment, tone mapping, gamma correction and high dynamic processing unit composed of a pipeline architecture.

[0047] The gamma correction unit is used to perform nonlinear operations or inverse operations on the left image, and send the processed image to the next unit of the pipeline. It can be understood that gamma correction is also called gamma nonlinearization or gamma encoding, which is used to perform nonlinear operations or inverse operations on the brightness or tri-color stimulus values ​​of light in a film or imaging system.

[0048] The epipolar correction unit is used to correct the pixel rows of the image and send the processed image to the next unit in the pipeline. It can be understood that epipolar correction is to mathematically align the camera to the same observation plane so that the pixel rows on the camera are strictly aligned. The purpose of the correction is to convert the two-dimensional matching search of the two images into one dimension, saving calculation and eliminating false matching points.

[0049] The denoising unit is used to analyze the temporal and spatial dependencies in the image to reduce or eliminate the noise in the image, and send the processed image to the next unit in the pipeline. It can be understood that the denoising unit uses spatiotemporal denoising, which is a technology for processing noise in videos or images, which uses temporal and spatial information to improve the denoising effect. This technology reduces or eliminates the noise in the image by analyzing the temporal and spatial dependencies in the image sequence while retaining the key information of the image.

[0050] The demosaicing unit is used to perform demosaicing on the image and send the processed image to the next unit in the pipeline. The high dynamic processing unit is used to perform high dynamic range imaging on the image to obtain the target left-way image, and finally send the target left-way image to the stereo matching module. It can be understood that in computer graphics and film photography, it is a group of technologies used to achieve a larger exposure dynamic range (i.e., a larger difference in light and dark) than ordinary digital imaging technology. The purpose of high dynamic range imaging is to correctly represent the brightness of a large range in the real world, from direct sunlight to the darkest shadow.

[0051] The right image processing module is used to perform multi-stage processing on the right image through the same pipeline architecture to obtain a target right image, and send the target right image to the stereo matching module.

[0052] It should be noted that the right image processing module and the left image processing module are identical in structure, including black level correction, white balance, demosaicing, color correction, denoising, sharpening, exposure control, autofocus, epipolar correction, lens shading correction, saturation adjustment, tone mapping, gamma correction and high dynamic processing units composed of the same pipeline architecture. The function of each unit is also exactly the same, except that the data processed here is the right image, which will not be repeated here.

[0053] The stereo matching module is used to perform stereo matching on the target left image and the target right image to obtain a target image, and send the target image to a target image output module for output.

[0054] It is understandable that binocular stereo matching has always been a hot topic in binocular vision research. A binocular camera takes two viewpoint images of the same scene, the left and right, and uses a stereo matching algorithm to obtain a disparity map, and then a depth map. The application range of depth maps is very wide. Since it can record the distance between objects in the scene and the camera, it can be used for measurement, 3D reconstruction, and synthesis of virtual viewpoints.

[0055] Furthermore, the binocular camera image processing system further includes a laser fill light module; the laser fill light module is arranged between the image acquisition module and the left image processing module and the right image processing module;

[0056] The laser fill light module is used to obtain the light intensity of the current environment, compare the light intensity with a preset threshold, and if the light intensity is lower than the preset threshold, use diffraction infrared light fill light technology to increase the light intensity of the environment to the preset intensity.

[0057] In this embodiment, the present invention combines binocular stereo vision with laser technology, especially to improve the accuracy of binocular stereo matching in low light and low texture environments. Infrared light is an electromagnetic wave between visible light and microwaves with a wavelength range of 0.76 to 2000 microns, which is invisible to the human eye. The laser fill light module projects visible or invisible light patterns (such as infrared rays) into the scene through infrared lasers and diffractive optical elements. In the present invention, the diffraction infrared light fill light technology of the laser is used to increase the illumination of the environment, with the intention of obtaining a higher quality original image, and the illumination intensity is compared with a preset threshold. If the illumination intensity is lower than the preset threshold, the diffraction infrared light fill light technology is used to increase the illumination intensity of the environment, thereby enhancing the reliability of the binocular matching algorithm, especially the semi-global matching algorithm.

[0058] Furthermore, the storage structures of the black level correction, white balance, demosaicing, color correction, denoising, sharpening, exposure control, autofocus, epipolar correction, lens shading correction, saturation adjustment, tone mapping, gamma correction and high dynamic processing unit are all customized storage structures optimized according to the data processing size, which are used to alleviate the memory access bottleneck.

[0059] In this embodiment, the memory access requirements of image signal processing and binocular stereo matching algorithm can be optimized, that is, the storage structure of black level correction, white balance, demosaicing, color correction, denoising, sharpening, exposure control, autofocus, epipolar correction, lens shading correction, saturation adjustment, tone mapping, gamma correction and high dynamic processing unit can be customized. The customized storage structure can improve the efficient access to image data and alleviate the memory access bottleneck.

[0060] Furthermore, a clock synchronization module is provided between the left image processing module and the right image processing module;

[0061] The clock synchronization module is used to keep the output time synchronization of the same unit in the left image processing module and the right image.

[0062] Specifically, the clock synchronization module is used to obtain the clock signals sent by each processing unit in the left-channel image processing module and the right-channel processing module. When any operation of the black level correction operation, white balance operation, de-mosaic operation, color correction operation, denoising operation, sharpening operation, exposure control operation, autofocus operation, polar line correction operation, lens shading correction operation, saturation adjustment operation, tone mapping operation, gamma correction operation and high dynamic processing operation of any image is completed, the completion status of the corresponding operation of the other image is obtained; if the completion status of the corresponding operation of the other image is incomplete, the completion time required to complete the corresponding operation of the other image is calculated. When the completion time ends, the clock synchronization module will send control information to control the two images to enter the next step at the same time.

[0063] Furthermore, the binocular camera image processing system can be optimized for power consumption: the binocular camera image processing system can be designed as a chip specifically optimized for power consumption. By removing unnecessary general computing units and optimizing power management, the binocular camera image processing system provides high-performance computing capabilities at relatively low power consumption, which is suitable for embedded and mobile devices; at the same time, since the binocular camera image processing system is a specially designed hardware solution, it can also provide higher real-time performance, which is suitable for binocular stereo matching applications that require high computing speed, such as autonomous driving and robot navigation.

[0064] As can be seen from the above, the binocular camera image processing system provided by the present invention provides highly optimized modules to realize binocular stereo matching and image signal processing, which can meet the needs of specific tasks. In addition, in the field of binocular stereo matching, the proposed pipeline parallelism technology can significantly improve the computing performance and efficiency, solving the problem that the binocular camera image processing system in the prior art has low computing performance and efficiency, lacks parallelism technology, and the image results obtained by processing are not accurate enough.

[0065] Based on the binocular camera image processing system described in the above embodiment, the present invention also provides an image processing method based on the binocular camera image processing system, specifically as follows Figure 2 and Figure 3 As shown in , the user status detection method includes the following steps:

[0066] Step S10: obtaining the left image to be processed, the right image to be processed and the light intensity of the current environment, and comparing the light intensity with a preset threshold.

[0067] Specifically, a left image and a right image to be processed are obtained through a binocular camera, wherein the spectral range of the binocular camera covers visible light and infrared light, and the light intensity of the current environment is obtained, and the light intensity is compared with a preset threshold value, which is the optimal light intensity during image processing and can be set based on experience.

[0068] Step S20: If the light intensity is lower than the preset threshold, the diffraction infrared light fill-in technology is used to fill-in the left image and the right image to obtain a filled-in left image and a filled-in right image.

[0069] In this embodiment, if the light intensity is lower than the preset threshold, it means that the light intensity of the current environment is low and cannot meet the accuracy requirements of image processing. At this time, diffraction infrared light fill light technology is used to project visible or invisible light patterns (such as infrared rays) into the scene to improve the quality of the left image and the right image, and obtain the left image after fill light and the right image after fill light.

[0070] If the light intensity is not lower than the preset threshold, it means that the light intensity of the current environment is high and can meet the accuracy requirements of image processing. At this time, the left image and the right image are no longer supplemented with light, but the left image and the right image are directly input into the corresponding pipeline architecture for parallel processing.

[0071] Step S30 , performing multi-stage parallel processing on the left image after fill-in light and the right image after fill-in light through two identical pipeline architectures to obtain a target left image and a target right image.

[0072] It is understandable that it is difficult for a traditional processor architecture to fully utilize parallelism when executing a binocular stereo matching algorithm. Stereo matching involves comparing and searching each pixel in an image, and these operations have natural parallelism. However, the general computing unit of a traditional processor does not support large-scale data parallel operations sufficiently, and cannot efficiently utilize hardware resources. In the present invention, the two pipeline architectures are completely symmetrical and there are no differences. The present invention divides the different stages in image signal processing and binocular stereo matching algorithms into independent pipeline stages. Each stage performs a specific task and passes the result to the next stage. Such a design can achieve parallel processing and improve the overall processing speed.

[0073] Specifically, the left image after fill light is subjected to a black level correction operation, a white balance operation, a demosaicing operation, a color correction operation, a denoising operation, a sharpening operation, an exposure control operation, an autofocus operation, an epipolar correction operation, a lens shading correction operation, a saturation adjustment operation, a tone mapping operation, a gamma correction operation, and a high dynamic processing operation in sequence through the first pipeline architecture to obtain a target left image;

[0074] Through the second pipeline architecture, the right image after fill light is subjected to black level correction operation, white balance operation, de-mosaic operation, color correction operation, denoising operation, sharpening operation, exposure control operation, autofocus operation, epipolar correction operation, lens shading correction operation, saturation adjustment operation, tone mapping operation, gamma correction operation and high dynamic processing operation in sequence to obtain the target right image.

[0075] Among them, the gamma correction operation is used to perform nonlinear operations or inverse operations on the image; the epipolar correction operation is used to correct the pixel rows of the image; the denoising operation is used to analyze the time and space dependencies in the image to reduce or eliminate the noise in the image; the demosaicing operation is used to perform demosaicing processing on the image; the high dynamic processing operation is used to perform high dynamic range imaging on the image to obtain the target left (right) image.

[0076] Further, when any one of the black level correction operation, white balance operation, demosaicing operation, color correction operation, denoising operation, sharpening operation, exposure control operation, autofocus operation, epipolar correction operation, lens shading correction operation, saturation adjustment operation, tone mapping operation, gamma correction operation and high dynamic processing operation of any one of the images is completed, the completion status of the corresponding operation of the other image is obtained;

[0077] If the completion status of the operation corresponding to the other image is incomplete, the completion time required to complete the operation corresponding to the other image is calculated, and when the completion time is over, the two images are controlled to enter the next operation at the same time.

[0078] It is understandable that a stereo camera obtains depth information by calculating the disparity information between the left and right images. This is usually achieved using a stereo matching algorithm, which requires the left and right images to be synchronized in time to ensure the accuracy and consistency of the match. Therefore, in a stereo camera, frame synchronization technology is crucial, because if the frames of the two cameras are not synchronized, it will cause a time difference between the left and right images, thus affecting the accuracy of the depth information.

[0079] Step S40: stereo matching is performed on the target left image and the target right image to obtain a complete target image and output it.

[0080] Stereo matching is a process of finding the correspondence between spatial objects in the acquired left and right images, that is, finding the corresponding points of the binocular images, calculating the disparity, and estimating the depth based on the calculated disparity using the principle of similar triangles. Stereo matching is the most important part of establishing stereo vision. The disparity information obtained by stereo matching will directly affect the obtained three-dimensional information.

[0081] In this embodiment, the target left image and the target right image can be stereo matched by a pre-trained neural network model, and the neural network model includes a disparity edge prediction module and a disparity estimation network; wherein the disparity edge prediction module is used to perform disparity edge prediction on the image.

[0082] The disparity estimation network includes: a cost cube construction module for constructing a cost cube, a cost aggregation module for regularizing the cost cube to obtain an aggregated cost body; an aggregated cost optimization module for suppressing the edge area of ​​the aggregated cost body according to the disparity edge prediction map of the reference image; and a disparity regression module for upsampling and regressing the optimized aggregated cost body to obtain a target image.

[0083] Among them, the cost cube construction module is used to extract features of the left image and the right image to obtain low-scale features of the left image and the right image, and then perform a spatial pyramid pooling operation on the low-scale features to obtain high-scale features of the left image and the right image, and then perform weighted fusion of the low-scale features and the high-scale features to obtain fused features of the left image and the right image; finally, according to the preset disparity value, the fused features of the left image and the fused features of the right image are connected at each disparity level to obtain the cost cube between the left image and the right image.

[0084] It can be seen that the present invention designs an image processing method, which realizes high-precision binocular stereo matching and image signal processing, and can meet the needs of specific tasks. In the field of binocular stereo matching, the proposed pipeline parallelism technology can significantly improve the computing performance and efficiency, and solve the problem that the computing performance and efficiency of the binocular camera image processing system in the prior art are low, and the image results obtained by processing are not accurate enough due to the lack of parallelism technology.

[0085] Furthermore, if Figure 4 As shown, based on the above-mentioned image processing method and system, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 4 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0086] The memory 20 may be an internal storage unit of the terminal in some embodiments, such as a hard disk or memory of the terminal. The memory 20 may also be an external storage device of the terminal in other embodiments, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed in the terminal, such as the program code of the installation terminal, etc. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, an image processing program 40 is stored on the memory 20, and the image processing program 40 can be executed by the processor 10, thereby realizing the image processing method in the present application.

[0087] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the image processing method.

[0088] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0089] In one embodiment, when the processor 10 executes the image processing program 40 in the memory 20, the following steps are implemented:

[0090] Obtaining the left image to be processed, the right image and the light intensity of the current environment, and comparing the light intensity with a preset threshold;

[0091] If the light intensity is lower than the preset threshold, the diffraction infrared light fill-in technology is used to fill-in the left image and the right image to obtain a filled-in left image and a filled-in right image;

[0092] The left image after fill-in light and the right image after fill-in light are respectively processed in parallel in multiple stages through two identical pipeline architectures to obtain a target left image and a target right image;

[0093] Stereo matching is performed on the target left image and the target right image to obtain a complete target image and output it.

[0094] The method of performing multi-stage parallel processing on the left image after fill light and the right image after fill light by two identical pipeline architectures to obtain the target left image and the target right image specifically includes:

[0095] The left image after fill light is subjected to black level correction, white balance, demosaicing, color correction, denoising, sharpening, exposure control, autofocus, epipolar correction, lens shading correction, saturation adjustment, tone mapping, gamma correction and high dynamic processing in sequence through the first pipeline architecture to obtain a target left image;

[0096] Through the second pipeline architecture, the right image after fill light is subjected to black level correction operation, white balance operation, demosaicing operation, color correction operation, denoising operation, sharpening operation, exposure control operation, autofocus operation, epipolar line correction operation, lens shading correction operation, saturation adjustment operation, tone mapping operation, gamma correction operation and high dynamic processing operation in sequence to obtain the target right image.

[0097] Wherein, the image processing method further includes:

[0098] When any one of the black level correction operation, white balance operation, demosaicing operation, color correction operation, denoising operation, sharpening operation, exposure control operation, autofocus operation, epipolar correction operation, lens shading correction operation, saturation adjustment operation, tone mapping operation, gamma correction operation and high dynamic processing operation of any one of the images is completed, the completion status of the corresponding operation of the other image is obtained;

[0099] If the completion status of the operation corresponding to the other image is incomplete, the completion time required to complete the operation corresponding to the other image is calculated, and when the completion time is over, the two images are controlled to enter the next operation at the same time.

[0100] In summary, the present invention provides an image processing method, system and terminal, the method comprising: obtaining the left image to be processed, the right image and the light intensity of the current environment, and comparing the light intensity with a preset threshold; if the light intensity is lower than the preset threshold, using the diffraction infrared light fill-in technology to fill-in the left image and the right image, and obtaining the left image after fill-in and the right image after fill-in; performing multi-stage parallel processing on the left image after fill-in and the right image after fill-in through two identical pipeline architectures, respectively, to obtain the target left image and the target right image; performing stereo matching on the target left image and the target right image, obtaining a complete target image and outputting it. The present invention realizes high-precision binocular stereo matching and image signal processing, which can meet the needs of specific tasks. In addition, in the field of binocular stereo matching, the proposed pipeline parallelism technology can significantly improve the computing performance and efficiency, and solves the problem that the computing performance and efficiency of the binocular camera image processing system in the prior art are low, and the image results obtained by processing are not accurate enough due to the lack of parallelism technology.

[0101] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0102] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0103] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A binocular camera image processing system, characterized in that: The binocular camera image processing system comprises: an image acquisition module, a left image processing module, a right image processing module and a stereo matching module; the image acquisition module is connected to the left image processing module and the right image processing module, the left image processing module and the right image processing module are connected to the stereo matching module, and the stereo matching module is connected to the target image output module; The image acquisition module is used to acquire a left image and a right image to be processed, send the left image to the left image processing module, and send the right image to the right image processing module; The left-channel image processing module is used to perform multi-stage processing on the left-channel image through a pipeline architecture to obtain a target left-channel image, and send the target left-channel image to the stereo matching module; The right image processing module is used to perform multi-stage processing on the right image through the same pipeline architecture to obtain a target right image, and send the target right image to the stereo matching module; The stereo matching module is used to perform stereo matching on the target left image and the target right image to obtain a target image, and send the target image to a target image output module for output.

2. The binocular camera image processing system according to claim 1, characterized in that: The left image processing module and the right image processing module include black level correction, white balance, demosaicing, color correction, denoising, sharpening, exposure control, autofocus, epipolar correction, lens shading correction, saturation adjustment, tone mapping, gamma correction and high dynamic processing units composed of the same pipeline architecture.

3. The binocular camera image processing system according to claim 2, characterized in that: The binocular camera image processing system also includes a laser fill light module; The laser fill light module is arranged between the image acquisition module and the left image processing module and the right image processing module; The laser fill light module is used to obtain the light intensity of the current environment, compare the light intensity with a preset threshold, and if the light intensity is lower than the preset threshold, use diffraction infrared light fill light technology to increase the light intensity of the environment to the preset intensity.

4. The binocular camera image processing system according to claim 2, characterized in that: The storage structures of the black level correction, white balance, demosaicing, color correction, denoising, sharpening, exposure control, autofocus, epipolar correction, lens shading correction, saturation adjustment, tone mapping, gamma correction and high dynamic processing unit are all customized storage structures optimized according to the data processing size, which are used to alleviate the memory access bottleneck.

5. The binocular camera image processing system according to claim 1, characterized in that: A clock synchronization module is also provided between the left image processing module and the right image processing module; The clock synchronization module is used to keep the output time synchronization of the same unit in the left image processing module and the right image.

6. An image processing method, characterized in that: The image processing method comprises: Obtaining the left image to be processed, the right image and the light intensity of the current environment, and comparing the light intensity with a preset threshold; If the light intensity is lower than the preset threshold, the diffraction infrared light fill-in technology is used to fill-in the left image and the right image to obtain a filled-in left image and a filled-in right image; The left image after fill-in light and the right image after fill-in light are respectively processed in parallel in multiple stages through two identical pipeline architectures to obtain a target left image and a target right image; Stereo matching is performed on the target left image and the target right image to obtain a complete target image and output it.

7. The image processing method according to claim 6, characterized in that: The method of performing multi-stage parallel processing on the left image after fill light and the right image after fill light by two identical pipeline architectures to obtain the target left image and the target right image specifically includes: The left image after fill light is subjected to black level correction, white balance, demosaicing, color correction, denoising, sharpening, exposure control, autofocus, epipolar correction, lens shading correction, saturation adjustment, tone mapping, gamma correction and high dynamic processing in sequence through the first pipeline architecture to obtain a target left image; Through the second pipeline architecture, the right image after fill light is subjected to black level correction operation, white balance operation, demosaicing operation, color correction operation, denoising operation, sharpening operation, exposure control operation, autofocus operation, epipolar line correction operation, lens shading correction operation, saturation adjustment operation, tone mapping operation, gamma correction operation and high dynamic processing operation in sequence to obtain the target right image.

8. The image processing method according to claim 7, characterized in that: The image processing method further includes: When any one of the black level correction operation, white balance operation, demosaicing operation, color correction operation, denoising operation, sharpening operation, exposure control operation, autofocus operation, epipolar correction operation, lens shading correction operation, saturation adjustment operation, tone mapping operation, gamma correction operation and high dynamic processing operation of any one of the images is completed, the completion status of the corresponding operation of the other image is obtained; If the completion status of the operation corresponding to the other image is incomplete, the completion time required to complete the operation corresponding to the other image is calculated, and when the completion time is over, the two images are controlled to enter the next operation at the same time.

9. A terminal, characterized in that: The terminal comprises: a memory, a processor, and an image processing program stored in the memory and executable on the processor, wherein the image processing program implements the steps of the image processing method according to any one of claims 6 to 8 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an image processing program, and when the image processing program is executed by a processor, the steps of the image processing method according to any one of claims 6 to 8 are implemented.

Images